Find hidden pages of website

Manish Goyal

Manish Goyal

@manish-r2Hoep Oct 14, 2024
Some times there are some pages that web crawlers are not allowed to crawl
but you can view those pages

Every web server has a file named robots.txt stored that accomplish this purpose but with the help of this file you can view those pages also

like for eg if you want to see hidden pages of crazyengineers site
then just type this in your address bar

#-Link-Snipped-#

Replies

Welcome, guest

Join CrazyEngineers to reply, ask questions, and participate in conversations.

CrazyEngineers powered by Jatra Community Platform

  • Kaustubh Katdare

    Kaustubh Katdare

    @thebigk Feb 27, 2010

    goyal420
    Some times there are some pages that web crawlers are not allowed to crawl
    but you can view those pages

    Every web server has a file named robots.txt stored that accomplish this purpose but with the help of this file you can view those pages also

    like for eg if you want to see hidden pages of crazyengineers site
    then just type this in your address bar

    #-Link-Snipped-#
    .

    Can you access those pages, really?
  • Manish Goyal

    Manish Goyal

    @manish-r2Hoep Feb 27, 2010

    I have not tried to access these pages but if some authorization is required then i can't..
  • Manish Goyal

    Manish Goyal

    @manish-r2Hoep Feb 28, 2010

    one question biggie the directories that i am currently viewing in this file exists or not ..
  • MaRo

    MaRo

    @maro-Ce3knx Feb 28, 2010

    Robots.txt is not a tool for Admins to hide secret page from search engines or because they're harmful if known to public, it's a tool to tell the crawler that these pages aren't useful, redirecting it to the core content & sitemaps links.

    - To narrow the caching focus that increases keyword relevancy that increases the rank.

    - It's a helpful tool beside sitemaps to let Admins make search engine focus on the actual content of the website.

    BTW, everybody has robots.txt even Microsoft & Google.
  • Manish Goyal

    Manish Goyal

    @manish-r2Hoep Feb 28, 2010

    Thanks Maro sir for adding to my knowledge further