Find hidden pages of website

Some times there are some pages that web crawlers are not allowed to crawl
but you can view those pages

Every web server has a file named robots.txt stored that accomplish this purpose but with the help of this file you can view those pages also

like for eg if you want to see hidden pages of crazyengineers site
then just type this in your address bar

#-Link-Snipped-#

Like 0 Replies 5

Replies

Welcome, guest

Join CrazyEngineers to reply, ask questions, and participate in conversations.

Sign up Log in

CrazyEngineers powered by Jatra Community Platform

  • Kaustubh Katdare
    Kaustubh Katdare

    @thebigk • Feb 27, 2010

    goyal420Some times there are some pages that web crawlers are not allowed to crawl
    but you can view those pages

    Every web server has a file named robots.txt stored that accomplish this purpose but with the help of this file you can view those pages also

    like for eg if you want to see hidden pages of crazyengineers site
    then just type this in your address bar

    #-Link-Snipped-#

    .

    Can you access those pages, really?

    0
  • Manish Goyal
    Manish Goyal

    @manish-r2Hoep • Feb 27, 2010

    I have not tried to access these pages but if some authorization is required then i can't..

    0
  • Manish Goyal
    Manish Goyal

    @manish-r2Hoep • Feb 28, 2010

    one question biggie the directories that i am currently viewing in this file exists or not ..

    0
  • MaRo
    MaRo

    @maro-Ce3knx • Feb 28, 2010

    Robots.txt is not a tool for Admins to hide secret page from search engines or because they're harmful if known to public, it's a tool to tell the crawler that these pages aren't useful, redirecting it to the core content & sitemaps links.

    - To narrow the caching focus that increases keyword relevancy that increases the rank.

    - It's a helpful tool beside sitemaps to let Admins make search engine focus on the actual content of the website.

    BTW, everybody has robots.txt even Microsoft & Google.

    0
  • Manish Goyal
    Manish Goyal

    @manish-r2Hoep • Feb 28, 2010

    Thanks Maro sir for adding to my knowledge further

    0