CrazyEngineers Logo

Find hidden pages of website

Some times there are some pages that web crawlers are not allowed to crawl
but you can view those pages

Every web server has a file named robots.txt stored that accomplish this purpose but with the help of this file you can view those pages also

like for eg if you want to see hidden pages of crazyengineers site
then just type this in your address bar

#-Link-Snipped-#

Replies

  • Kaustubh Katdare
    Kaustubh Katdare
    goyal420
    Some times there are some pages that web crawlers are not allowed to crawl
    but you can view those pages

    Every web server has a file named robots.txt stored that accomplish this purpose but with the help of this file you can view those pages also

    like for eg if you want to see hidden pages of crazyengineers site
    then just type this in your address bar

    #-Link-Snipped-#
    .

    Can you access those pages, really?
  • Manish Goyal
    Manish Goyal
    I have not tried to access these pages but if some authorization is required then i can't..
  • Manish Goyal
    Manish Goyal
    one question biggie the directories that i am currently viewing in this file exists or not ..
  • MaRo
    MaRo
    Robots.txt is not a tool for Admins to hide secret page from search engines or because they're harmful if known to public, it's a tool to tell the crawler that these pages aren't useful, redirecting it to the core content & sitemaps links.

    - To narrow the caching focus that increases keyword relevancy that increases the rank.

    - It's a helpful tool beside sitemaps to let Admins make search engine focus on the actual content of the website.

    BTW, everybody has robots.txt even Microsoft & Google.
  • Manish Goyal
    Manish Goyal
    Thanks Maro sir for adding to my knowledge further

You are reading an archived discussion.

Related Posts

automatic door opening system

i want to do a project on automatic door opening system in the coming 3 month and i want some advice

Engineering should not be a 4 year course!

I know few people who think engineering can be compressed into two year course and people who think it should be minimum 5 years. What's your opinion and why?

boot mng missing

installing linux over vista in dell .linux is start but for vista message is boot mng missing ctrl+alt+del restart.i want to know about this?

embedded/vlsi/dsp/communication

I want to know 1)Which is the most demanding (in India or abroad)among embedded,vlsi & dsp? 2)which subjects i should brush up with for these fields? 3)apart frm these &...

Solve this Puzzle.

I got this problem through SMS. Am not able to solve this. Lets see, our CEans will solve this? Here it is : The fishermen love me. But Doctor hate...