Find hidden pages of website

@manish-r2Hoep • Oct 14, 2024

Oct 14, 2024

1.3K

Some times there are some pages that web crawlers are not allowed to crawl
but you can view those pages

Every web server has a file named robots.txt stored that accomplish this purpose but with the help of this file you can view those pages also

like for eg if you want to see hidden pages of crazyengineers site
then just type this in your address bar

#-Link-Snipped-#

Like 0 Replies 5

Replies

Welcome, guest

Join CrazyEngineers to reply, ask questions, and participate in conversations.

CrazyEngineers powered by Jatra Community Platform

Kaustubh Katdare

@thebigk • Feb 27, 2010

goyal420
Some times there are some pages that web crawlers are not allowed to crawl
but you can view those pages

Every web server has a file named robots.txt stored that accomplish this purpose but with the help of this file you can view those pages also

like for eg if you want to see hidden pages of crazyengineers site
then just type this in your address bar

#-Link-Snipped-#
.

Can you access those pages, really?

0
Manish Goyal

@manish-r2Hoep • Feb 27, 2010

I have not tried to access these pages but if some authorization is required then i can't..
0
Manish Goyal

@manish-r2Hoep • Feb 28, 2010

one question biggie the directories that i am currently viewing in this file exists or not ..
0
MaRo

@maro-Ce3knx • Feb 28, 2010

Robots.txt is not a tool for Admins to hide secret page from search engines or because they're harmful if known to public, it's a tool to tell the crawler that these pages aren't useful, redirecting it to the core content & sitemaps links.

- To narrow the caching focus that increases keyword relevancy that increases the rank.

- It's a helpful tool beside sitemaps to let Admins make search engine focus on the actual content of the website.

BTW, everybody has robots.txt even Microsoft & Google.
0
Manish Goyal

@manish-r2Hoep • Feb 28, 2010

Thanks Maro sir for adding to my knowledge further
0