Can Google Spider Discover Web Pages With No Inbound Links?
I think Google's algorithm rely on the links to discover new web pages. That made me wonder if I create a domain with no incoming links to it; will Google still be able to discover it? I best guess is 'No', but I might be wrong - because I don't know how Google functions.
I believe only way for Google to determine the new domains is through the registry information. I'm not sure whether the domain registrars share the domain registration information with Google (they should not, actually). But a new domain IP might trigger Google's spiders to check it out.
Now consider a new web page added on an existing domain. Unless specifically blocked by robots.txt file entry; Google will figure out all the links by following the links on the homepage. I doubt they can discover random pages on the domain that haven't been linked by anone, not even the home page.
What are your thoughts on this? Can there be a mechanism to discover the web-pages that don't have any inbound links?
I believe only way for Google to determine the new domains is through the registry information. I'm not sure whether the domain registrars share the domain registration information with Google (they should not, actually). But a new domain IP might trigger Google's spiders to check it out.
Now consider a new web page added on an existing domain. Unless specifically blocked by robots.txt file entry; Google will figure out all the links by following the links on the homepage. I doubt they can discover random pages on the domain that haven't been linked by anone, not even the home page.
What are your thoughts on this? Can there be a mechanism to discover the web-pages that don't have any inbound links?
Replies
-
rahul69We can't be too sure what Google actually does (as only they know it themselves), but I think they can discover those new web pages, after all Google provides its own DNS service (GoogleDNS), so they may be having the new domain information😀 .
Now the random pages on the domain that haven't been linked by anyone (and of course not mentioned in the site map) are the one that are difficult to be discovered, and generally slip by the web crawlers. -
Kaustubh KatdareYeah - Google DNS service might have a record of all the new domains. But it can't discover the pages that haven't been linked. I want to know - whether Google performs a random guess on the websites to discover web-pages? For example, Google knows what kind of structures / frameworks are offered by popular web content management systems (WordPress, Joomla, Drupal). It can use that information to possibly guess the new URLs. It looks highly unlikely though.
-
rahul69I don't think random guessing leads Google to new web-pages. I think Google spiders keep on crawling the websites on regular basis, and keeps updating if new web pages are found. However they may miss the updates due to some certain reasons, for example if website is temporarily not available when crawling is attempted on that website etc.
Also I too would like to know whether the structure of CMS can help in discovering web pages. Willing to hear more on this 😒 -
max0077I don't know how much this is related to the question but i found this very interesting demonstration ,so i want to share it here..how google spider works in SEO
You are reading an archived discussion.
Related Posts
sir/mam can u please explain me about fuzzy logic????
We've the latest news and I hope it's a rumor. Infosys might be laying off about 5000 employees in the coming days, the TOI is reporting. I'm not sure if...
hi im new here so hello fellas!
i'm using an old school 21 inch CRt monitor as those used in graphics industry around 6 years ago.
i was curious about...
I'm looking for M.Tech level seminar on power system engineering. I'm currently a studen tof 1st year M.Tech (electrical engineering) student.
Rules are against it and still we find this real common in CE
Instead of using the rating many post Thank you as a reply to a thread that really...