Can Google Spider Discover Web Pages With No Inbound Links?

Kaustubh Katdare

Kaustubh Katdare

@thebigk • Oct 22, 2024
I think Google's algorithm rely on the links to discover new web pages. That made me wonder if I create a domain with no incoming links to it; will Google still be able to discover it? I best guess is 'No', but I might be wrong - because I don't know how Google functions.

I believe only way for Google to determine the new domains is through the registry information. I'm not sure whether the domain registrars share the domain registration information with Google (they should not, actually). But a new domain IP might trigger Google's spiders to check it out.

Now consider a new web page added on an existing domain. Unless specifically blocked by robots.txt file entry; Google will figure out all the links by following the links on the homepage. I doubt they can discover random pages on the domain that haven't been linked by anone, not even the home page.

What are your thoughts on this? Can there be a mechanism to discover the web-pages that don't have any inbound links?
Like 0 Replies 4

Replies

Welcome, guest

Join CrazyEngineers to reply, ask questions, and participate in conversations.

Sign up Log in

CrazyEngineers powered by Jatra Community Platform

  • rahul69

    rahul69

    @rahul69-97fAOs • Jan 4, 2013

    We can't be too sure what Google actually does (as only they know it themselves), but I think they can discover those new web pages, after all Google provides its own DNS service (GoogleDNS), so they may be having the new domain information😀 .
    Now the random pages on the domain that haven't been linked by anyone (and of course not mentioned in the site map) are the one that are difficult to be discovered, and generally slip by the web crawlers.
    0
  • Kaustubh Katdare

    Kaustubh Katdare

    @thebigk • Jan 4, 2013

    Yeah - Google DNS service might have a record of all the new domains. But it can't discover the pages that haven't been linked. I want to know - whether Google performs a random guess on the websites to discover web-pages? For example, Google knows what kind of structures / frameworks are offered by popular web content management systems (WordPress, Joomla, Drupal). It can use that information to possibly guess the new URLs. It looks highly unlikely though.
    0
  • rahul69

    rahul69

    @rahul69-97fAOs • Jan 4, 2013

    I don't think random guessing leads Google to new web-pages. I think Google spiders keep on crawling the websites on regular basis, and keeps updating if new web pages are found. However they may miss the updates due to some certain reasons, for example if website is temporarily not available when crawling is attempted on that website etc.
    Also I too would like to know whether the structure of CMS can help in discovering web pages. Willing to hear more on this 😒
    0
  • max0077

    max0077

    @max0077-RbbXxe • Jan 7, 2013

    I don't know how much this is related to the question but i found this very interesting demonstration ,so i want to share it here..how google spider works in SEO
    0