CrazyEngineers
CrazyEngineers

  • Can Google Spider Discover Web Pages With No Inbound Links?

    Kaustubh Katdare

    Kaustubh Katdare

    @thebigk
    Updated: Oct 22, 2024
    Views: 836
    I think Google's algorithm rely on the links to discover new web pages. That made me wonder if I create a domain with no incoming links to it; will Google still be able to discover it? I best guess is 'No', but I might be wrong - because I don't know how Google functions.

    I believe only way for Google to determine the new domains is through the registry information. I'm not sure whether the domain registrars share the domain registration information with Google (they should not, actually). But a new domain IP might trigger Google's spiders to check it out.

    Now consider a new web page added on an existing domain. Unless specifically blocked by robots.txt file entry; Google will figure out all the links by following the links on the homepage. I doubt they can discover random pages on the domain that haven't been linked by anone, not even the home page.

    What are your thoughts on this? Can there be a mechanism to discover the web-pages that don't have any inbound links?
    0
    Replies
Howdy guest!
Dear guest, you must be logged-in to participate on CrazyEngineers. We would love to have you as a member of our community. Consider creating an account or login.
Replies

  • rahul69

    Member • Jan 4, 2013

    We can't be too sure what Google actually does (as only they know it themselves), but I think they can discover those new web pages, after all Google provides its own DNS service (GoogleDNS), so they may be having the new domain information😀 .
    Now the random pages on the domain that haven't been linked by anyone (and of course not mentioned in the site map) are the one that are difficult to be discovered, and generally slip by the web crawlers.
    Are you sure? This action cannot be undone.
    Cancel

  • Kaustubh Katdare

    Administrator • Jan 4, 2013

    Yeah - Google DNS service might have a record of all the new domains. But it can't discover the pages that haven't been linked. I want to know - whether Google performs a random guess on the websites to discover web-pages? For example, Google knows what kind of structures / frameworks are offered by popular web content management systems (WordPress, Joomla, Drupal). It can use that information to possibly guess the new URLs. It looks highly unlikely though.
    Are you sure? This action cannot be undone.
    Cancel

  • rahul69

    Member • Jan 4, 2013

    I don't think random guessing leads Google to new web-pages. I think Google spiders keep on crawling the websites on regular basis, and keeps updating if new web pages are found. However they may miss the updates due to some certain reasons, for example if website is temporarily not available when crawling is attempted on that website etc.
    Also I too would like to know whether the structure of CMS can help in discovering web pages. Willing to hear more on this 😒
    Are you sure? This action cannot be undone.
    Cancel

  • max0077

    Member • Jan 7, 2013

    I don't know how much this is related to the question but i found this very interesting demonstration ,so i want to share it here..how google spider works in SEO
    Are you sure? This action cannot be undone.
    Cancel
Home Channels Search Login Register