How to build a web crawler and which languages are required?

MaRo

This is most of my work now 😁

As you may know, we're indexing media content, so we're making freakishly different indexing techniques.

For text we're researching in several indexing techniques.

1- Google's technique.
2- Keyword analysis, the spider suppose to understand the content of a single article & rank the page according to previously saved keywords.

means, the spider knows for example the keyword 'Engineer' & has a network of keywords related to the word & crawls for the topic's keyword & analyze the contents of the found pages & rank them according to their relevancy, like if the spider found an article talking about engineers having vacancy will has a rank less than engineers invented a tool with technical details.

mahesh_dahale

technology:- ASP.NET supports the searching of files using the Windows Indexing Service, Microsoft .NET Framework or above is required. The code are in C#, Building a word index for a website by using a web crawler also not dependent on the underlying technology used on a website

methods like SetQuery(query as string),GetSearchResults(),

and still thinking on indexing methodology and web crawling product -The Website Utility