How Search Engines Work: Seo 101

By Author: Rommel Lim
Total Articles: 1
Comment this article

Search engines can be considered as answer machines. They exist to understand, discover and organize the content of the internet to provide the most relevant results to the questions that searchers are asking. For content to show up in search results, your content must first be visible to search engines. It is perhaps the most important thing in SEO: If your site can’t be found, there is no way you will show up in the Search Engine Results Page (SERP). You can learn about this from websites that feature digital marketing courses in Singapore.
How do search engines work?
Search engines work in terms of three primary functions:
1. Crawling: Search the internet for content, looking over the code/content for every URL they find.
2. Indexing: Store and organize the content found during the crawling process. As soon as a page is in index, it is a candidate to be displayed as a result of relevant queries.
3. Ranking: Provide the pieces of content that are best going to answer a searcher’s query, which means that results are ordered by most relevant to least relevant.
What is search engine crawling?
...
... Crawling is the discovery process in which search engines send out a team of robots, also known as crawlers or spiders, to find updated and new content. Content may vary – it might be a webpage, a video, an image, a PDF, etc. Regardless of the format, content is discovered through links. Googlebot begins by fetching out a few pages, and then follows the links on those webpages to find new URLs. By following through this path of links, the crawler finds new content and add it to their index called Caffeine, which is a massive database of discovered URLs, to later be retrieved when a searcher is finding information that the content on that URL is a good match for.
What is a search engine index?
Search engines process and store information they find in an index, which is a huge database of all the content they have discovered and consider good enough to serve up to searchers.
Search engine ranking
When someone does a search, search engines scan their index for highly relevant content and then orders that content with the hopes of solving the query of the searcher. This ordering of search results in terms of relevance is called ranking. Typically, you can assume that the higher a website is ranked, the more relevant the search engine believes that site is to the query.
It is possible to block search engine crawlers from part of or all of your site, or instruct search engines to avoid storing specific pages in their index. While there can be reasons for doing this, if you want your content found by searchers, you have to first make sure it’s accessible to crawlers and is indexable. Otherwise, it’s as good as invisible.
In SEO, not all search engines are equal
A lot of beginners wonder about the relative importance of certain search engines. Most people know that Google has the biggest market share, but how important is it to optimize for Bing, Yahoo and other search engines? The truth is that even if there are more than 30 major search engines, the SEO community only pays attention to Google. The reason for this is that Google is where the vast majority of people search the web. If we include Google maps, Google images and YouTube, more than 90% of web searches happen in Google. This is almost 20 times Bing and Yahoo combined.
Crawling: Are search engines able to find your pages?
As you have just learned, making sure that your site gets crawled and indexed is a prerequisite to showing up in the SERPs. If you already have a website, it might be a good idea to start by seeing how many of your pages are in the index. This is going to yield some great insights into whether Google is crawling and finding all the pages you want it to.
One way to check your indexed pages is “site: yourdomain.com”, an advanced search operator. Go to Google and type “site: yourdomain.com” into the search bar. This will return results Google has in its index for the site specified.
The number of results Google displays is not accurate, but it gives you a solid idea of which pages are indexed on your site as well as how they are currently showing up in search results.
For more exact results, use the Index Coverage report in Google Search Console. You can sign up for a free Google Search Console account if you presently do not have one. Using this tool, you can submit sitemaps for your site and monitor how many submitted pages have actually been added to Google’s index, among other things.
If you are not showing up anywhere in the search results, there are a few reasons why:
- Your site is new and has not been crawled yet.
- Your site is not linked to from any external websites.
- Your site’s navigation makes it difficult for a robot to crawl it effectively.
- Your site contains some basic code called crawler directives that is blocking search engines.
- Your site has been penalized by Google for spammy tactics.
Robots.txt
Robots.txt files are found in the root directory of websites and recommend which parts of your site search engines should and should not crawl and also the speed at which they crawl your site through specific robots.txt directives. If Googlebot can’t find a robots.txt file for a site, it proceeds to crawl the site. If Googlebot finds a robots.txt file for a site, it usually abides by the suggestions and proceeds to crawl the site. If Googlebot encounters an error while trying to access the site’s robots.txt file and cannot determine if one exists or not, it will not crawl the site. Not all web robots follow robots.txt. For example, people with bad intentions build bots that do not follow this protocol.