Lesson 2: Crawling: How Googlebot Discovers Pages

Why crawling comes first

In Level 1 you saw how a results page is built. Before Google can show a page there, it has to know the page exists and be able to download it. That stage is called crawling. The program that does it is Googlebot, also called a crawler, robot, bot or spider.

Think of Googlebot as a delivery driver with a growing list of addresses. It only visits addresses that are on its list, and it can only deliver to buildings whose doors are open. This lesson covers both parts: how addresses (URLs) get onto the list, and what keeps the doors closed.

How Googlebot discovers URLs

Google's documentation describes three ways a URL becomes known:

1. Already known. Google revisits pages it has crawled before to check for changes.
2. Links. Google extracts a link from a page it already knows and finds a new URL. This is the main way discovery happens. A new blog post linked from your home page will be found much sooner than one with no links pointing to it.
3. Sitemaps. You give Google a list of URLs in a file called a sitemap.

A link only counts if Google can read it. Google states that it can only crawl a link if it is an <a> HTML element with an href attribute:

• Crawlable: <a href="https://example.com/shoes">Shoes</a>
• Not reliably crawlable: <a onclick="goto('/shoes')">, <span href="/shoes">, <a routerLink="shoes">

If your developer builds navigation from click handlers rather than real href links, Googlebot may never find the pages behind them.

What Googlebot does when it visits

• Two crawlers. Googlebot Smartphone simulates a mobile user. Googlebot Desktop simulates a desktop user.
• Polite speed. Google says Googlebot should not access a site more than once every few seconds on average. It is built to avoid overloading servers.
• Size cap. Googlebot fetches the first 2 MB of a supported file type (64 MB for PDFs). Content after that limit may not be seen.

What stops crawling

Google names three main obstacles:

1. Server problems. If your server returns errors such as HTTP 500, Google reads that as "slow down" and crawls less.
2. Network issues. Timeouts and DNS failures stop the download.
3. robots.txt rules. A rule in robots.txt can tell Googlebot not to fetch certain URLs.

Task 1: Open and read your robots.txt

robots.txt is a plain text file that must sit in the top-level directory of your site, for example https://example.com/robots.txt.

1. Type your domain followed by /robots.txt into your browser.
2. Find the User-agent lines. User-agent: * applies to all crawlers. User-agent: Googlebot applies only to Google.
3. Read each Disallow and Allow line under that user agent. A Disallow path tells the crawler not to fetch matching URLs.
4. Look for a Sitemap: line and write down its URL.
5. Check for dangerous rules:
   • Disallow: / blocks the whole site. This is often left over from a staging site.
   • Rules that block important sections, such as /products/ or /blog/.
   • Rules that block CSS or JavaScript folders your pages need.

Example:

User-agent: *
Disallow: /cart/
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

This blocks the cart and admin areas, which is sensible, and points to the sitemap.

Facts worth knowing:

• Google supports four fields: user-agent, allow, disallow and sitemap. Google ignores crawl-delay.
• Google reads only the first 500 KiB of the file.
• Google usually caches robots.txt for up to 24 hours, so changes do not take effect instantly.
• If robots.txt returns a 404 (or most other 4xx errors), Google treats the site as having no restrictions. If it returns a 5xx server error, Google stops crawling the site for 12 hours and may then fall back to a cached copy.
• Blocking is not hiding. robots.txt controls crawling, not indexing. Google may still show a blocked URL in results, without a description. You will learn the right tools for keeping pages out of the index in Level 3.

If you have Search Console access, open the robots.txt report under Settings. It shows which robots.txt files Google found, when it last fetched them and any errors.

Task 2: Find your XML sitemap

A sitemap is a file that lists the URLs you want Google to know about.

1. Check the Sitemap: line you found in robots.txt.
2. If there is none, try common addresses: /sitemap.xml, /sitemap_index.xml or /wp-sitemap.xml (WordPress).
3. Open the file. You should see <loc> entries with full URLs, or a sitemap index that links to several sitemaps.
4. Note the sitemap URL in your SEO notes.
5. Check a few listed URLs. They should be live pages you actually want found, not redirects or error pages.

What Google says about sitemaps:

• Each file can hold up to 50,000 URLs or 50 MB uncompressed. Bigger sites split their URLs across several files listed in a sitemap index.
• Google uses <lastmod> only when it is consistently accurate. It ignores <priority> and <changefreq>.
• You can submit a sitemap in Search Console or reference it in robots.txt.
• A sitemap is a hint, not a guarantee. Google may not download it or crawl every URL in it. Good internal links still matter.

Key takeaways

• Googlebot discovers URLs by revisiting known pages, following links and reading sitemaps.
• Only <a> elements with an href are reliably crawlable links.
• Server errors and network issues slow or stop crawling. robots.txt rules block it on purpose.
• robots.txt lives at the root of your domain. Look for Disallow: / and for blocks on important sections.
• robots.txt blocks crawling, not indexing.
• Sitemaps: up to 50,000 URLs or 50 MB per file. They are a hint, and Google ignores priority and changefreq.
• Do now: record your robots.txt findings and your sitemap URL for the next levels.