Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Technical SEO > Crawler Management > Flashcards

Crawler Management Flashcards

1
Q

Why does crawl budget matter to Google?

A

Two reasons:

  1. They don’t want to overwhelm your servers
  2. The web is huge. Crawling costs money. They want to conserve resources and only use them on the most important pages.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What is ‘crawl rate’?

A

The number of requests that Google’s crawlers make to your site each second

Bonus tip: In ‘SEO Mythbusting’ Martin Splitt described crawl rate as the amount of stress Google can put on your server without overwhelming it

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What affects crawl demand?

A
  1. The popularity of the site
  2. The size of the site, also known as perceived URL inventory (though a large, unpopular site may not necessarily be crawled very often)
  3. The perceived freshness of a site - e.g. a Wiki page on the history of WWI may not change as frequently as a news site
  4. How many pages on a site have not been crawled (especially in the case of a migration)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What is crawl demand?

A

How often Google wants to crawl your site

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Hypothetically, if a site ranked really well but was not crawled very often (low crawl demand), why could that be?

A

They have quality content, but the content does not change very often. This could be due to the nature of the content (dictated by the niche the website is in)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

What are some ways you can indicate to Google how frequently content needs to be crawled?

A
  1. The last modified date in sitemap
  2. dateModified in structured data
  3. The etag header (used to indicate a specific version of the resource)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Which types of sites should worry about crawl budget?

A
  1. Large sites (1m+ pages)
  2. Medium or large sites (10,000+ pages) with very rapidly changing content
  3. Sites with very ‘flaky’ server setups (but then your issue is not really your site, but your servers)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

List two ways to control crawl budget for large sites with UGC

A
  1. Limit crawling in robots.txt to content that you know to be low quality
  2. No follow internal links
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Does crawl budget affect Google’s rendering of web pages?

A

Yes, because requests for rendering-related resources are counted in your crawl budget

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

How do you reduce the Google crawl rate?

A
  1. Set it manually usually Google Search Console
  2. Serve 500, 503, or 429s TEMPORARILY, as this will reduce Google’s crawl rate if they receive a lot of these response codes. If you do this for more than 1-2 days you may be dropped from the index
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

How can you verify that it is Googlebot crawling your website?

A

You can use a reverse DNS lookup, using the host command in terminal.

E.g. host 66.249.66.1

Returns

1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

Then, you must use a forward DNS lookup to verify that the domain name is indeed associated with the IP address you performed the reverse DNS on. This is to ensure that the original IP address is not DNS spoofing.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

What is the crawl capacity limit?

A

Googlebot wants to crawl your site without overwhelming your servers. To prevent this, Googlebot calculates a crawl capacity limit, which is the maximum number of simultaneous parallel connections that Googlebot can use to crawl a site, as well as the time delay between fetches. This is calculated to provide coverage of all your important content without overloading your servers.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

What are the ways that maximise the efficiency with which Google crawls a site?

A
  1. Manage URL inventory: consolidate duplicate content & block crawling of URLs using robots.txt (Gaston: use no follow)
  2. Return a 404 for permanently removed pages. Also manage soft 404s
  3. Keep up to date sitemaps
  4. Avoid long redirect chains
  5. Fast loading pages
  6. Adjust crawl settings
  7. Increase your server capacity
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

How do you figure out whether Google is encountering crawl availability issues?

A

Use the Crawl Stats report

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Broadly, what data is available in the Crawl Stats report?

A
  1. Overview: request number, bytes, response time
  2. Response codes
  3. File type
  4. Host status (robots.txt, dns, server availability)
  5. Googlebot type
  6. Purpose of crawling (discovery, refresh)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

What affects Google’s crawl rate?

A

Largely the responsiveness of your server, but also whether you limit it in search console

17
Q

Why would you want to limit Google’s crawl rate?

A

Because Google is overloading your servers with requests

18
Q

What is crawl budget?

A

The number of URLs that Google can (crawl rate) and wants (crawl demand) to crawl

19
Q

What are some ways to increase crawl budget?

A
  1. Faster servers
  2. Ensure crawl budget is not limited in Search Console settings
  3. Create valuable content that Google wants to crawl and that is popular with users (increases crawl demand)

Technical SEO (1 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2025 Bold Learning Solutions. Terms and Conditions