DFIELDSOLUTIONS

Web and delivery

GlossaryCrawl budget

How much of your site Googlebot is willing to fetch — only a constraint on large sites, but a real one there.

Googlebot fetches with finite patience: how many URLs it will crawl per visit depends on server speed and how much the site is judged worth fetching. Small sites never touch the limit — a few hundred pages all get seen. It becomes a constraint on sites with thousands of URLs, where crawl capacity spent on parameter URLs, faceted pages, tag archives and redirect chains is capacity not spent on pages that matter.

The maintenance is housekeeping: let robots.txt and canonicals keep junk URLs out, keep internal links pointing at final destinations instead of redirecting hops, keep the sitemap clean of 404s, and keep the server fast — crawl rate adapts to a site that answers quickly.

Related terms

The bench this belongs to

Web performance

A rebuild that loads on a phone on mobile data, in the country your customers are actually in. Core Web Vitals are the measurement, revenue is the point.

All termsStart a conversationMarkdown version

DField Bt. · Dunakeszi · dezso@dfieldsolutions.com
5.0
“From LinkedIn DM to live site. Two tiny tweaks, then shipped.”Michael J Ringer · Vilya ProtectionFounder · Spain