Robots.txt for Large Websites: Crawl Budget, Sitemaps & Crawl Governance

For small websites, robots.txt is often treated as a one-off setup task. Someone adds a few lines, the site launches, and the file is barely thought about again. On larger platforms, that approach rarely holds up. Once a website starts producing filtered URLs, internal search pages, parameter combinations, staging paths, duplicate views, or JavaScript-driven route variations, crawl behaviour stops being a background detail and starts becoming an infrastructure concern.

At DBETA, we do not see robots.txt as a trick for “doing SEO”. We see it as one of the first layers of communication between a website and the systems trying to interpret it. That matters because modern visibility is shaped by more than rankings alone. Search engines, crawlers, and AI-driven systems all depend on being able to reach the right parts of a site without being dragged into structural noise. Good robots.txt decisions do not create authority on their own, but they do help protect the conditions that allow authority to build.

The problem is that robots.txt is still widely misunderstood. It is often used as a security measure, treated as a de-indexing tool, or overloaded with rules that belong somewhere else in the stack. On large websites, those misunderstandings become expensive. They waste crawl resources, slow down discovery of important content, and create messy situations where blocked URLs still surface in search results with weak or empty snippets.

The deeper issue is architectural. Robots.txt is useful, but it is not a substitute for sound structure. If a platform keeps generating low-value URLs faster than you can block them, the real problem is usually upstream. In practice, the strongest robots.txt files tend to belong to websites that already have clear content models, stable URL logic, sensible sitemap strategy, and a realistic understanding of which pages are worth crawl attention in the first place.

Robots.txt matters more when a site starts producing noise

Google’s own crawl budget guidance is quite clear on this point: most smaller sites do not need to obsess over crawl budget. The issue becomes meaningful when a site is very large, changes frequently, or creates enough low-value URL variation to waste crawler attention. That is the threshold where robots.txt becomes more than a housekeeping file. It becomes part of traffic control.

One of the patterns we often see is that growth creates crawl noise long before anyone notices visible failure. A site expands. Filters are added. Search functionality improves. Campaign parameters multiply. Preview URLs, account states, paginated views, and faceted combinations start to accumulate. Individually, none of these looks dramatic. Collectively, they create a broader crawl surface than the business ever intended. At that point, the site is no longer simply being crawled; it is being explored for meaning, and the wrong paths start competing with the right ones.

This has practical consequences. Google’s faceted navigation guidance explicitly warns that useless filtered URLs can lead to overcrawling, slower discovery of valuable pages, and unnecessary use of server resources. That is not just a technical inconvenience. For a business site, slower discovery means slower response to updates, weaker visibility for new or revised pages, and more friction between what the business publishes and what search systems actually prioritise.

What robots.txt actually controls, and what it does not

Robots.txt is a crawl control file. For Google, the supported fields are user-agent, allow, disallow, and sitemap. The file must live at the top level of the host it governs, it applies only to that protocol, host, and port, it must be UTF-8 plain text, and Google ignores anything beyond 500 KiB. Those details sound technical, but they matter on large sites because they define the real boundary of control. A robots.txt file on www does not govern shop, blog, a different protocol, or a non-standard port.

Just as importantly, robots.txt is not an indexing control. Google states this directly. A blocked URL can still appear in search if it is discovered elsewhere, and Google may show it without a snippet because it could not crawl the page content. That is one of the most common sources of confusion in technical SEO: teams block a URL in robots.txt and assume the page is safely out of view, when in reality they have only prevented crawling, not guaranteed removal from search.

This is where noindex and X-Robots-Tag matter. Google also makes clear that those directives are discovered when a page is crawled. So if a page is disallowed in robots.txt, Google may never see the noindex instruction at all. In other words, the combination many teams reach for first — “block it in robots.txt and also set noindex” — can quietly cancel itself out. If indexing rules must be followed, the URL must remain crawlable.

It is also not a security mechanism. Google warns that robots.txt instructions rely on crawler compliance, and RFC 9309 states this even more plainly: these rules are not a form of access authorisation. If something genuinely needs protection, the correct answer is authentication, permissions, or removal from the public web surface altogether. Listing a sensitive directory in robots.txt does not secure it. It announces it.

Multiple sitemaps in robots.txt: useful, but not a priority switch

Your impression data points to a very specific area of confusion: whether multiple sitemaps in robots.txt help Google prioritise crawling. This is where the topic needs more precision.

Google supports multiple sitemaps and sitemap index files. If a site exceeds 50,000 URLs or 50 MB per sitemap, it should be split across multiple files, and Google explicitly allows either submitting multiple sitemaps or using a sitemap index. It also says that adding a sitemap reference in robots.txt is simply one way to make Google aware of it. Submission is a hint, not a guarantee, and Google has no preference for one sitemap format over another.

That last point matters. Robots.txt can help Google discover sitemap locations, but it does not act as a crawl priority engine. Declaring five sitemap lines does not tell Google that the first one is “most important” or that one content type should outrank another in crawl scheduling. Google also says the order of URLs in a sitemap does not matter. So if a team is hoping to force crawl priority through sitemap order or robots.txt placement, they are solving the wrong problem.

In practice, multiple sitemaps are still valuable on large sites. They help separate content sets, reduce maintenance overhead, and make technical oversight easier. Splitting by content type, language, or update cadence can be sensible. It makes troubleshooting cleaner, supports clearer reporting in Search Console, and helps teams understand whether the site’s most important sections are being surfaced and refreshed properly. But that is operational clarity, not crawler coercion. The real signals still come from quality, internal linking, canonical consistency, crawl demand, and the wider clarity of the platform.

From our perspective, this is a good example of the wider DBETA point of view. Businesses often look for a control that feels simple because the underlying issue is complex. Multiple sitemaps can absolutely be part of a strong large-site setup. They just cannot compensate for weak structure, unclear canonical logic, or low-value URL inventory.

The real crawl-budget win is usually waste reduction

When large sites struggle with crawl efficiency, the strongest gains rarely come from adding more directives. They usually come from removing waste.

Google’s faceted navigation documentation is direct on this: if you do not need filtered URLs to appear in search, prevent them from being crawled. It even provides examples using robots.txt patterns for parameter-based combinations and recommends allowing crawling of core listing pages while restricting low-value filter variations. This is one of the clearest legitimate uses of robots.txt on large commercial sites.

The same logic applies to internal search results, tracking parameters, and utility views that create novelty without adding search value. In practice, one of the patterns we see is that businesses underestimate how quickly these low-value paths accumulate. What starts as harmless convenience for users becomes a sizeable crawler footprint over time. That is why robots.txt works best when it reflects a deliberate inventory decision: which URLs support discovery, and which ones are simply operational by-products of the platform.

This is also where JavaScript-heavy sites can get into trouble. Google can process JavaScript, but it still documents limitations and warns about search issues caused by blocked resources, too many resources, slow or very large assets, and rendering problems. So the question is not whether JavaScript is “SEO-friendly” in some abstract sense. The better question is whether the implementation creates unnecessary crawl and rendering friction. Large JavaScript frameworks can become crawl-efficient, but only when route logic, parameter handling, internal linking, and resource delivery are governed carefully.

That is why we would always be cautious about blocking CSS or JavaScript broadly. Google says you can block unimportant resource files if the page remains understandable without them, but if blocking those resources makes the page harder to interpret, it will hurt Google’s ability to analyse it properly. On modern sites, especially those that rely on front-end rendering, blocking the wrong resources is an easy way to create technical blindness by accident.

Why crawl-delay is usually the wrong answer

Another one of your impression clusters points to crawl-delay, which tells us the article needs to make one correction very clearly.

Google does not support the crawl-delay rule for Googlebot. It has said this for years, and its current documentation still treats crawl-delay as an unsupported rule for Google Search, even though some other crawlers may choose to support it. So if the problem you are trying to solve is Googlebot overcrawling, adding crawl-delay to robots.txt will not fix it.

When Googlebot genuinely is too aggressive for the available serving capacity, Google’s own troubleshooting guidance points elsewhere. It recommends diagnosing the problem through server monitoring and crawl data, improving crawl efficiency, increasing capacity where justified, and, in emergencies, temporarily returning 503 or 429 status codes while the server is overloaded. That is a very different mindset from “let’s put a delay in robots.txt and hope the problem goes away”. It treats excessive crawling as an infrastructure issue, not a syntax issue.

This distinction matters strategically. A technically valid choice is not always a strategically sensible one, and crawl-delay is a good example. On the surface it looks neat and controllable. In practice, for Google, it is not the control you think it is. On large sites, that kind of misunderstanding can send teams off into file edits when the real answer sits in URL inventory, rendering load, or server capacity planning.

Website builders and CMSs change where crawl control lives

The query set also suggests readers want help with implementation, not just theory. That makes sense, because robots.txt is not managed the same way on every platform.

Google’s documentation says that if you use a CMS such as Wix or Blogger, you may not need to — or even be able to — edit robots.txt directly. Instead, the platform may expose a search settings page or another control mechanism that decides whether pages are crawlable or visible to search engines. The same sitemap guidance also notes that many CMSs automatically generate sitemaps.

For businesses, the practical lesson is this: the real question is not “do we have a robots.txt file?” The real question is “where does crawl control actually live on this platform, and how does it interact with sitemaps, canonicals, rendering, and page-level visibility settings?” Managed systems often abstract the file away, but they do not remove the architectural consequences. They just move the control surface.

In practice, that means the same principle still applies. You need to know which URLs the platform creates, which ones matter, which ones should remain crawlable, and which ones are simply side effects of how the builder or CMS works. Strong governance still matters, even when the interface makes the file itself less visible.

Robots.txt now has an AI crawler layer as well

Traditional crawl governance is still the foundation, but it is no longer the whole story. Public websites are now also being requested by AI-related crawlers and product tokens, which means robots.txt increasingly sits inside a wider conversation about AI visibility, content use, and machine-readable governance.

OpenAI’s documentation is explicit here. It uses separate robots.txt controls for OAI-SearchBot and GPTBot, and those controls are independent. A site can allow OAI-SearchBot in order to appear in ChatGPT search results while disallowing GPTBot to signal that content should not be used for training OpenAI’s generative foundation models. OpenAI also says that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Google has created a similar distinction with Google-Extended. Its current crawler documentation says Google-Extended is a standalone product token publishers can use to manage whether content crawled from their sites may be used for future Gemini training and for grounding in Gemini-related products. It also states that Google-Extended does not affect a site’s inclusion in Google Search and is not used as a ranking signal.

This is important, but it should be framed properly. For most businesses, AI crawler rules should not take priority over basic crawl governance. If the main site architecture is messy, the site has bigger problems than whether a training bot is allowed. But once the fundamentals are in place, these newer controls become part of a more mature governance model. They let businesses make deliberate decisions about discovery and data use instead of treating all crawler access as one undifferentiated category.

What good robots.txt practice looks like on a large website

The strongest robots.txt setups are rarely the most complicated. They are the ones that reflect a well-governed platform.

What that usually looks like in practice is fairly straightforward:

• a file at the correct root for each relevant host or subdomain
• supported directives only, written clearly and kept maintainable
• low-value parameter spaces blocked where they do not need search visibility
• render-critical resources left crawlable
• sitemap references used for discovery, not mistaken for crawl priority controls
• page-level index controls handled with noindex or X-Robots-Tag where required
• regular review after major platform changes, migrations, or CMS behaviour changes

From our experience, the hardest part is not writing the file. It is deciding what the site is actually trying to be. If a website is treated as infrastructure, those decisions become clearer. You know which URLs support discovery, which ones are operational only, and which ones create drag. If the site is treated as a loose collection of pages and tools, robots.txt often ends up carrying too much weight because nobody has defined the system properly elsewhere.

Final thoughts

On large websites, robots.txt is not a trick, a security measure, or a substitute for architecture. It is a crawl governance layer. Its job is to help machines spend time where the business wants visibility to accumulate, and to keep them away from paths that create little value and plenty of noise.

That means a few conclusions are worth keeping in mind.

Multiple sitemap lines can help discovery, but they do not force crawl priority. crawl-delay may exist in the wider ecosystem, but it is not a Google solution. Robots.txt can restrict crawling, but it cannot guarantee removal from search. And once a site grows large enough, the best robots.txt decisions are usually the ones made in service of a wider architectural strategy rather than in response to isolated symptoms.

At DBETA, we believe that is the more useful way to think about the file. Not as a checklist item, but as part of how a website explains itself. On the modern web, that explanation has to work for both people and machines. The businesses that do this well usually make the same move elsewhere too: they stop treating the site as a surface and start treating it as an operational system with real structural consequences.