Technical SEO

Crawl budget

Also known as: crawling budget · crawl capacity

How many pages of a website a crawler can and wants to retrieve in a given period.

A practical example

A large store limits vast numbers of low-value filter URLs.

What should I know?

It is not a fixed indexing allowance assigned to every small website.

Technical SEO in practice

Check crawler access, server responses, and content processing separately. Then assess indexing and search appearance. These stages are connected but distinct and need different diagnostic methods.

Related Semly features

See where this concept applies when working with Semly.

Google Search ConsoleConnect search data to Semly. Analyze queries and page performance when planning next steps.AI Agent LeonWork with Leon on recommendations, action plans, and brand content.

Related terms

CrawlingA program discovering and retrieving website resources, such as a search crawler.Faceted navigationFiltering a catalog by multiple properties, such as size, brand, and color.Server logsRequest records containing details such as time, URL, status, and client information.

Documentation and further reading

Definition prepared by Semly. Standards and behavior are described by their creators:

Find out if AI ChatGPTGeminiClaudeGoogle AI OverviewsGoogle AI Mode can see you

Enter your website address to get your AI visibility report.