Technical SEO

Meta robots

Also known as: robots meta tag

An HTML tag giving crawlers supported instructions for indexing and displaying a page.

A practical example

An internal search page receives a noindex instruction.

What should I know?

Supported directives and interpretation depend on the system.

Technical SEO in practice

Check crawler access, server responses, and content processing separately. Then assess indexing and search appearance. These stages are connected but distinct and need different diagnostic methods.

Related Semly features

See where this concept applies when working with Semly.

Google Search ConsoleConnect search data to Semly. Analyze queries and page performance when planning next steps.AI Agent LeonWork with Leon on recommendations, action plans, and brand content.

Related terms

noindexAn instruction telling a supporting system not to include a page in search results.nofollowA rel attribute value describing a link relationship and signaling how search engines should consider it.X-Robots-TagAn HTTP response header providing crawler instructions, including for non-HTML resources.

Documentation and further reading

Definition prepared by Semly. Standards and behavior are described by their creators:

Find out if AI ChatGPTGeminiClaudeGoogle AI OverviewsGoogle AI Mode can see you

Enter your website address to get your AI visibility report.