Description
AI Crawler Control shows which AI crawlers can read your WordPress site and names whatever blocks the rest. You can keep AI training crawlers out of your content and stay in ChatGPT search, Perplexity and other AI answers. It is free, works without an account, and never blocks Google, Bing or other search engines.
Robots.txt is the file AI crawlers check before they read a page, which makes it the first step of answer engine optimization (AEO) and generative engine optimization (GEO). It is also easy to get wrong. A plugin, a theme, your host or your CDN can add rules you never see, and one wrong line can remove you from AI search while doing nothing about AI training.
Free, no account needed
- AI crawler access check. Reads the robots.txt your site actually serves, the way crawlers receive it, and shows which of 44 known AI crawlers it lets in: AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot), AI assistants that fetch pages on request (ChatGPT-User, Claude-User, Perplexity-User) and AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot).
- Finds what blocks them. Names the plugin, theme or setting behind each blocking rule. It also flags a robots.txt file on the server that overrides WordPress, the « Discourage search engines » setting, Cloudflare’s managed robots.txt, and hosts or security plugins that can turn crawlers away before robots.txt is read.
- AI crawler rules with a preview. Block AI training and stay in AI search, block all AI crawlers, or choose crawlers one by one. You see the exact lines before saving, and you can undo. Search engines such as Google and Bing are never blocked from here.
- Content-Signal line (optional). Declares your AI training preference in robots.txt for crawlers that read the Content Signals convention.
- Rules for parts of your site. Keep AI training crawlers, or every AI crawler, out of sections such as /members/ or /downloads/ while the rest of the site stays open. The check marks crawlers kept out of a section as « Limited ».
- AI readiness in the editor. A panel in the block editor, or a box in the classic editor, scores each post for AI answers from 0 to 100: crawler access for that exact page, one clear H1, question-shaped headings, short answers under headings, tables and lists, schema markup, and title and meta description. Content checks update as you write; the full score runs after you save. It uses the same scoring as the Citlyze Chrome extension.
- AI suggestions, only when you ask. On WordPress 7.0 and later, with an AI provider set up in WordPress, the editor panel can suggest a question for a heading or draft a short answer under it. You read the suggestion first; nothing changes until you click Insert, and Undo takes it back out.
- Keep watching. Once a day the plugin checks the robots.txt crawlers receive. If something you did not choose starts blocking AI search crawlers, AI assistants or search engines, a notice appears in wp-admin and Site Health reports it. Email alerts are optional.
- llms.txt, handled honestly. Finds the llms.txt your site serves, whatever generates it, and checks it against the llmstxt.org format. If your site has none, the plugin can serve a simple one listing your pages and recent posts. If Yoast SEO, Rank Math or another plugin already generates one, it stays out of the way.
- Works with WordPress AI agents. On WordPress 6.9 and later, AI agents connected through the Abilities API and the WordPress MCP adapter can read your AI crawler access and llms.txt status. They cannot change anything.
Everything above runs inside your WordPress site and your browser, and nothing is sent to Citlyze. AI suggestions are the one exception, and only when you click: they go to the AI provider you set up in WordPress.
Block AI training, stay in AI search
OpenAI, Anthropic, Perplexity and others run separate crawlers for separate jobs. GPTBot collects training data. OAI-SearchBot builds the index ChatGPT search cites. ChatGPT-User fetches a page when someone asks ChatGPT about it. Blocking GPTBot keeps your content out of future model training. Blocking OAI-SearchBot removes you from ChatGPT search answers, which is rarely what a site owner wants.
The « Block AI training, stay in AI search » choice blocks training crawlers only. Google-Extended is a special case: it controls whether Gemini uses your content for training and for grounding answers in the Gemini app and Vertex AI. It has no effect on Google Search, AI Overviews or AI Mode.
Optional: connect Citlyze
Robots.txt says what crawlers may do, not what they actually do. With a Citlyze account, the plugin also reports AI crawler visits and visits from AI answers to your Citlyze workspace:
- Server-side AI bot detection that JavaScript analytics misses: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Bingbot, Amazonbot, Applebot, Google and Meta AI crawlers, and many more.
- Verified AI crawler analytics: Citlyze independently confirms that a request came from the bot it claims to be, so spoofed user agents never inflate your numbers.
- AI referral tracking: see when ChatGPT, Perplexity, Gemini, Copilot, Claude, Meta AI, Grok, DeepSeek or Mistral send real visitors to your pages.
- Web Bot Auth support: cryptographically signed AI agents are recognised and passed to Citlyze for verification.
- No impact on page speed: a matching request is noted in the site’s own database, and delivery happens after the response is finished or on a one-minute schedule. Nothing loads in visitors’ browsers, and the plugin sets no cookies.
- Exact status codes and redirects: counts show the real response (200, 301, 404, 410, 500), including requests another plugin redirects before WordPress renders a page, and where each redirect pointed.
- Reliable delivery: events are sent in signed batches and retried if Citlyze is briefly unreachable.
- Security-first settings: the signing secret is write-only and stored encrypted, with optional wp-config.php constants so it never touches the database.
Citlyze itself tracks how ChatGPT, Gemini, Perplexity and Google’s AI answers respond to the questions your customers ask, and whether they cite you.
A Citlyze account and a site key generated in Citlyze are required for tracking only. Tracking stays inactive until an administrator enters valid settings.
External services
The free tools in this plugin make no requests to Citlyze. They read your own site from your browser, such as its robots.txt file and the pages being checked.
Once a day, the crawler check also requests your site’s own robots.txt from your server. Alert emails, if you turn them on, are sent by your site through WordPress’s own mail.
AI suggestions in the editor use the AI provider set up in WordPress (WordPress 7.0 and later), and only when an author clicks « Suggest a question » or « Draft a short answer ». The heading and up to about 4,000 characters of the text under it then go to that provider through WordPress’s AI client, under that provider’s terms and privacy policy. Nothing goes to Citlyze. Without an AI provider set up in WordPress, the editor shows no suggestions and sends nothing.
The optional Citlyze connection connects to the Citlyze endpoint entered by the site administrator. It requires a Citlyze account and cannot provide tracking without that service.
For a request from a known AI crawler or signed agent, the plugin sends the visitor IP address, user agent, requested path, query string, HTTP status code, HTTP method, the request host, and for redirects the redirect destination (a same-site path, or only the domain for an external destination) and the name of the plugin that issued it. When Web Bot Auth is present, it also sends bounded Signature, Signature-Input, Signature-Agent, and safe original-request metadata so Citlyze can independently verify the signature. When a human visitor arrives from a supported AI answer engine, the plugin sends the referrer URL without its query string. The site key ID, a timestamp, a request signature, and a one-time nonce authenticate each batch. Once a day the plugin also downloads Citlyze’s current list of known crawlers, signed with your site key, so newly launched crawlers are recognised without a plugin update.
Data is sent only when the connection is configured and the request carries a recognised crawler user agent, contains bounded Web Bot Auth headers, comes from a supported AI-referrer host, or when an administrator explicitly sends a connection test. Requests are never sent for ordinary human page views, and a human visitor’s query string is never sent. Matching requests wait in a table in your WordPress database until delivery, at most one day. Citlyze uses the IP address and signature metadata transiently for verification and does not store them. Referrer URLs are not stored. Citlyze persists aggregate counters, and on plans that include the crawl log it also keeps a per-request record of verified crawler and signed agent visits (time, path, sanitized query string, status code, and redirect destination; never an IP address or user agent) for up to 30 days. Sanitized user-agent samples may be retained in bounded diagnostics for unrecognized bots.
Service terms: https://citlyze.com/terms-of-service
Privacy policy: https://citlyze.com/privacy-policy
Screenshots
Installation
- In WordPress, go to Plugins > Add New, search for « AI Crawler Control », then install and activate the plugin.
- Open Citlyze > AI crawler access. The check runs straight away.
- Optional: choose AI crawler rules, review the preview, and save.
- Optional, to track AI crawler visits: in Citlyze, open AI Crawlers > Install tracking and generate a site key. In WordPress, open Citlyze > Connect Citlyze, enter the tracker URL, key ID and signing secret, save, and use « Send test event » to verify the connection.
For security-conscious hosts, define CITLYZE_CRAWLER_SIGNING_KEY in wp-config.php. The plugin hashes it in memory and does not store it in the WordPress database. CITLYZE_CRAWLER_ENDPOINT and CITLYZE_CRAWLER_KEY_ID may be defined there as well.
FAQ
-
How do I block GPTBot without leaving ChatGPT search?
-
Open Citlyze > AI crawler access and choose « Block AI training, stay in AI search ». It blocks GPTBot and the other training crawlers and leaves OAI-SearchBot and ChatGPT-User alone, so ChatGPT search can still read and cite your pages. The preview shows the exact lines before you save.
-
Why can’t ChatGPT or Perplexity find my site?
-
Start with Citlyze > AI crawler access. It shows whether your robots.txt blocks OAI-SearchBot, PerplexityBot or the assistants that fetch pages, and which plugin, theme or setting wrote the rule. It also flags the « Discourage search engines » setting and hosts or firewalls that can block crawlers before robots.txt is read. Being readable is the first requirement; whether AI answers then cite you depends on your content and your competitors, which is what Citlyze measures.
-
Will this change my robots.txt?
-
Only when you choose AI crawler rules and save them. The plugin then adds a marked block of rules to the end of the robots.txt WordPress serves, after anything other plugins write. Until then it changes nothing, and « Undo last change » puts back your previous choice.
-
Can I block AI crawlers from part of my site only?
-
Yes. Under « Parts of your site », add a path such as /members/ and choose whether to keep out AI training crawlers or all AI crawlers. The rest of the site keeps your choice for the whole site. Crawlers that get rules for a section keep following your site’s other robots.txt rules too, because the plugin repeats them in their group.
-
Where do AI suggestions come from?
-
From the AI provider set up in WordPress 7.0 and later, such as OpenAI, Anthropic or Google, never from Citlyze. A suggestion is only requested when you click, you see it before anything changes, and Insert adds it as plain text that Undo removes. Without a provider set up in WordPress, the panel shows no suggestions.
-
Will the plugin tell me if AI crawlers get blocked later?
-
Yes. Once a day it checks the robots.txt crawlers receive. If another plugin, a theme, a file on the server or the « Discourage search engines » setting starts blocking AI search crawlers, AI assistants or search engines, you get a notice in wp-admin, a Site Health result, and, if you turn it on, an email. Rules you chose on the AI crawler access screen never raise an alert.
-
Does it work with Yoast SEO, Rank Math and All in One SEO?
-
Yes. Their robots.txt rules stay as they are, and the check names them when they block a crawler. The plugin’s own rules are added after theirs.
If one of them already generates llms.txt, this plugin checks that file instead of serving its own. -
My site has a robots.txt file on the server. What happens?
-
A robots.txt file on the server is sent directly by your web server, so WordPress and its plugins cannot add to it. The check reads that file, and the plugin shows the rules you chose so you can copy them into it.
-
Is a Citlyze account required?
-
No. The AI crawler tools work without an account. An account is only needed to track AI crawler visits and visits from AI answers in Citlyze.
-
Does the AI readiness score match the Citlyze Chrome extension?
-
Yes. The editor panel uses the same checks and scoring as the Citlyze Chrome extension and the free AEO grader on citlyze.com. A published page gets the same score on all three.
-
Does llms.txt work?
-
No major AI engine has documented reading llms.txt, and public log studies show little evidence that they request it. It costs nothing to have, so the plugin helps you keep a valid one, but robots.txt matters far more: it decides whether AI crawlers can read your site at all. Our read of the evidence: https://citlyze.com/blog/does-llms-txt-work
-
Does it slow my site down?
-
No. The checks run only on the plugin’s admin screens. On the public site, the plugin only adds its rules when robots.txt is requested, and, if you connect Citlyze, notes AI crawler requests without loading anything in visitors’ browsers.
-
With Citlyze: does tracking work with full-page caching or a CDN?
-
Only when the request reaches WordPress. A full-page cache or CDN can respond without running PHP, so those crawler visits are not visible to this plugin. The connection screen tells you when it detects a page cache. Use the Citlyze Cloudflare Worker, or upload your server logs in Citlyze, for sites where most traffic is served from a cache or the edge.
-
With Citlyze: what can tracking not see?
-
Anything that never reaches WordPress: redirects done by your web server or CDN (for example rules in .htaccess or nginx, or a host-level redirect to https or www), requests a firewall blocks before WordPress loads, and pages served from a full-page cache. The Citlyze Cloudflare Worker or a log upload covers those.
-
With Citlyze: my site is behind a proxy or load balancer. Does verification still work?
-
Citlyze confirms a crawler by its IP address, so the plugin needs the real visitor address. Behind Cloudflare this works automatically: the plugin trusts Cloudflare’s visitor header only for requests that come from Cloudflare’s own network. Behind another proxy, choose the header it sets and list its addresses under Citlyze > Connect Citlyze > Visitor IP behind a proxy.
-
With Citlyze: which crawlers are counted?
-
The plugin recognises crawlers that issue live page requests, including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, Google Gemini Notebook, Google-Agent, Atlassian Rovo, Amazonbot, Applebot, Bingbot, Meta crawlers, and others. The list is updated daily from Citlyze and with each release.
Citlyze independently confirms that a request genuinely came from the crawler it claims to be before counting it as verified. A user-agent string on its own is only a claim, so unconfirmed requests are reported separately as diagnostics and never inflate verified crawler totals.
-
Where does the data go?
-
The free tools send nothing anywhere. With the Citlyze connection configured, events go to the Citlyze endpoint entered in Citlyze > Connect Citlyze. See the External services section for the exact fields, timing, retention summary, and policy links.
-
How is the signing secret stored?
-
The secret you paste is never stored. It is reduced to a derived signing key immediately, and that key is encrypted using your site’s own WordPress salts before it is written to the database. The field is write-only: once saved, the value cannot be read back through the admin screen.
You can avoid database storage entirely by defining CITLYZE_CRAWLER_SIGNING_KEY in wp-config.php instead.
If a database backup leaks, rotate the site key in Citlyze under AI Crawlers > Install tracking. Rotation invalidates the old key immediately.
Reviews
There are no reviews for this plugin.
Contributors & Developers
“AI Crawler Control by Citlyze – robots.txt, llms.txt & AEO Audit” is open source software. The following people have contributed to this plugin.
Contributors
“AI Crawler Control by Citlyze – robots.txt, llms.txt & AEO Audit” has been translated into 1 locale. Thank you to the translators for their contributions.
Translate “AI Crawler Control by Citlyze – robots.txt, llms.txt & AEO Audit” into your language.
Interested in development?
Browse the code, check out the SVN repository, or subscribe to the development log by RSS.
Changelog
3.2.0
- New: llms.txt screen. Finds the llms.txt your site serves, whatever generates it, and checks it against the llmstxt.org format.
- New: serve a simple llms.txt listing your pages and recent posts when your site has none. It steps aside when another plugin or a file provides one.
- New: read-only abilities for AI agents on WordPress 6.9 and later: list AI crawlers, read AI crawler access, and read the llms.txt status.
- New: daily crawler check. A notice in wp-admin and a Site Health result when something you did not choose starts blocking AI search crawlers, AI assistants or search engines, or robots.txt stops loading. Optional alert emails.
- New: an « AI Crawler Control » section under Site Health > Info for support requests.
3.1.0
- New: AI readiness panel in the block editor and a box in the classic editor. Scores each post for AI answers with the same checks as the Citlyze Chrome extension.
- New: pre-publish AI readiness checks.
- New: AI crawler rules for parts of your site, for AI training crawlers or all AI crawlers. Groups repeat your site’s other robots.txt rules so nothing else changes for those crawlers.
- New: AI suggestions in the block editor, from the AI provider set up in WordPress 7.0 and later: a question for a heading, or a short answer under it, inserted only when you click.
3.0.0
- New: free AI crawler access check. Reads the robots.txt your site serves and shows which of 44 AI crawlers it lets in, grouped into AI search, AI assistants and AI training.
- New: names the plugin, theme, file or setting behind each blocking rule, and flags hosts, CDNs and security plugins that can block crawlers before robots.txt is read.
- New: AI crawler rules for robots.txt: block AI training and stay in AI search, block all AI crawlers, or choose crawlers one by one, with a preview and undo. Search engines are never blocked.
- New: optional Content-Signal line.
- New: Citlyze menu in wp-admin. The connection settings moved from Settings > AI Crawler Tracker to Citlyze > Connect Citlyze; old links redirect.
- Renamed to AI Crawler Control by Citlyze.
- Requires WordPress 6.4 or later.
2.3.0
- Reports the exact HTTP status code (301, 302, 404, 410, 500 and others) instead of only 404.
- Counts requests that are redirected before WordPress renders a page, such as redirects from the Redirection plugin, canonical redirects, and REST API requests, and records where each redirect pointed.
- Delivery is now queued in the site database and sent in signed batches after the response is finished, with retries, so a short outage no longer loses events and crawlers never wait for it.
- Recognises the real visitor IP behind Cloudflare automatically, and behind other proxies through a new trusted-proxy setting, so crawlers can be verified.
- Downloads Citlyze’s signed list of known crawlers daily, so new crawlers are recognised without waiting for a plugin release.
- Detects full-page caches and explains what the plugin cannot see.
- Requires the Citlyze beacon format introduced with this release.
2.2.0
- Hits reported by the plugin are now labeled with their own source in the Citlyze dashboard, so plugin-collected traffic can be told apart from other collectors.
2.1.0
- Expanded the AI answer-engine referrer list (Copilot, Duck.ai, Poe, Phind, HuggingChat and more).
- AI referrals are now also detected from utm_source values that AI engines append when no referrer is sent (for example ChatGPT’s utm_source=chatgpt.com).
- Reports the matched utm_source with referral events for stripped-referrer attribution.
2.0.0
- Renamed the plugin to AI Crawler Tracker by Citlyze.
- Added encrypted, write-only signing-secret storage and wp-config overrides.
- Enforced HTTPS endpoints and improved request-input sanitization.
- Added a signed, no-count connection test.
- Added privacy disclosures, translatable admin text, and cleanup on uninstall.
- Moved delivery to a stricter, signed request format.
- Added bounded Web Bot Auth signature metadata forwarding.
- Expanded crawler coverage while separating unconfirmed claims from verified totals.