Every AI Bot on Squarespace's Block List, Named and Sourced

The list comes from Squarespace's own documentation¹ and is the verbatim set of user-agents the platform writes into robots.txt when the box is checked. Most of these are training crawlers. A few are bulk web-corpus collectors. None of them are the live retrieval bots that decide whether ChatGPT, Claude, or Perplexity will cite your page when a user is actively asking. The rest of this article groups the 26 by owner so you can decide each one on its merits, then names the retrieval bots Squarespace did not cover at all.

What the list is, and what it is not

26

named user-agents disallowed when the Squarespace AI block is on.

→ Squarespace Help · 2026

~75%

of the list are training crawlers; the rest are bulk web-corpus or aggregator bots.

→ Squarespace Help · 2026

6+

live retrieval user-agents from major AI vendors are NOT on Squarespace's list.

→ OpenAI Docs · 2026

GPTBot. OpenAI describes GPTBot as the crawler that "collects publicly available web content to train future generations of OpenAI's models"². Blocking GPTBot in robots.txt opts you out of training. It has no effect on live ChatGPT citations. On Squarespace's list.

ChatGPT-User. The user-initiated retrieval agent. OpenAI documents it as the user-agent that "may visit a web page to help answer" when a user asks ChatGPT a question that requires up-to-date information². Allowing it is the difference between being citable and not being citable in live ChatGPT answers. Not on Squarespace's list.

OAI-SearchBot. The ChatGPT Search index crawler. OpenAI documents it as "used to link to and surface websites inside AI-driven search results" and explicitly "not used to crawl content to train OpenAI's generative AI foundation models"². Required for inclusion in the ChatGPT Search results surface. Not on Squarespace's list.

ClaudeBot. The training crawler. Anthropic's documentation states that blocking ClaudeBot in robots.txt will "exclude your site's future content from AI training datasets"³. On Squarespace's list as ClaudeBot (and separately as the older anthropic-ai token).

Claude-User. The retrieval crawler. Anthropic documents it as the user-agent that "retrieves content when a user asks Claude a question that requires access to a webpage"³. Blocking Claude-User means Anthropic cannot fetch your pages in response to user queries. Not on Squarespace's list.

Claude-SearchBot. The search index crawler. Anthropic documents it as the user-agent that "indexes content to improve Claude's search answers"³. Required if you want Claude's search answer feature to surface your pages. Not on Squarespace's list.

claude-code. The user-agent sent by Anthropic's Claude Code CLI when a developer asks the assistant to fetch a URL via the WebFetch tool. Volume is low compared with the other three. Not on Squarespace's list.

Google-Extended. The robots.txt token that controls whether Google uses your content to train and ground Gemini models. Google's documentation states: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity"⁶. The same docs confirm: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

GoogleOther. A family of three crawlers (GoogleOther, GoogleOther-Image, GoogleOther-Video) used for internal Google product development, research, and one-off projects outside of Google Search. All three appear on Squarespace's list.

Applebot. Apple's primary web crawler. Apple's documentation states it powers "the search technology that is integrated into many user experiences in Apple's ecosystem including Spotlight, Siri, and Safari"⁵. Not on Squarespace's list, which means the search-engine checkbox in the Crawlers panel governs it (allowed by default).

Applebot-Extended. The AI training opt-out signal. Apple is explicit: "Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results"⁵. Blocking it opts your content out of training Apple Intelligence; it does not affect Apple search visibility. On Squarespace's list.

Meta-ExternalAgent. Meta's AI training crawler. Documented user-agent string: meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler). Meta states the crawler collects publicly available content to train Llama family models. On Squarespace's list.

FacebookBot. Meta's older crawler, used for Meta product surfaces and previously for Facebook's training pipelines before Meta-ExternalAgent. On Squarespace's list. Distinct from facebookexternalhit, which is the link-preview fetcher that scrapes Open Graph tags when a URL is shared; that one is not affected by Squarespace's AI toggle.

PerplexityBot. Perplexity's index crawler. Documented as "designed to surface and link websites in search results on Perplexity" and "not used to crawl content for AI foundation models"⁴. User-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot).

Perplexity-User. The live retrieval agent. Documented as the user-agent activated when "users ask Perplexity a question, it might visit a web page to help provide an accurate answer"⁴. The docs note Perplexity-User "generally ignores robots.txt rules" because a user initiated the request. User-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user).

MistralAI-User. Documented as the user-agent that "may visit a web page to help answer and include a link to the source in its response" when a user asks Le Chat a question⁷. User-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots). Mistral's docs state the agent honours robots.txt directives, X-Robots-Tag headers, and cache headers.

CCBot (Common Crawl). The largest open web-corpus collector. CCBot identifies itself as CCBot/2.0 (https://commoncrawl.org/faq/) and Common Crawl documents it as respecting robots.txt⁸. Many academic and commercial training datasets are derived from Common Crawl, so blocking CCBot has compound downstream effects.

Bytespider (ByteDance). User-agent string includes Bytespider with a feedback contact at bytedance.com. Widely reported as ignoring robots.txt in practice, so the Squarespace block on this bot is best understood as a polite request rather than an enforced rule. TikTokSpider is ByteDance's companion crawler and behaves similarly.

cohere-ai and cohere-training-data-crawler. Cohere's two documented user-agents. The training-data-crawler is the bulk collector; cohere-ai is the on-demand retrieval agent for Cohere products. Both are on Squarespace's list, which is unusual for a retrieval agent.

AI2Bot and Ai2Bot-Dolma. The Allen Institute for AI's research crawlers. Dolma is the name of AI2's open-source training corpus, so Ai2Bot-Dolma is the crawler that feeds it.

Amazonbot, DuckAssistBot, YouBot, Quora-Bot. Smaller AI-product crawlers from Amazon, DuckDuckGo (DuckAssist), You.com, and Quora. Each one has the standard training-or-retrieval question; the Squarespace toggle blocks them all at once.

aiHitBot, anthropic-ai, img2dataset, MyCentralAIScraperBot, omgili, omgilibot. Long-tail entries. anthropic-ai is the older Anthropic user-agent superseded by ClaudeBot. img2dataset is an open-source bulk image dataset builder. omgili and omgilibot are forum and aggregator crawlers that pre-date the current AI wave but ended up on the list because their corpora feed AI training pipelines.

For a Squarespace owner whose goal is AI citations, this gap is the entire point. Toggling the AI block on or off does nothing to the bots that decide whether ChatGPT, Claude, Perplexity, and Mistral will pull your page into their next answer. Those decisions are governed by the platform's default robots.txt rules, by any X-Robots-Tag headers your site emits, and by the live behaviour of each bot.

The retrieval agents not on the list, as documented by their respective vendors: ChatGPT-User², OAI-SearchBot², Claude-User³, Claude-SearchBot³, Perplexity-User⁴, MistralAI-User⁷, Applebot⁵. By default, all seven are allowed on a Squarespace site because Squarespace's robots.txt does not disallow them. That is the configuration most AI-visibility playbooks recommend, and it is what a fresh Squarespace site already has.

For per-bot control beyond the Squarespace UI, the workarounds are documented in the robots-txt-custom leaf. For the recommended setting per bot, the training-vs-retrieval leaf covers the full matrix.