Skip to content

$299 once Live in 7 business days

See pricing

Free kit · SquareRanked

The AI Crawler Unblock Kit

Whether ChatGPT, Perplexity or Claude can cite your Squarespace site starts with one question: are their bots allowed to read it? This kit answers that in about five minutes. Four parts, in order.

  1. The 5-minute Squarespace check
  2. The bot card: allow, block, or your call
  3. Test your live site with verify-bots.sh
  4. The 40-bot reference sheet
Download verify-bots.sh Download the bot sheet (CSV)

Part 1

The 5-minute Squarespace check

Squarespace doesn't give you a switch per bot. Under Settings → Crawlers there are two checkboxes, and between them they decide what your robots.txt tells every crawler.

  1. Log in to Squarespace and open Settings, then Crawlers.
  2. Look at Block search engine crawlers. It should be unchecked. If it's checked, Google and Bing can't index you at all, and nothing else in this kit matters until you fix it.
  3. Look at Block known artificial intelligence crawlers. New sites ship with it unchecked. If yours is checked (often after 2024 advice to "protect" content from AI), uncheck it if you want AI answers to cite you. One box covers the whole group of AI bots Squarespace lists, training and retrieval alike.
  4. Save.
  5. Open yoursite.com/robots.txt in a private window. With the AI box checked you'll see a Disallow: / block for each named AI bot. With it unchecked, those blocks are gone.

Two catches. Checking the AI box doesn't remove anything already collected, it only stops future crawling. And some retrieval bots aren't on Squarespace's list at all, so the script in Part 3 tests them directly. The full list of bots the checkbox controls is in our Squarespace AI bot list.

Part 2 · print it

The bot card

Most advice mixes up two kinds of AI bot. Retrieval bots fetch your page when someone asks an AI a question; they decide whether you get cited today. Training bots collect pages for the next model; blocking them doesn't remove you from today's answers. Decide on each group separately.

Allow: the 10 that decide search and citations

BotOperatorWhat it's for
GooglebotGoogleGoogle Search. Without it you aren't in Google at all.
BingbotMicrosoftBing, plus the index behind Copilot answers and DuckDuckGo results.
OAI-SearchBotOpenAIIndexes pages for ChatGPT Search. This is the one that decides whether ChatGPT can cite you.
ChatGPT-UserOpenAIFetches your page live when a ChatGPT user asks about it or clicks through.
PerplexityBotPerplexityIndexes pages so Perplexity can cite them.
Perplexity-UserPerplexityLive fetches during a Perplexity session.
Claude-SearchBotAnthropicIndexes pages for Claude's search results.
Claude-UserAnthropicFetches your page when a Claude user asks Claude to read it.
ApplebotAppleSiri, Spotlight and Safari Suggestions.
DuckDuckBotDuckDuckGoDuckDuckGo's own crawler.

Your call: training bots

We recommend allowing these if you want AI models to know your business over time. Blocking them is a defensible privacy choice.

BotOperatorWhat it's for
GPTBotOpenAICollects training data for future OpenAI models.
ClaudeBotAnthropicCollects training data for future Claude models.
Google-ExtendedGoogleToken that controls Gemini and Vertex AI training and grounding. Doesn't affect Google Search.
Applebot-ExtendedAppleOpt-out token for Apple Intelligence training. Doesn't affect Siri or Spotlight.
CCBotCommon CrawlBuilds the open dataset many AI labs train on.
Meta-ExternalAgentMetaCollects training data for Meta's Llama models.

Keep allowed: link previews

Block these and links to your site lose their preview card when shared: facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, WhatsApp, Pinterestbot.

Usually block

BotOperatorWhy
BytespiderByteDanceKnown for aggressive crawl rates. Allow it only if you want ByteDance products to surface your content.
DiffbotDiffbotCommercial knowledge-graph crawler that resells extracted web data.
OmgilibotWebz.ioCommercial web-data aggregator, same pattern as Diffbot.

SEO-tool crawlers (AhrefsBot, SemrushBot, DotBot, MJ12bot) are optional. Blocking them saves crawl load. It doesn't hide your backlinks, because those tools find links on the sites that link to you. Squarespace doesn't let you edit robots.txt per bot; the workarounds are in our robots.txt limits guide.

Part 3

Test your live site with verify-bots.sh

The script reads your robots.txt and requests your homepage as each of the 10 "allow" bots above, then prints PASS or BLOCKED for each one. It takes about a minute and changes nothing on your site.

  1. Download verify-bots.sh.
  2. Open Terminal (Mac or Linux). On Windows, use Git Bash or WSL.
  3. Go to the folder you saved it in (usually cd ~/Downloads) and run:
bash verify-bots.sh https://yoursite.com
  • PASS: robots.txt doesn't block the bot and your site answered it normally.
  • BLOCKED: robots.txt disallows it, or the server refused it. Go back to Part 1.
  • CHECK: the request timed out or errored. Run it again in a few minutes.

What it can't see: firewall rules that only fire on a bot's real IP address. A PASS means the door is open. It doesn't mean an AI engine will cite you.

Part 4

The 40-bot reference sheet

Every crawler you're likely to meet in a log or a robots.txt: who runs it, its user-agent string, what it does, our recommendation, and a link to the operator's own documentation. Opens in Excel, Numbers or Google Sheets.

Download the bot sheet (CSV)

What the kit doesn't do

Unblocking only means AI engines are allowed to read your site. Whether they cite you depends on what they find there: structured data that says what each page is, an llms.txt that maps the site, answers written so a model can quote them, and a real person behind the content. That's what the $299 install puts on your site.

Questions about anything in the kit? Email [email protected].