User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

# The Content-Signal line above is Cloudflare's declaration format, and it says out
# loud what the Allow rules below already do: this site is happy to be indexed, happy
# to be used as grounding for an AI answer, and happy to be trained on. All three read
# yes because every crawler in every tier is allowed, and a declaration that argued
# with its own Allow rules would be worse than none. It is a statement of intent
# rather than a ranking signal, and Google does not read it.

# No Disallow rules on purpose. /inc/ answers 403 and /send.php answers 405 to
# GET with X-Robots-Tag: noindex, so crawlers drop them on their own. Blocking
# them here only made Google file them under "Blocked by robots.txt".

# The AI crawlers are named explicitly below. Functionally this changes nothing:
# the wildcard above already allows them and nothing here is disallowed. It is
# spelled out because this firm sells GEO, robots.txt is the first file a
# technically literate reader checks, and "allowed by omission" reads as an
# oversight where an explicit Allow reads as a decision. Listed one agent per
# group so a future block or crawl-delay can target a single crawler without
# disturbing the others.

# OpenAI
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

# Anthropic
User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Anthropic-AI
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Common Crawl — the corpus many models are trained from
User-agent: CCBot
Allow: /

# Apple
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

# ByteDance
User-agent: Bytespider
Allow: /

# Cohere
User-agent: cohere-ai
Allow: /

# Google: Google-Extended is a control token, not a crawler. It covers Gemini
# training and grounding in Gemini Apps. AI Overviews and AI Mode follow Googlebot.
User-agent: Google-Extended
Allow: /

User-agent: AdsBot-Google
Allow: /

User-agent: AdsBot-Google-Mobile
Allow: /

Sitemap: https://concept211.com/sitemap.xml