# K&D Countertops — robots.txt
#
# This file COMPLEMENTS Cloudflare's Managed robots.txt (which is
# prepended automatically at the edge). Cloudflare handles the
# AI-training-bot blocklist and keeps it updated. This file adds:
#   1. Explicit allows for AI answer engines (so they can cite K&D
#      with links back — drives referral traffic)
#   2. Site-specific disallows (/api/, /_pagefind/, /preview/)
#   3. Sitemap pointer
#
# To see the full served file: https://countertopsstl.com/robots.txt
# To disable Cloudflare's managed prefix: Cloudflare dashboard →
# AI Crawl Control → Overview → toggle "Managed robots.txt" off.
#
# NOTE on /preview/: a bot only reads the block that names it specifically
# (falling back to `User-agent: *` only if no specific block exists) — so
# `Disallow: /preview/` has to be repeated in every named block below, not
# just the wildcard one, or the AI answer engines would still be "Allowed"
# to crawl unlisted draft previews. Each preview URL also carries an
# unguessable random token and serves `noindex`, so this is defense in
# depth, not the only thing stopping it.

# ── AI answer engines (explicitly allowed) ──────────────────────────
# These bots crawl to answer user questions with links back to K&D —
# they drive referral traffic, unlike pure training bots.
User-agent: OAI-SearchBot
Allow: /
Disallow: /preview/

User-agent: ChatGPT-User
Allow: /
Disallow: /preview/

# OpenAI's ads crawler — reviews the site and landing pages for the
# ChatGPT Ads account. Required by OpenAI for ad approval.
User-agent: OAI-AdsBot
Allow: /
Disallow: /preview/

User-agent: PerplexityBot
Allow: /
Disallow: /preview/

User-agent: Perplexity-User
Allow: /
Disallow: /preview/

# ── Site-specific rules for all other bots ──────────────────────────
User-agent: *
Disallow: /api/
Disallow: /_pagefind/
Disallow: /preview/

Sitemap: https://countertopsstl.com/sitemap.xml