# Search engines and AI assistants are welcome; AI training is not. User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / # AI training crawlers User-agent: Applebot-Extended User-agent: Bytespider User-agent: CCBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: cohere-ai User-agent: DeepseekBot User-agent: Diffbot User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: GoogleOther User-agent: GPTBot User-agent: meta-externalagent User-agent: MistralAI-Training User-agent: omgili User-agent: xAI-Bot User-agent: AwarioSmartBot User-agent: AwarioRssBot User-agent: Baiduspider User-agent: BorderxBot User-agent: CitibotSiteCrawler User-agent: Cotoyogi User-agent: FishBot User-agent: ICC-Crawler User-agent: KimiBot User-agent: magpie-crawler User-agent: NavuBot User-agent: PetalBot User-agent: QualifiedBot User-agent: SemrushBot-SWA User-agent: WARDBot User-agent: atlassian-bot Disallow: / User-agent: mj12 User-agent: Dotbot Disallow: / Sitemap: https://spotify-dedup.com/sitemap.xml