=================================================================== Hellorf (www.zcool.com.cn) robots.txt Last updated: 2025-10-21 This file provides directives to search engine crawlers and other bots. Our goal is to ensure efficient indexing of our high-quality creative assets while protecting our crawl budget and preventing scraper abuse. =================================================================== --- 核心指令：Sitemap 索引文件入口 --- 这是给所有搜索引擎的最重要指令，指明了我们所有内容的地图。 Sitemap: https://www.hellorf.com/sitemap-list.xml Sitemap: https://www.hellorf.com/sitemap-tag.xml Sitemap: https://www.hellorf.com/sitemap-topic.xml =================================================================== Section 1: 对所有主流搜索引擎和AI爬虫的通用规则 我们采用“默认允许，按需禁止”的策略。 适用于 Googlebot, Bingbot, Baiduspider, Sogou Spider, 360Spider, YandexBot, DuckDuckBot, Google-Extended, ChatGPT-User, PerplexityBot 等。 =================================================================== User-agent: * --- 禁止抓取的部分 --- 1. 后台、管理和内部系统路径 Disallow: /login Disallow: /logout 2. 用户个人中心、购物车和订单流程 Disallow: /account 3. 站内搜索结果页 防止搜索引擎抓取无限的、低质量的搜索结果组合。 Disallow: /search/ Disallow: /search 4. 参数化URL、筛选、排序和分页 Disallow: /*?* # 5. 禁止访问特定的文件类型 Disallow: /*.php$ Disallow: /*.inc$ Disallow: /*.cgi$ Disallow: /*.pl$ Disallow: /*.zip$ Disallow: /*.rar$ Disallow: /*.pdf$ --- 明确允许的部分 --- Allow:/image Allow:/photo Allow:/template Allow:/music Allow:/help Allow:/contents Allow:/ip Allow:/font Allow:/video Allow:/subject Allow:/topic Allow:/image/ Allow:/photo/ Allow:/template/ Allow:/music/ Allow:/help/ Allow:/contents/ Allow:/ip/ Allow:/font/ Allow:/video/ Allow:/subject/ Allow:/topic/ ---明确不允许的部分--- User-agent: Googlebot-Image Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Mediapartners-Google Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Baiduspider Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Baiduspider-news Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Baiduspider-render Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Baiduspider-image Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Sogou Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Yahoo! Slurp Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: bingbot Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: 360Spider Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: HaosouSpider Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: yisouspider Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: YoudaoBot Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Sogou Orion spider Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Sogou News Spider Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Sogou blog Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Sogou spider2 Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Sogou web spider Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: EasouSpider Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: MSNBot Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: Yandex Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: ia_archiver Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: IPS-Agent Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search User-agent: BLEXBot Disallow: /login Disallow: /logout Disallow: /account Disallow: /search/ Disallow: /search =================================================================== Section 2: 防御性规则 - 限制已知的非必要爬虫和恶意爬虫 robots.txt 是一个“君子协定”。流氓爬虫和恶意脚本会完全忽略这些规则 这部分是第一道脆弱的防线。 =================================================================== # --- 屏蔽一些对 SEO 无益的 SEO 工具爬虫 --- User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: AhrefsBot Disallow: / --- 对一些行为尚可但过于频繁的爬虫设置抓取延迟 --- 注意：Googlebot 、百度、搜搜不遵守此指令，但 Bing, Yandex 等会遵守。 设置一个延迟可以减轻服务器压力。单位：秒。 User-agent: Bytespider Crawl-delay: 5 User-agent: PetalBot Crawl-delay: 5 User-agent: bingbot Crawl-delay: 5 User-agent: YandexBot Crawl-delay: 5 User-agent: Slurp Crawl-delay: 5 =================================================================== Section 3: 针对特定图片、视频爬虫的规则 (可选，通常*规则已足够) # 确保图片爬虫可以无障碍地访问所有内容。 #=================================================================== User-agent: Googlebot-Image Allow:/image Allow:/photo Allow:/ip Allow:/font Allow:/video Allow:/subject Allow:/topic Allow:/image/ Allow:/photo/ Allow:/ip/ Allow:/font/ Allow:/video/ Allow:/subject/ Allow:/topic/ User-agent: Baiduspider-Image Allow:/image Allow:/photo Allow:/ip Allow:/font Allow:/video Allow:/subject Allow:/topic Allow:/image/ Allow:/photo/ Allow:/ip/ Allow:/font/ Allow:/video/ Allow:/subject/ Allow:/topic/ User-agent: BingPreview Allow:/image Allow:/photo Allow:/ip Allow:/font Allow:/video Allow:/subject Allow:/topic Allow:/image/ Allow:/photo/ Allow:/ip/ Allow:/font/ Allow:/video/ Allow:/subject/ Allow:/topic/ User-agent: Pinterestbot Allow:/image Allow:/photo Allow:/ip Allow:/font Allow:/video Allow:/subject Allow:/topic Allow:/image/ Allow:/photo/ Allow:/ip/ Allow:/font/ Allow:/video/ Allow:/subject/ Allow:/topic/ User-agent: Googlebot-Video Allow:/image Allow:/photo Allow:/ip Allow:/font Allow:/video Allow:/subject Allow:/topic Allow:/image/ Allow:/photo/ Allow:/ip/ Allow:/font/ Allow:/video/ Allow:/subject/ Allow:/topic/ User-agent: Baiduspider-image Allow:/image Allow:/photo Allow:/ip Allow:/font Allow:/video Allow:/subject Allow:/topic Allow:/image/ Allow:/photo/ Allow:/ip/ Allow:/font/ Allow:/video/ Allow:/subject/ Allow:/topic/ # =================================================================== # Section 4: AI 模型训练爬虫规则 (保护策略) # # 我们欢迎AI用于搜索发现，但禁止将我们的版权内容用于模型训练。暂时禁止CCBot、AppleBot、Alexabot对SEO无较大影响 # These directives are intended to opt out of training for generative AI models. # =================================================================== User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: YisouSpider Allow: / User-agent: DeepseekSpider Allow: / User-agent: CCBot Disallow: / User-agent: Applebot # 苹果的规则比较模糊，保守起见可以先禁止 Disallow: / User-agent: Alexabot Disallow: /