GyaanSetu WebDev

Frontend, backend and devops for the modern web.

209 articlesDeep, practical knowledge

How to Stop Training Bots While Keeping Citation Bots for ChatGPT Traffic

A generic “Disallow: /” for GPTBot only blocks the training crawler; the citation bots that surface your pages in ChatGPT’s answer pane remain active. Use explicit robots.txt rules or Cloudflare’s AI Crawl Control to allow OAI-SearchBot, ChatGPT-User, and their peers while denying GPTBot, ClaudeBot,

WebDev · 4 min read

Cleanup Script Erased Code Snippets From Dozens of Articles

A routine maintenance job mis-identified Liquid’s {% raw %} tags, treating them as unclosed variables and stripping the surrounding code blocks from every article that used them, forcing an emergency rollback and a rethink of migration testing.

WebDev · 3 min read