YouTube Video
Can you use ⛔ emoji to keep crawlers out? What about ASCII art chicken or non English characters in your robots.txt file? And can robots.txt tell AI bots to stay out of chicken's personal diary? Is chicken bot real?
Welcome back to the Non-Canonical Cabinet, where Cherry and Gary from Google Search Relations drag weird, edge-case technical questions asked by the community out of a dusty filing cabinet and onto your screen.
In this episode, they try to break their robots.txt file with 🐔:
🎨 ASCII art in robots.txt
🍽️ Picky parsers and what happens when you write custom, non-standard rules
🐔 User-agent tokens: Emojis and non-English characters? Is chicken bot real?
🔣 Emojis and non-English paths: Does /🐔-gear work? Should you percent-encode it?
🤖 Can robots.txt stop generative AI from training on chicken's secret diary?
TL;DR: Keep your robots.txt beautifully boring
📌 Timestamps:
00:00 – The sign in the Non-Canonical Cabinet
00:46 – What is robots.txt & RFC 9309?
02:10 – ASCII art in a robots.txt file?
03:13 – How parsers handle formatting mistakes and custom rules
04:04 – user-agent tokens: 🐔 and non-English characters?
04:54 – URL paths: 🐔 and non-English characters?
06:15 – Why percent-encode non-ASCII paths in robots.txt
07:06 – Generative AI & The New Frontier
08:26 – Recap
📚 Learn more:
Introduction to robots.txt: http://goo.gle/robotstxt
How Google interprets robots.txt: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec
List of Google's common crawlers: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
🗄️ Have a snippet of weird source code in your robots.txt? Got a weird question you want us to pull from the cabinet? Don't let your edge-case questions gather dust! Drop it in the comments below!
👻 Next episode: 404 status code!
Like. Subscribe. Keep it Canonical.
#GoogleSearch #TechnicalSEO #DevBytes #WebDevelopment #NonCanonical #SEO #SearchCentral #RobotsTxt
Welcome back to the Non-Canonical Cabinet, where Cherry and Gary from Google Search Relations drag weird, edge-case technical questions asked by the community out of a dusty filing cabinet and onto your screen.
In this episode, they try to break their robots.txt file with 🐔:
🎨 ASCII art in robots.txt
🍽️ Picky parsers and what happens when you write custom, non-standard rules
🐔 User-agent tokens: Emojis and non-English characters? Is chicken bot real?
🔣 Emojis and non-English paths: Does /🐔-gear work? Should you percent-encode it?
🤖 Can robots.txt stop generative AI from training on chicken's secret diary?
TL;DR: Keep your robots.txt beautifully boring
📌 Timestamps:
00:00 – The sign in the Non-Canonical Cabinet
00:46 – What is robots.txt & RFC 9309?
02:10 – ASCII art in a robots.txt file?
03:13 – How parsers handle formatting mistakes and custom rules
04:04 – user-agent tokens: 🐔 and non-English characters?
04:54 – URL paths: 🐔 and non-English characters?
06:15 – Why percent-encode non-ASCII paths in robots.txt
07:06 – Generative AI & The New Frontier
08:26 – Recap
📚 Learn more:
Introduction to robots.txt: http://goo.gle/robotstxt
How Google interprets robots.txt: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec
List of Google's common crawlers: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
🗄️ Have a snippet of weird source code in your robots.txt? Got a weird question you want us to pull from the cabinet? Don't let your edge-case questions gather dust! Drop it in the comments below!
👻 Next episode: 404 status code!
Like. Subscribe. Keep it Canonical.
#GoogleSearch #TechnicalSEO #DevBytes #WebDevelopment #NonCanonical #SEO #SearchCentral #RobotsTxt