Google open-sources its robots.txt parser and submits a proposal to IETF to make the Robots Exclusion Protocol an official standard
Code Issues 1 Pull requests 12 Projects Security Insights Google Webmaster Central Blog : A note on unsupported rules in robots.txt Roland Moore-Colyer / Inquirer : Google open sources robots.txt to curious webmasters Charlie Osborne / ZDNet : Google hopes to standardize robots.txt by going open source Thomas Claburn / The Register : Google open sources standardized code in bid to become Mr Robots.txt Ravie Lakshmanan / The Next Web : Google wants to make the 25-year-old robots.txt protocol an internet standard Barry Schwartz / Search Engine Roundtable : Google Shares Its Robots.txt Parser Code With Open Source World Mike Wheatley / SiliconANGLE : Google pushes for its robots.txt parser to become internet standard Jon Fingas / Engadget : Google pushes for an official web crawler standard Matt Southern / Search Engine Journal : Google Wants to Establish an Official Standard for Using Robots.txt Joe Fay / DEVCLASS : Google aims to standardise robots.txt, 25 years on Abner Li / 9to5Google : Google wants to make robots.txt an Internet standard after 25 years Barry Schwartz / Search Engine Land : Google posts draft to formalize Robots Exclusion Protocol Specification Tweets: David Humphrey / @humphd : Reading the code for https://opensource.googleblog.com/ ... on GitHub. It warms my heart to see all these spelling mistakes petrified into the parser: “disallow”, “dissallow”, “dissalow”, “disalow”, “diasllow”, “disallaw” The web is everything we make, mistakes and all https://github.com/... @googlewmc : In 25 years, robots.txt has been widely adopted- in fact over 500 million websites use it! While user-agent, disallow, and allow are the most popular lines in all robots.txt files, we've also seen rules that allowed Googlebot to “Learn Emotion” or “Assimilate The Pickled Pixie”. pic.twitter.com/tmCApqVesh Ilya Grigorik / @igrigorik : robots.txt was penned 25 years ago, is now used by 500M+ sites, and — finally — has an IETF spec: https://tools.ietf.org/... 🎉 Google Webmaster blog post @ https://webmasters.googleblog.com/ ... Bonus: Google's robots.txt parser is now open source! https://github.com/... 🎉 https://twitter.com/... Matt Holt / @mholt6 : Googlebot is very generous with how you spell “disallow” in your robots.txt: https://github.com/...