Thread

1. skybrian · 1789527919 · focus · HN ↗

   I didn't know websites could opt out of providing data to Google's AI training. Looks Google added support for this via 'Google-Extended' in robots.txt back in 2023:

   <a href="https:&#x2F;&#x2F;blog.google&#x2F;innovation-and-ai&#x2F;products&#x2F;an-update-on-web-publisher-controls&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.google&#x2F;innovation-and-ai&#x2F;products&#x2F;an-update-on-...

   1. qtqtqt · 1789569692 · focus · HN ↗

      doc: <a href="https:&#x2F;&#x2F;developers.google.com&#x2F;crawling&#x2F;docs&#x2F;crawlers-fetchers&#x2F;google-common-crawlers#google-extended" rel="nofollow">https:&#x2F;&#x2F;developers.google.com&#x2F;crawling&#x2F;docs&#x2F;crawlers-fetcher...

Unofficial Hacker News client; not affiliated with Y Combinator.