Google-Agent: The Web's New Visitor Just Got An Identity

Search Engine Journal covers Google’s addition of “Google-Agent” to its official web fetcher documentation, first used by Project Mariner. The key detail worth your attention: Google classifies it as user-triggered, meaning it ignores robots.txt — a different stance than OpenAI and Anthropic take with their equivalent agents.

What’s actually new

Google-Agent is the user agent string for AI systems on Google infrastructure that visit websites when a human asks them to — product research, form submissions, comparisons. Google categorizes it alongside tools like Google Read Aloud and Feedfetcher: because a person initiated the request, robots.txt rules are treated as non-applicable. This directly contrasts with ChatGPT-User and Claude-User, which do respect robots.txt. The second notable detail is that Google-Agent is experimenting with the web-bot-auth IETF draft protocol, using the identity https://agent.bot.goog to cryptographically sign requests — making agent identity verifiable in a way that user agent strings alone never could.

What it means for your config

This is less about application config files and more about server-level access control and infrastructure config. If you manage robots.txt, the takeaway is blunt: it won’t stop Google-Agent. For content you need to restrict from AI agents, you’ll need server-side authentication or access controls — the same mechanisms you’d use against a human visitor.

On the infrastructure side, if you manage CDN or WAF configs (Cloudflare, Akamai, etc.), check whether your rules are aggressively blocking non-browser user agents. Google publishes IP ranges for Google-Agent, and those need to be permitted if you want agent traffic to reach your server. The article notes that Akamai, Cloudflare, and Amazon already support web-bot-auth, so there may be config surface area emerging in those platforms for cryptographic bot verification — but specifics on how to configure that aren’t covered here.

If your site has multi-step forms, checkout flows, or booking processes that rely heavily on JavaScript patterns hostile to automation, those will fail silently for agent visitors. Semantic HTML and clear label associations aren’t just accessibility best practice anymore; they’re the difference between your forms working for this new class of visitor or not.

The announcement doesn’t detail any specific config file formats or migration paths — this is infrastructure-level awareness, not a schema change. We’ll revisit if Google publishes integration guides for web-bot-auth verification on common server stacks.

Recommended next step

Start by grepping your access logs for Google-Agent in the user agent string and cross-referencing against Google’s published IP ranges. That gives you a baseline for how much agent traffic you’re already seeing and which pages are being hit. From there, audit your WAF/CDN rules to make sure you’re making a deliberate allow-or-deny decision rather than accidentally blocking agent traffic as part of generic bot mitigation. If you serve content that truly needs restriction, begin planning a move from robots.txt-based controls to proper authentication — robots.txt was never a security boundary, and the agent era makes that gap visible.

---

Read the full announcement on Search Engine Journal → Google-Agent: The Web’s New Visitor Just Got An Identity