Content productionOrganic reachiGamingFor agenciesBlogAI wikiAbout Book a call

Google's AI Crawlers: Google-Extended, Google-CloudVertexBot, and Gemini

Google-Extended isn't a crawler, it's an opt-out token. How Google-Extended, Google-CloudVertexBot, Google-Agent, Googlebot and the coming AIPREF training signal relate to Gemini and AI features.

By Andrej Ruckij · · 5 min read

Google’s AI Crawlers: Google-Extended, Google-CloudVertexBot, and Gemini

By Andrej Ruckij · June 17, 2026 · updated November 4, 2026

TL;DR: Google’s main AI control, Google-Extended, doesn’t crawl anything. It’s a robots.txt token that tells Google not to use content Googlebot already fetched for training Gemini models or for grounding answers in Gemini Apps and the Vertex AI Gemini API. Disallowing it doesn’t affect Google Search, AI Overviews or AI Mode, which all run on Googlebot. Google-CloudVertexBot only crawls when a site owner asks it to, to build Vertex AI Agents, so it never cites you. Google-Agent acts for users and generally ignores robots.txt. And Google says it intends to support the AIPREF training signal (User-agent: __usage_ai-training) in early 2027.

Part of the AI crawler directory. Google is the vendor people misread most, because its primary AI control works unlike every other entry on the list.

Google-Extended is a token, not a crawler

Google-Extended makes no HTTP requests. It’s a robots.txt product token that tells Google how it may use what Googlebot fetched for Search: not for training future Gemini models, and not for grounding answers in Gemini Apps and the Vertex AI Gemini API. Google states that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”

User-agent: Google-Extended
Disallow: /

In practice:

  • It changes how your content is used, not whether it’s crawled. Googlebot still crawls and ranks you.
  • It doesn’t keep you out of AI Overviews or AI Mode. Those are Search features running on Googlebot, so only a Search-level control would remove you, and that removes you from Search too.
  • The cost: along with training, you opt out of grounding in Gemini Apps and Vertex AI, so Gemini is less likely to draw on your pages there.
  • There are no IP ranges to check, because nothing visits.

Google-CloudVertexBot crawls on the site owner’s request

Google-CloudVertexBot is a real crawler, but a narrow one. Google says its rules “affect crawls requested by the site owners’ for building Vertex AI Agents” and have “no effect on Google Search or other products.” It isn’t a search crawler and won’t cite you anywhere. Like Google’s other common crawlers, it crawls from the IP ranges in common-crawlers.json. Most site owners never need a rule for it.

Google-Agent, the user-triggered agent

Google-Agent was added to Google’s crawler documentation on March 20, 2026. Google describes it as “used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.” It sits with Google’s user-triggered fetchers, which Google says “generally ignore robots.txt rules”, because a person started the request. Its IP ranges are in user-triggered-agents.json, and Google is experimenting with Web Bot Auth signatures (https://agent.bot.goog) so you can verify it cryptographically. Treat it like other user-fetch agents and don’t block it by default.

AIPREF: a training signal Google plans to read

AIPREF is a shared, machine-readable way to state a training preference, written in robots.txt as User-agent: __usage_ai-training. Google says it “intends to support” it in early 2027. Until then, Google-Extended is the control Google actually honours. My read: once both work, a single AIPREF line could cover several vendors at once, but keep Google-Extended until Google confirms the switch.

  • To opt out of Gemini training and Gemini Apps / Vertex grounding: disallow Google-Extended. Search is unaffected, but so are AI Overviews and AI Mode.
  • Leave Googlebot allowed. Blocking it removes you from Google Search.
  • Google-CloudVertexBot: ignore unless you’re building a Vertex AI Agent over your own site.
  • Google-Agent: leave it alone. robots.txt won’t reliably stop it, and it acts for a real user. If you must control it, verify by IP range or signature and act at the firewall.

For the full cross-vendor file, use the template in block-or-allow-ai-crawlers. For how Gemini and other engines choose products, see how-do-ai-shopping-agents-work.

Key takeaways

  • Google-Extended is an opt-out token, not a crawler: no requests, no IP ranges.
  • Disallowing it opts you out of Gemini training and Gemini Apps / Vertex grounding, and doesn’t affect Google Search, AI Overviews or AI Mode.
  • Google-CloudVertexBot crawls only for site owners building Vertex AI Agents; it doesn’t cite you. IPs are in common-crawlers.json.
  • Google-Agent is one of Google’s user-triggered fetchers, which Google says generally ignore robots.txt; verify it by IP range or Web Bot Auth signature.
  • Google intends to support the AIPREF __usage_ai-training signal in early 2027.

Sources

#ai-crawlers#google#google-extended#gemini#robots-txt