If your startup wants to show up in ChatGPT search, Gemini answers or other AI tools, your robots.txt file now does more work than it used to. AI companies publish separate crawler names for search, for model training and for user-triggered visits, and each one responds to its own rule. Blocking the wrong one can keep you out of AI answers, while allowing everything by default can send your content into training sets you never thought about.
This guide walks through the AI crawler names that matter most for a startup site in 2026, what each one controls according to the companies' own documentation, and a simple decision process for your team. It is written for founders and operators who own the website but don't have a full-time SEO person.
Why AI crawlers belong in your robots.txt plan
robots.txt has long been how sites talk to crawlers, and AI products now split crawling into separate jobs. One crawler may index pages so they can appear in AI search answers, another may collect pages for training future models, and a third may fetch a page because a user asked a question in a chat window. Your choices about each job can differ, and the right answer depends on whether your content is marketing you want everywhere or proprietary material you want to protect.
We ran an AI-readiness audit on our own site, and the report included a table of AI bots showing whether each one was allowed. Our PixyScan review walks through that test, and it is a good example of how fast you can check your current settings before changing anything.
OpenAI crawlers: OAI-SearchBot, GPTBot and ChatGPT-User
OpenAI documents its crawlers on its Overview of OpenAI Crawlers page, and the key point is that each setting is independent. OAI-SearchBot is used to surface websites in ChatGPT's search features. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and it recommends allowing OAI-SearchBot if you want to appear in results.
GPTBot is the training crawler. Per the same page, disallowing GPTBot indicates your content should not be used to train OpenAI's generative AI foundation models. That means a startup can allow OAI-SearchBot for visibility while disallowing GPTBot for training, and OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt update.
ChatGPT-User works differently. OpenAI says it visits pages when a user asks ChatGPT or a custom GPT a question, it is not used for automatic web crawling, and because these actions are user-initiated, robots.txt rules may not apply. The page also lists OAI-AdsBot, which only visits pages submitted as ads on ChatGPT; per OpenAI, the data it collects is not used to train foundation models.
Google-Extended and Gemini training
Google handles AI training through a control token instead of a separate crawler. According to Google's common crawlers documentation, Google-Extended is a standalone product token that lets publishers manage whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. It has no separate user agent string, since crawling happens with existing Google user agents.
The detail founders care about most is on the same page: Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. Google also notes it intends to support a shared machine-readable training preference signal called AIPREF in early 2027, so this area will keep changing and deserves a calendar reminder.
Where llms.txt fits
robots.txt tells crawlers what they may fetch. The llms.txt proposal, first published by Jeremy Howard on Sep 3, 2024 and last modified on Aug 10, 2026, takes on a different job. It proposes a standard /llms.txt file that provides information to help agents use a website. It is a proposal and an evolving convention, so treat it as a low-cost addition rather than a requirement.
For a startup, a short llms.txt that points to your product overview, pricing, docs and FAQ is usually enough. Keep it in sync with your sitemap, and update it when you ship a major page.
A simple robots.txt decision process for startups
Start by listing your content types. Marketing pages, blog posts, docs and pricing usually benefit from broad visibility, while gated content, customer data, internal tools and anything behind a login should already be blocked for every crawler, AI or not.
Next, decide on 2 separate questions. Do you want to appear in AI search answers? If yes, allow OAI-SearchBot and keep normal Google crawling open. Do you want your public content used for model training? If no, add disallow rules for GPTBot and Google-Extended. Some startups choose visibility and leave training allowed, since their blog exists to be found, but that is a business decision for your team.
Then check your work. Run an audit tool, read your live robots.txt in a browser, and confirm that the rules match what you decided. Our AI product trust checklist is a useful companion if you also publish AI features, because the same users who ask how you train on their data will ask how you handle their content.
Finally, write the decision down. Whether your team works from Palo Alto, Mountain View or Sunnyvale, site builders and agencies change over time, and a 1-page note on your crawler policy saves the next person from undoing it by accident. If you want to compare notes in person, Tribe organizes South Bay Renaissance to rebuild the startup and social scene of the South Bay and Peninsula, with founders' dinners, AI deep dives and demo nights.
Common mistakes to avoid
One mistake to avoid is a blanket block on every AI bot after reading a scary headline, which can remove you from AI search answers that send real visitors. Another is assuming ChatGPT-User follows the same rules as a crawler, when OpenAI says robots.txt may not apply to user-triggered visits. A third is forgetting that these documents change: OpenAI's page now lists 4 agents, and Google has already flagged a new signal for 2027.
For the bigger picture on AI this year, see our roundup of the biggest AI trends of 2026, and our best ChatGPT prompts guide is handy for drafting the llms.txt summary itself.
Tribe has hosted AI rooms like the Open Weights AI Summit with Aaron Levie in Redwood City on Sep 1, and more AI conversations are on the way.
Get help from people who ship
Crawler policy is one of those topics where a 10-minute conversation with someone who already set it up saves an afternoon. Follow the Tribe Events calendar on Luma for upcoming AI meetups, including South Bay Renaissance: Designing AI for humans in Redwood City on Nov 19, and join the AI Trends Tribe to ask questions any day of the week.
