How to Make Your Product Catalog Legible to AI Shopping Agents

Traffic to United States retail sites from generative AI tools rose 693.4% year over year between November 1 and December 31, 2025, measured across more than a trillion visits by Adobe Analytics. That number is usually quoted as the reason to start an AI readiness project. It is also routinely misread. It counts referral sessions: people who clicked a link inside a chat answer and landed on a retailer's site. It does not count agents buying anything.

This guide uses one working definition of legible, and it is deliberately narrow: a machine can fetch the product, parse it into typed fields, and trust that the price and stock status it reads are current. That is retrievability, not visibility. Whether a brand gets named in an AI answer is a measurement question, and AI visibility tracking tools exist to answer it. Whether the underlying product record can be fetched and relied on is a data question, and it is the one an operator can close this quarter.

The answer is deflating in a useful way. The machine-readable surfaces AI systems actually consume today are the boring, decade-old ones: schema.org `Product` and `Offer` markup, the Google Merchant Center feed specification, and server-rendered HTML sitting at a canonical URL. Google's own guidance for generative AI features states that you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Search, including its AI experiences. Making a catalog legible to an agent is almost exactly the same work as making it correct, and almost none of it is new spend.

Key Takeaways

• Adobe Analytics recorded a 693.4% year-over-year increase in generative AI referral traffic to US retail sites in November and December 2025, then a further 138% increase in May 2026, for cumulative growth of 1,324% since October 2024. Every one of those numbers counts referral sessions, not agent purchases.
• Salesforce, measuring more than 1.5 billion shoppers across more than 89 countries in the same November to December window, found shoppers arriving from AI search channels converted nine times more often than shoppers arriving from social media referrals. That is a channel-to-channel comparison, not proof that agents complete checkouts.
• Cloudflare reports that 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025, with mixed-purpose crawlers accounting for over 36%. Crawl volume measures supply-side interest, not shopper demand.
• Google's merchant listing structured data requires only `name`, `image` and `offers`, plus `price` and `priceCurrency` inside the offer. `availability` and `priceValidUntil` are merely recommended, which is precisely why they go stale on so many catalogs.
• Google Merchant Center requires `brand` for all new products except movies, books and musical recordings, and requires `item_group_id` for variants in the US, UK, Germany, France, Japan and Brazil.
• Google states that Search does not use `llms.txt`, and no major model provider has committed to reading it in production. It appears nowhere in this guide's checklist.

What the Demand Numbers Count, and What They Do Not

Three publishers produce citable data on this, and each measures something different. Reading them precisely is the difference between a funded project and a misdirected one.

Adobe measures referral sessions to retail sites

Adobe's holiday figure covers November 1 to December 31, 2025, and is drawn from more than one trillion visits to US retail sites, 100 million SKUs and 18 product categories. Beyond the headline growth rate, Adobe's data reported by Digital Commerce 360 shows AI-referred visitors converting 31% better than visitors from other sources, spending 45% more time on site, viewing 13% more pages per visit, and bouncing 33% less often. Revenue per visit from AI referrals was up 254% year over year.

The trend continued past the holidays. Adobe reported AI-referred retail traffic up 138% year over year in May 2026 and up 1,324% since October 2024, when it began tracking the channel. In the same analysis, Adobe scored how much page content its tooling could actually read: cosmetics pages scored 63%, electronics 56%, sporting goods and apparel 51%, grocery 48%, and furniture and home 47%. Even the leaders left roughly a third of high-value page content uncaptured.

What the Adobe numbers establish: a real and fast-growing referral channel, with engaged visitors, and a measurable gap between what merchants publish and what machines can read. What they do not establish: that anything is buying autonomously.

Salesforce measures influence, across a very large panel

Salesforce's 2025 holiday data covers more than 1.5 billion global shoppers across more than 89 countries for the same two-month window. Its widely repeated claim is that AI and agents drove 20% of all retail sales and $262 billion in revenue. Read the verb carefully: that is influenced or powered, a category that includes on-site recommendation engines, service chat and search assistants, not agents pressing buy on a shopper's behalf. The conversion comparison is more specific and more useful: AI search referrals converted nine times more often than social media referrals, which is a statement about two referral channels and nothing more.

Cloudflare measures crawlers, which are supply, not demand

Cloudflare's July 2026 analysis of agentic Internet traffic puts training at 52% of crawler requests as of June 2026, up from 22% in spring 2025, with mixed-use crawlers that blend search, agent use and training at over 36%. Pure search crawling is a small and shrinking share. More than half of all internet traffic is now non-human.

Cloudflare Radar also publishes a crawl-to-refer ratio, which it defines as HTML page requests from a platform's crawler user agents divided by HTML page requests carrying that platform's referrer header. For the week of June 19 to 26, 2025, Anthropic's ratio was roughly 70,900 crawls per referral. Two things follow. First, crawler volume is not an audience. Second, the retrieval that produces referrals is a much smaller slice of crawl traffic than the raw numbers suggest, which matters a great deal when deciding what to allow.

What Legible Means in Practice

A product record is legible when a retrieval system, acting on a shopper's question, can complete three steps without a human in the loop.

• Fetch. The product's facts are present in the initial HTML response at a stable, canonical URL, or in a feed the system already ingests. Nothing essential waits on a client-side render.
• Parse. Identity, price, currency, availability and variant dimensions are typed fields, not prose. `color: Navy` rather than a sentence mentioning navy.
• Trust. The price and stock status the machine reads match what a shopper sees at checkout, within minutes rather than days.

Most catalogs pass the first test, partially pass the second, and fail the third silently. The failures are specific and findable, which is the point of the next section.

Six Catalog Defects to Go Look For This Week

Each of these is a real defect with a check you can run against your own live site in a few minutes. Run them in order on your five highest-revenue product pages before doing anything else.

1. Offer markup that misstates price or availability

Google's merchant listing structured data reference requires only `name`, `image` and `offers` on the product, plus `price` and `priceCurrency` inside the offer. `availability`, `priceValidUntil`, `itemCondition`, `shippingDetails` and `hasMerchantReturnPolicy` are all recommended rather than required. Recommended fields are the ones that get hardcoded once in a template and never touched again. The two classic failures are `availability` pinned to `https://schema.org/InStock` for every product regardless of stock level, and a `priceValidUntil` date that expired eighteen months ago.

Check it yourself: run `curl -s <your-product-url> | grep -oE '"(availability|priceValidUntil|price)":\s*"[^"]*"'` against a product you know is out of stock. If it reports `InStock`, your markup is lying to every consumer that reads it, including Google's rich results and any retrieval system parsing your JSON-LD.

2. AggregateRating on products nobody has reviewed

Google's review snippet guidelines require `ratingValue` plus at least one of `ratingCount` or `reviewCount`, and require that the review content be readily visible to users on the marked-up page. Aggregating ratings from other websites is explicitly disallowed. Catalogs that emit `aggregateRating` from a template produce blocks like `ratingValue: 5, reviewCount: 0` on brand-new SKUs, or worse, a site-wide average stamped onto every product. Both are grounds for a manual action, and both teach a retrieval system that your rating data is noise.

Check it yourself: query your catalog for products where the review count is zero, then fetch three of those product pages and confirm no `aggregateRating` block is present in the response.

3. Feed rows failing on brand, GTIN or MPN

The Google Merchant Center product data specification requires `id`, `title`, `description`, `link`, `image_link`, `availability` and `price` for every product. `brand` is required for all new products except movies, books and musical recordings. `gtin` is labelled *It depends (strongly recommended if available)* rather than required, and `mpn` is required only for products without a manufacturer-assigned GTIN. Google's guidance is that products with missing or incorrect GTIN values may have limited visibility, so the failure is usually degraded eligibility rather than a hard rejection. Either way it does not fail loudly on your storefront. It fails quietly inside the feed.

Check it yourself: open the diagnostics view on your primary feed and sort disapprovals by count. The top three reasons are usually identifier problems, image problems, and price or availability mismatches against the landing page. Fix them in that order.

4. Variants distinguished only by free-text labels

This is the most common and most expensive defect, because it looks fine to a human. A size option value of `Lrg` on one product and `Large` on another, or a colour called `Blue-ish`, is legible to a shopper and useless to any consumer trying to group, filter or compare. Merchant Center requires `item_group_id` for variants in the US, UK, Germany, France, Japan and Brazil, and requires `color`, `size`, `material` or `pattern` where those are the dimensions that distinguish the variants. schema.org models the same idea through `ProductGroup`, with `productGroupID`, `hasVariant` and `variesBy` on the group declaring exactly which dimensions vary, and `isVariantOf` on each member product pointing back to it.

The discipline here is the opposite of the flexibility argument made for custom fields and arbitrary attributes. Internal-facing custom attributes benefit from being open-ended. Anything a machine is meant to group by needs a closed, enumerated value set with one canonical spelling per value.

Check it yourself: export the distinct values of every variant-generating option across your catalog and count them. If `Size` has 40 distinct values across 600 products, you have a taxonomy problem, not a size range.

5. One product, three URLs, no canonical

Category paths, tracking parameters, selected-variant query strings and locale prefixes routinely produce three or more addresses for the same product. Google's guidance on consolidating duplicate URLs explains that without an explicit canonical, Google picks a version itself, and signals for the product are split across addresses rather than consolidated. For retrieval systems the cost compounds: three near-identical records, potentially with three different cached prices, and no declared authority among them.

Check it yourself: append `?utm_source=test` to a product URL, fetch it, and check that the `rel="canonical"` in the response points to the clean URL, not to the parameterised one. Repeat for a variant-selected URL.

6. Product facts that exist only after JavaScript runs

This is a retrieval mechanism problem, distinct from page speed and unrelated to the ranking benefits usually claimed for Jamstack performance work. Google processes JavaScript in three phases: crawling, rendering, then indexing, with rendering deferred to a queue where a page "may stay for a few seconds, but it can take longer than that." Google's own advice is that server-side or pre-rendering remains a good idea because "not all bots can run JavaScript."

That last clause is the whole argument. A retrieval fetcher answering a shopper's question in real time is frequently a plain HTTP client, not a headless browser. If your price, stock status, variant matrix or specification table is injected after hydration, a fetcher that does not execute scripts sees an empty shell. This is also the defect most likely to be invisible internally, because every human who checks the page is using a browser.

Check it yourself: run `curl -s <your-product-url> | grep -c "<price string>"`. If the count is zero while the browser shows the price, the content is not in the initial response. The same test applies to stock status, the variant list and the specification table. Teams weighing this tradeoff at the architecture stage will find the rendering decision covered in more depth alongside custom storefront development.

The Feed Is the Most Widely Consumed Surface You Have

Structured data on the page is necessary. A feed is what actually gets ingested at scale, because it is pushed rather than discovered, it is typed by specification, and it can be refreshed many times a day without waiting on a crawl.

Google Merchant Center

Google's own AI optimization guidance names Merchant Center feeds as a way to make products visible in AI responses as well as ordinary Search results. Merchant Center can also find products automatically by reading structured data on your store pages to understand title, price and availability, and when the same product arrives both ways, the uploaded version wins and the auto-discovered one is removed. That precedence rule is the reason an accurate feed is the higher-leverage artifact.

The ChatGPT product feed

OpenAI publishes a product feed specification with nine required fields: `item_id`, `title`, `description`, `url`, `brand`, `seller_name`, `image_url`, `availability` and `price`. Availability is a closed enum of `in_stock`, `out_of_stock`, `pre_order`, `backorder` and `unknown`. Identity resolves through a GTIN with a valid check digit, or an MPN paired with brand. The guidance on freshness is direct: submit the current price, update the feed when a sale starts or ends, and submit current stock status with each row.

The overlap with Merchant Center is close to total. Any catalog that can already produce a clean Merchant Center feed can produce this one, which is the strongest practical argument for treating feed quality as the core investment rather than a channel-specific chore.

Keeping the feed and the page in agreement

A feed row that says `in_stock` at $79.99 while the landing page shows sold out at $94.00 is worse than no feed row. It produces disapprovals in Merchant Center, and it produces a shopper who arrives from an AI answer and immediately bounces. Both surfaces must derive from the same source of truth, on the same refresh cadence, and both must reflect the currency and price actually offered in the shopper's market, which is its own discipline for merchants running multiple currencies and localised prices.

Crawler Access Is a Tradeoff, Not a Best Practice

Most advice on AI crawlers picks a side. The honest position is that blocking has a real benefit and a real cost, and which dominates depends on your business.

Training crawlers and retrieval fetchers are different agents

The major operators publish separate user agents for separate purposes, and the distinction is the entire basis for a nuanced policy.

• OpenAI documents three: `GPTBot` for training data, `OAI-SearchBot` for ChatGPT search features, and `ChatGPT-User` for user-initiated fetches. Its bot documentation states that sites opting out of `OAI-SearchBot` "will not be shown in ChatGPT search answers, though can still appear as navigational links," and that for `ChatGPT-User`, "because these actions are initiated by a user, robots.txt rules may not apply."
• Google separates Search crawling from AI training. Google-Extended governs whether crawled content may train Gemini models and ground Gemini responses, and Google states it does not affect inclusion or ranking in Search.
• Anthropic runs distinct robots for training, search and user-directed retrieval, and documents how to set preferences per robot, including support for `Crawl-delay`.
• Perplexity separates `PerplexityBot`, its search crawler, from `Perplexity-User`, which it describes as visiting a page when a user asks Perplexity a question, and which is "not used for web crawling or to collect content for training AI foundation models." It also marks the limit of any robots.txt policy: Perplexity states that because a user requested the fetch, "this fetcher generally ignores robots.txt rules."

A blanket `Disallow: /` for every AI-associated agent blocks training and retrieval together. That removes you from the retrieval set that produces the referral traffic in the Adobe and Salesforce figures above. A per-agent policy lets you decline training while remaining fetchable for answers, which is the configuration most retailers actually want.

Compliance is voluntary

RFC 9309, published in September 2022, standardised how robots.txt is parsed and matched, and is explicit that the rules "are not a form of access authorization." Crawlers are requested to honour them. A robots.txt directive is a published preference, not a control. If a catalog genuinely must not be scraped, that requires authentication, rate limiting or WAF rules, and those are engineering projects with their own costs.

A decision framework instead of an answer

• Your product pages are your marketing. Allow retrieval and search agents, decline training agents, and monitor referral sessions by source. The referral upside is the whole reason the catalog work above is worth doing.
• Your content is the product you sell. Editorial, research, pattern libraries and similar assets have a defensible case for blocking training agents broadly, and often retrieval too, because a cited summary can substitute for the visit.
• You are under scraping pressure from competitors. robots.txt will not help. Rate limiting, bot management and authentication will. Treat these as separate problems with separate tools.
• You genuinely do not know. Allow retrieval agents, decline training agents, and revisit in a quarter with referral data in hand. This is reversible in a single file edit.

What Does Not Help Yet

Two things get recommended constantly and should not be on your list this quarter. Both deserve a plain statement rather than a hedge.

Agentic checkout protocols are not broadly deployed

The Agentic Commerce Protocol is an open standard for programmatic commerce flows between buyers, AI agents and businesses, published under the Apache 2.0 licence. The operative detail for an operator is on the protocol's own site: businesses wanting to participate through ChatGPT "need to apply." An application gate is the clearest available signal of where deployment actually stands. Building against a protocol you cannot yet be admitted to is not a use of this quarter's engineering time. If the vendor-side picture is what you need, the agentic commerce platform roundup covers who has shipped what.

Note also what the protocol does not remove. Product discovery in these systems still runs on the feed specification described above. The catalog work is a prerequisite for agentic checkout, not an alternative to it, which means doing it now is the correct sequence regardless of how the protocol question resolves.

llms.txt has no meaningful adoption

Google's AI optimization guidance states that Search does not use `llms.txt` and that you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Search or its generative AI features. No major model provider has committed to reading the file in production. A self-declared manifest of what a site considers important is, structurally, the kind of operator-controlled signal that consumers learn to discount. It is absent from every checklist in this guide, and that is deliberate.

The Reality Check

The unglamorous parts of this work are where projects stall, so they are worth naming before the plan is written.

Payments and checkout are a separate project

Nothing described here touches payments. Making a catalog retrievable does not create a path for an agent to transact, does not change your payment service provider relationship, and does not resolve who bears chargeback liability when a purchase originates from an automated intermediary. Those questions are open and they are not answered by feed hygiene. Any vendor claim that structured data makes a store agent-ready for transactions is conflating two projects.

Compliance and the rules on review markup

Review structured data is the part of this work with real downside risk. Marking up ratings that are not visible on the page, aggregating ratings collected elsewhere, or emitting rating blocks for products with no reviews all violate Google's published guidelines and can attract a manual action that removes rich results across the site. Price accuracy carries consumer-protection exposure in most markets, and a feed advertising a price the checkout will not honour is a regulatory problem before it is a technical one.

What this actually costs

The diagnostic pass is a day for one engineer with catalog access. Fixing template-level markup is typically a sprint. The expensive line is the variant taxonomy: normalising option values across a catalog of any size is manual data work, it cannot be fully automated because the correct value is a judgment call, and it touches merchandising as well as engineering. A 5,000-SKU catalog with an unmanaged option vocabulary is a multi-week project with a merchandiser in the loop. Budget for that rather than discovering it in week three.

What breaks

Three things, reliably: saved filters and ad campaigns keyed to old option strings, whichever URL ranked before canonical tags, and cache behaviour once product facts move server-side.

Choosing Where to Start

Sequence matters more than completeness. Pick the branch that matches your situation.

• Choose the rendering fix first if your curl test comes back empty for price or stock. Nothing downstream matters while the facts are absent from the response a non-browser fetcher receives. This is the only defect that can make every other fix invisible.
• Choose the feed first if your pages render server-side and your Merchant Center diagnostics show disapprovals. A feed is pushed rather than discovered, it is the surface with the most consumers, and fixing identifiers usually unblocks several hundred SKUs at once.
• Choose the variant taxonomy first if your product pages and feed are both healthy but shoppers cannot filter your catalog reliably on your own site. Filtering failure and machine-grouping failure have exactly the same root cause, and fixing it pays twice.
• Choose the crawler policy first if you are currently blocking AI agents wholesale and have no referral data. Switching to a per-agent policy takes minutes and starts a measurement clock that will inform everything else.
• Choose none of this yet if you are mid-replatform. Doing catalog normalisation once, on the destination data model, is far cheaper than doing it twice.

Where the Platform Fits

Almost everything above is platform-independent. It is a data-quality and rendering problem, and a merchant on any commerce backend can solve it with the team they already have. That is worth saying plainly, because the work is frequently sold as a platform decision when it is not one.

Swell's contribution to this specific problem is narrow. Its REST API describes its own data models, so the fields a feed row or a JSON-LD block requires can be read programmatically and kept in sync as the catalog changes rather than being hand-mapped and left to drift. And its product model holds options and variants as structured records with typed option values rather than free text, which is the condition a storefront needs in order to render a complete and correct product page. That is the extent of it. For the broader argument about a single typed backend behind multiple surfaces, the unified backend API data covers it separately.

A reader should finish this guide knowing exactly what is real. Structured data, feeds, canonical URLs, clean attribute taxonomies, accurate availability and a deliberate crawler policy are real, deployed and consumed at scale today. Agentic checkout protocols and `llms.txt` are not. The first list is entirely within your control and is the same list you would work through to make your catalog correct for humans, which is the most reassuring thing about it.

Frequently Asked Questions

Does adding structured data make my products appear in ChatGPT or Gemini?

Not directly, and no one can promise placement. Google's AI optimization guidance states that structured data is not required for its generative AI features and that no special schema.org markup exists for them, while still recommending structured data as part of ordinary SEO. What structured data reliably does is make your product facts parseable when a system chooses to read your page, and make them consistent with what your feed says. Treat it as removing a reason to be skipped, not as a placement mechanism.

Should I block AI crawlers in my robots.txt?

It depends on whether your product pages are marketing or inventory. Blocking training agents while allowing retrieval and search agents is the configuration most retailers want, because it declines model training without removing you from the set of sources that produce referral sessions. Remember that RFC 9309 is explicit that robots.txt is not access authorization, so compliance is voluntary and enforcement requires different tools.

Is a product feed worth maintaining if I do not run Shopping ads?

Yes, and increasingly so. Google's AI optimization guidance names Merchant Center feeds as a way to surface products in AI responses, and OpenAI publishes its own product feed specification with nine required fields that overlap heavily with Merchant Center's. The feed is the machine-readable surface with the most consumers and the fastest refresh cycle, and free listings do not require ad spend.

Do I have to move off client-side rendering?

Only for the content that has to be retrievable. Price, availability, variant options, specifications and the product description belong in the initial HTML response. Interactive elements, personalisation and recommendation carousels can stay client-side without penalty. The test is narrow: fetch the page without executing JavaScript and confirm the commercially load-bearing facts are present.

Does llms.txt help with AI visibility?

No. Google states that Search does not use it and that no new machine-readable files are needed to appear in Search or its AI features, and no major model provider has committed to reading it in production. Adding one will not harm a site, but it consumes time that the six defects listed above would repay. It is not included in any checklist here.

How do I measure whether this work paid off?

Use three separate measures rather than one. Server logs and CDN analytics show which AI agents fetch your product pages and how often. Referral sessions segmented by AI source show whether retrieval turns into visits. Feed diagnostics show how many SKUs are actually eligible in each surface. Brand mention tracking is a fourth, distinct question with its own category of tooling, and it should not be used as a proxy for whether your catalog is retrievable.

Should I wait for agentic checkout standards to settle before doing any of this?

No, because the sequence runs the other way. Product discovery in the agentic checkout specifications published so far depends on the same feed data and the same identifiers described in this guide. Clean catalog data is a prerequisite for those protocols rather than an alternative to them, so the work is not wasted under any outcome, and it produces returns in ordinary search and on-site filtering while the protocol question is open.