Executive summary
On 8 September 2026 we pointed an automated reader at 20 real B2B catalogs. Only 5 of them served a product page it could read.
We expected to find gatekeeping. We found the opposite. Of the 13 robots.txt files we could read, zero blocked any of the 12 AI agents we checked. Nobody was shutting the door on purpose.
The refusals happened somewhere else entirely. Seven of the 20 would not answer an automated request at all. Of the 13 that did answer, 7 exposed no product URL a machine could find, and 1 returned an error on the product page itself.
So the published policy says come in. The plumbing says no. That is a very different problem, and a much more fixable one.
Introduction
Fifteen of twenty B2B catalogs gave an automated reader nothing usable at product level. That is the finding. We ran it ourselves, today, and you can re-run it.
No survey. No vendor report. Two scripts, 20 named distributors and manufacturers, one product page each.
Here is why we bothered. Buyers are increasingly sending software ahead of themselves, which is what agentic commerce means in practice. That software reads whatever your site hands it and scores you on that alone.
It does not call to clarify. It does not fill in your form. If it cannot find your part number, your price, or your stock position, you are simply not on the short list.
We assumed the industry was quietly locking AI crawlers out while legal figured out a position. The doors were unlocked. There was just nothing behind most of them a machine could read.
What did we test, and how?
Twenty public B2B catalogs, one product page each, on 8 September 2026. Every number in this post comes from that run.
Here is the full list, so you can check our work: Grainger, McMaster-Carr, Fastenal, MSC Industrial, Motion Industries, Applied Industrial, Zoro, Global Industrial, DigiKey, Mouser, Newark, Allied Electronics, Ferguson, Wesco, Border States, Rexel USA, Hisco, Kaman, Genuine Parts, and Sonepar. All are real distributors or manufacturers. None is a competitor of ours.
The method was deliberately boring:
- One request per host. We asked for robots.txt, then for a single product page. No crawling, no load.
- The site chose the product page. We took it from the company’s own declared sitemap, or failing that from a product link on its own homepage.
- We used a normal desktop browser User-Agent. This is the friendly version of the test. A crawler identifying honestly as GPTBot or ClaudeBot would very likely do worse, not better.
- JavaScript was not executed. Some AI crawlers render pages. Others do not. What we measured is what a non-rendering retrieval client receives.
- n = 20, one page each. This is an audit, not a statistical sample. We report counts, never percentages of the industry.
The scripts are re-runnable and the results are a snapshot dated 8 September 2026. Sites change. Ours might have changed by the time you read this, and so might theirs.
Is anyone blocking AI agents on purpose?
No. Not one of them.
Robots.txt is the plain text file at the root of a website that tells automated readers which parts they may fetch. It is the closest thing the web has to a published door policy.
Of the 13 catalogs whose robots.txt we could read, zero blocked any of the 12 AI agents we tested. That list included GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Amazonbot, and Bytespider.
Not a single Disallow rule aimed at any of them. Whatever your board thinks the industry position is on AI crawlers, the published position is open.
The other 7 robots.txt files could not be read at all. Which is a finding of its own.
Then why did 7 of 20 refuse the connection?
Bot protection at the edge, almost certainly. Not policy.
Seven hosts would not answer a plain automated request at all. This happened before any question of content, markup, or product data. The connection itself was the problem.
| What happened | Count | What it means |
|---|---|---|
| HTTP 403, refused | 5 of 20 | Something in front of the site decided the request was not a person |
| Connection timed out | 2 of 20 | No answer at all within the wait window |
| Answered the connection | 13 of 20 | Got as far as the content question |
Remember the User-Agent detail. We were presenting as an ordinary desktop browser and still got shut out. An agent that identified itself honestly would have a harder time, not an easier one.
This is the gap that should worry you most, because it is invisible from the inside. The policy layer says yes. The edge layer says no. The person who wrote your robots.txt is rarely the person who configured your bot protection, and neither one is checking the other’s work.
How many catalogs gave an agent a product page it could read?
Five of twenty.
Work through the 13 that answered. Seven exposed no product URL at all that could be found in their sitemap or in their homepage HTML. A sitemap is the machine-readable index of a site’s pages, and if products are missing from it, an agent has no path in.
In most of those cases the category menus and product links are built by JavaScript after the page loads. A person sees a full catalog. A non-rendering client sees a shell with no way through to a product.
One more returned an error on the product page itself, after serving robots.txt without complaint. So the door opened, then the room was locked.
That leaves 5 of 20 catalogs that put a readable product page in front of an automated reader. If you want the mechanics of what happens next, we have written up how AI product search works once the data is actually reachable.
What was on the five pages that did answer?
Less than you would guess. And the gaps were not deliberate.
Structured data is a labelled copy of the facts on a page, written for machines, so software does not have to guess which number on screen is the price. Schema.org is the shared vocabulary most search engines and AI systems use for those labels.
| Catalog | Visible words | Structured data | Price in structured data | Availability |
|---|---|---|---|---|
| Kaman | 1,721 | Yes, not Product schema | No | No |
| Global Industrial | 784 | Yes, not Product schema | Yes | Yes |
| Zoro | 689 | Yes, full Product schema | Yes | Yes |
| Unnamed catalog 1 | 115 | None | No | No |
| Unnamed catalog 2 | 75 | None | No | No |
Two of the five are unnamed on purpose. Their pages are built by JavaScript, so an agent that renders may well see a full product page where our non-rendering client saw a stub. Naming them would be a claim our method cannot support.
Across all five: schema.org Product markup on 1 of 5. Price in structured data on 2 of 5. Availability on 2 of 5. A manufacturer part number or GTIN in structured data on 2 of 5.
The bright spots deserve names. Zoro was the only site of the twenty with full Product structured data. Global Industrial published price and availability in structured data and backed it with an 18-row spec table. Kaman served the deepest readable page in the set at 1,721 words, with a 25-row spec table.
Now the detail that reframes everything. Zero of the five put a login or a quote wall in the way. Nothing was being withheld as strategy. The data simply was not in the HTML.
And two of the five returned under 200 visible words. A buyer sees a full page. A retrieval client sees a headline and a menu.
What this audit does not prove
Plenty. We would rather say it than have you find it.
Twenty hosts and one product page each is an audit, not a survey of the industry. Treat every number here as “5 of 20”, never as a percentage of B2B distribution.
We also cannot tell you what any of this costs anyone. No revenue figure, no conversion figure, and no lost-deal estimate can be honestly derived from a set of 20 page fetches. Anyone selling you one from data like this is guessing.
And JavaScript rendering is a real variable, not a technicality. Some agents render. Some do not. A catalog that fails our test may pass for a different reader tomorrow.
What should you check on your own catalog this week?
Six things. None of them needs a project, and most take under an hour.
- Read your own product page with JavaScript disabled. Or just view the page source. If the price, specs, and stock line are not in the text, an agent may never see them.
- Ask whoever runs your bot protection what it does to non-browser traffic. Your robots.txt can say yes while your edge quietly says no. Nobody finds this out by accident.
- Open your sitemap and search it for a product URL. If products are not listed, an agent has no route to them.
- Confirm price and availability exist as labelled data, not only as styled text inside a widget that loads late.
- Check that part numbers and cross-references are on the page as text. A buyer holding a rival’s number should still land on yours.
- Count how much sits behind a login. Not one of our five readable pages used a wall, so the assumption that everybody gates this data is worth re-testing.
If the failure is in your source data rather than your site, start upstream. Our ERP export self-check covers the file itself: field integrity, duplicate SKUs, units of measure, price tiers. This post covers the website. They are two different surfaces and both can fail independently.
For the wider business case, we have argued elsewhere why machine-readable catalogs are becoming a commercial requirement rather than an IT preference.
Now the part we should say plainly, because we sell something in this market. ChatSKU does not fix what this audit measured. Adding an assistant to a page does not make that page readable to a crawler that never runs your JavaScript. If your product page returns 75 words to a machine today, it returns 75 words tomorrow with a widget on it.
The six checks above are the fix for that, and most of them belong to whoever owns your front end and your bot protection. Server-rendered facts, labelled price and availability, a sitemap that lists products, and an edge policy that matches your robots.txt.
What we do is the next moment, not that one. Once a buyer or an assistant is on your site, ChatSKU answers product questions from the catalog files your team already maintains, without a replatform. The two problems are worth separating: one decides whether you make the short list, the other decides what happens when someone arrives. If you are weighing them, our B2B catalog chatbot guide and the ROI calculator are the places to start. Because it was built for distributors and wholesalers, buyer-specific pricing tiers are native rather than bolted on, and the full list of ChatSKU’s catalog integrations is here.
People also ask
Does blocking AI crawlers protect my catalog data?
It might protect it from being copied. It will not protect your position on a short list. In our run, blocking was not the reason 15 of 20 catalogs came back empty. Broken access and missing labels were.
Is a 403 error to an AI agent a security setting or a mistake?
Usually neither, exactly. It is bot protection doing the job it was configured to do, before anyone considered that some bots are now buyers. It is a settings conversation, not a rebuild.
If my site works fine for Google, will it work for AI agents?
Not necessarily. Google has rendered JavaScript for years and has a crawl budget you cannot buy. A retrieval client fetching one page in real time has neither.
How many products need to be readable before this matters?
Start with the ones you actually want to sell. Your top movers, your stocked lines, and anything with a cross-reference. Fixing 200 pages that earn revenue beats fixing 40,000 that do not.
Conclusion
Twenty catalogs. Not one blocking AI agents by published policy. Five with a product page a machine could read.
The gap is not permission. It is plumbing and labels, and both of those are yours to fix.
So run the test yourself this week. Open your best-selling product page with JavaScript disabled, and read what is left on the screen. That is your catalog as software sees it.
If what is left is a headline and a menu, you already know what to do first.
Frequently asked questions
Two things. Whether robots.txt allowed 12 named AI agents, and what a single product page returned to a non-rendering client. One request per host, on 8 September 2026.
Because 20 hosts is not a sample of an industry. “5 of 20” is a fact about 20 catalogs. “25% of B2B distributors” would be a claim we cannot support, and you should distrust anyone who makes it from a set this size.
For some sites, yes. Two of the five readable pages would very likely look much fuller. The 7 refused connections and the 7 with no findable product URL would not improve, because those failed before rendering was ever a question.
It is a labelled copy of the facts already on your page, written in a format software can read without guessing which number is the price and which is the part number.
They cover different moments. An assistant answers a buyer who has arrived. Structured data helps software decide whether to arrive at all. Most catalogs benefit from both, and the second one is cheaper than it sounds.
Yes, and you should. Fetch your robots.txt in a browser, then view the page source on a product page and search it for your price and stock text. That is the entire test in five minutes.
Turn your catalog into a 24/7 sales channel
Stop sending PDFs. Start capturing demand.
Try ChatSKU Free →