Industry Report #005
Wide Open and Unread: Industrial America Crawls Easily and Answers Nothing
Where retail built a locked door, industrial America unlocked an empty room. The sector does not fail at the firewall; it fails at the citation layer, pairing pristine reachability with almost nothing an autonomous agent can ground an answer on. — The legibility gap
Fourteen Fortune 500 energy and manufacturing domains show the cleanest crawl mechanics in our index, yet score near zero on agent readiness. Where retail blocks automated access at the perimeter, industrial infrastructure invites crawlers into a press office and an ESG annex, leaving commercial and technical capabilities unrepresented.
TL;DR The fast read. Each line is measured below.
- 14/15 candidate sites crawled clean: the most open sector we have measured, the mirror image of retail's wall. B2B has no storefront to defend, and it shows.
- 19.6% of published content is link-invisible on average across the twelve measurable sites, reaching 41% and 61% on two supermajor energy and defense sites.
- 29.3% of pages cohort-wide are orphans with no inbound editorial link; on one supermajor's newsroom, 84% of pages.
- 3 answer-shaped annotations out of 23,823 pieces of structured data across all fourteen sites. Effectively zero, on the largest schema corpus in the series.
- 4.1% average agent-readiness score, the lowest of any industry measured. Twelve of fourteen sites score zero on all seven checks.
- 65.8% of one industrial conglomerate's pages sit four or more clicks deep (maximum depth: 11). Another site's deepest page is 18 clicks from home. Open doors, long hallways.
The Headline
After retail, this cohort was a relief to crawl, and that is the first finding. Fourteen of fifteen Fortune 500 energy and manufacturing candidates crawled clean on the first wave: no anti-bot walls, real sitemaps, static HTML, an average render dependency of just 2.3%. Business-to-business companies have no consumer storefront to defend, and their web infrastructure shows it. Where most of retail was a locked door, industrial America is an open one. The cohort is the largest in the series: 7,690 pages and 7.3 million tokens of public content across oil supermajors, defense primes, aerospace, heavy equipment, and diversified manufacturers.
Then comes the second finding. Behind the open door, less is wired for machines than in any sector we have measured. On the twelve sites where it can be counted, an average of 19.6% of published content is link-invisible: live, in the sitemap, and pointed at by nothing, rising to 41% on one energy supermajor and 61% on a defense prime. Cohort-wide, 29.3% of pages are orphans in the editorial graph. Across 23,823 structured-data annotations, the largest schema corpus in the series, exactly 3 are answer-shaped. And on the seven-point agent-readiness check, twelve of fourteen sites score zero, dragging the sector average to 4.1%, below healthcare, finance, technology, and even retail.
Retail’s problem is that agents cannot get in. Industrial’s problem is that agents get in easily and find a web that was never wired for them. Open is not the same as legible.
Three Tiers of Reachability
Every page falls into one of three reachability tiers, judged against the full link graph (navigation included) for reach and the content link graph for editorial authority:
- Reachable via editorial links. An in-content link path leads to the page. These are the pages a link-following agent finds on its own.
- Navigation-only. Reachable, but only through chrome: header, footer, or menu links that repeat site-wide. An agent that filters boilerplate, as most do, loses these.
- Link-invisible. Zero inbound links of any kind. The page exists as a URL and sits in the sitemap, but nothing on the site advertises it to a link-following crawler.
Two of the fourteen sites did not expose a usable sitemap, so their invisible tier cannot be counted (see Methodology). Averaged across the twelve measurable sites:
Two thirds of the industrial web is reachable the way an unaided agent reads a site, better than technology’s 58% or retail’s 62%. But the distribution has a heavy tail, and the tail is where the sector’s biggest names sit:
View data
| Site | Link-invisible page rate (% of site's published inventory) |
|---|---|
| M10 | 0.4% |
| M13 | 1.4% |
| M11 | 2% |
| M12 | 7.2% |
| M7 | 7.3% |
| M4 | 9.7% |
| M14 | 14% |
| M2 | 21.2% |
| M3 | 30% |
| M6 | 40% |
| M1 | 41.4% |
| M5 | 60.9% |
| Sample mean | 19.6% |
Half the cohort keeps invisibility under 10%, the best sustained performance in the series and proof that a large industrial site can stay wired. The other half climbs steeply. At the top sit a defense prime (M5) hiding 61% of a nearly thousand-page site, an energy supermajor (M1) hiding 41% of a newsroom-heavy archive, and a second defense prime (M6) hiding 40%. These are not small sites with thin content. They are the largest properties in the cohort, publishing at volume into a graph that no longer points at what they publish.
That is the commercial cost of the open door. An AI agent that walks in looking for capabilities, programs, or product truth finds live URLs the site itself never surfaces. The content exists; the citation path does not.
The Newsroom Machine
The pattern behind those extremes is the same publishing engine every time. On the two sites with a dedicated /newsroom path, that section absorbs on average 70% of all pages, the largest section share of any content type in any cohort so far, even though it appears on only those two properties. Press releases, feature stories, and update posts are stamped out on a template, linked once from a chronological index, and abandoned. The index rolls forward; the archive falls out of the graph.
The orphan rate shows what that template engine does to a site over time: across the cohort, an average of 29.3% of pages have zero inbound editorial links.
View data
| Site | Orphan rate (% of site's pages) |
|---|---|
| M10 | 0.4% |
| M13 | 1.4% |
| M9 | 1.7% |
| M11 | 3% |
| M8 | 5% |
| M12 | 5.6% |
| M4 | 9.3% |
| M14 | 17.2% |
| M3 | 30% |
| M5 | 30.1% |
| M7 | 63% |
| M6 | 65.6% |
| M1 | 84.2% |
| Sample mean | 29.3% |
The extreme, M1 at 84%, is an energy supermajor whose newsroom dominates its site: nearly nine hundred pages of genuinely substantial content (the site averages 659 words per page with near-perfect metadata), of which more than four in five have no editorial link pointing at them. It is the sharpest single case of the series-wide pattern: content operations that measure publishing volume and nothing downstream of it.
The navigation-only tier adds a second trap. One refiner (M7) routes 56% of its pages through navigation chrome alone, and a defense prime (M6) 34%, the same pattern as technology’s worst case, where a clean invisibility number hides a site an agent still cannot read once boilerplate is stripped.
Machine-Readability: The Same Switch, Flipped at Random
Structured-data coverage repeats retail’s bimodal switch at larger scale: half the cohort marks up everything, half marks up nothing.
View data
| Site | Pages carrying JSON-LD / schema.org structured data (%) |
|---|---|
| M2 | 0% |
| M10 | 0% |
| M12 | 0% |
| M14 | 0% |
| M4 | 0.1% |
| M8 | 0.6% |
| M13 | 0.7% |
| M5 | 68.7% |
| M9 | 73.3% |
| M1 | 97.7% |
| M6 | 98.9% |
| M3 | 99.2% |
| M11 | 99.3% |
| M7 | 100% |
| Sample mean | 45.6% |
Seven sites at or near zero, seven at or near one hundred percent, nothing in between. Schema here is a platform default, present or absent depending on which CMS the site runs, not a decision anyone made about machine consumption. Composition makes the point sharper than coverage ever could:
Of twenty-three thousand annotations, 86% are navigation and identity scaffolding. Exactly three annotations across the entire cohort are structured as direct answers that an AI agent could quote. Retail registered zero across a corpus a fifth of this size; industrial reproduces that same void at five times the scale. Two sectors in a row have now told machines exhaustively who they are and where to click, while offering nothing about what their content actually answers.
That vacuum carries a direct cost in synthetic search. When platforms such as ChatGPT Search, Perplexity, or Google AI Overviews synthesize answers regarding energy transition, defense programs, or equipment, unannotated pages force models to infer from unstructured body text. Hallucination risk escalates, and search engines routinely default to quoting trade publications or distributors rather than the authoritative corporate domain.
Agent-Readiness: Twelve Zeros
The agent-readiness score checks seven concrete affordances a site can offer an AI agent: llms.txt-style content signals, API catalogs, markdown content negotiation, agent skill manifests, and related machine-facing endpoints. These checks ask whether a domain actively invites machine interaction, or treats AI agents like legacy web browsers. Twelve of the fourteen sites pass none of them. The cohort average of 4.1% is the lowest of any industry in the series, below retail’s 10.4%, on a cohort whose sites are otherwise far healthier. The two non-zero scores (one defense prime at 43%, one refiner at 14%) trace to platform defaults rather than deliberate agent support. As in retail, three checks (content signal, API catalog, markdown negotiation) fail on every site without exception.
The juxtaposition is the sector in one line: the easiest industry to crawl is the least prepared for what crawls it.
Deep Trees and Crawl Budgets
Betweenness concentration (the share of a site’s structural load carried by its top 1% of pages) averages 39.9%, the gentlest of the series so far. The range still spans 10% to 83%, and the extreme is instructive: the same defense prime that hides 40% of its content (M6) also routes 83% of all betweenness through 1% of its pages, mostly careers and corporate hubs. Its structure is a bottleneck wrapped around a void.
This cohort also introduces a shape the series has not seen: the deep tree. One industrial conglomerate (M8) buries 66% of its pages four or more clicks from home, with a maximum depth of 11. A diversified manufacturer (M13) reaches 18 clicks at its deepest, with half its pages at depth four or beyond. Both keep orphan rates near zero (every page is linked), but linked in a chain an agent must walk link by link.
That chain is the problem. Autonomous agents and search crawlers operate on crawl budgets and depth ceilings, typically a few hops from the entry point, not eighteen. A page at depth 6+ can sit in an XML sitemap and still be functionally nonexistent to a real-time agent. Where the newsroom sites orphan their archives, the deep-tree sites bury theirs alive. The cohort’s average maximum depth of 6 is double retail’s. Openness at the front door does not mean the interior is navigable.
The Industry Scorecard
The five-lens analysis assigns each site a green, amber, or red score per lens. Across all fourteen:
| Site | Skeleton Size, density, and average path length. How big and how connected the site is at the body level. | Circulation PageRank distribution and structural bottlenecks. How importance flows between pages, and which hubs hold it all together. | Organs Community detection. Whether the site's topical clusters cleanly separate, or whether one mega-cluster dominates everything. | Health Islands, orphans, and dead-ends. Where content is structurally dying: unreachable, unlinked, or terminating. | Nervous Sys. Click depth, bridges, and cross-community linking. Whether the site is a well-designed building or a pile of disconnected rooms. |
|---|---|---|---|---|---|
| M1 | |||||
| M2 | |||||
| M3 | |||||
| M4 | |||||
| M5 | |||||
| M6 | |||||
| M7 | |||||
| M8 | |||||
| M9 | |||||
| M10 | |||||
| M11 | |||||
| M12 | |||||
| M13 | |||||
| M14 |
- Green: healthy
- Amber: moderate concern
- Red: critical
Thirteen of fourteen skeletons are green. These companies can build a site tree, no surprise for engineering firms. Health is red or amber on eleven of fourteen, continuing the series-wide pattern: the failure is never the frame, always the wiring. Circulation is the second-worst lens (three reds, eight ambers), driven by the newsroom hubs and careers sections that soak up authority. One site (M11, a mid-size energy independent) earns green on four of five lenses, proof that nothing about the sector requires the pattern.
Effort Without Payoff
The content itself is substantial. Typical industrial pages run 500–980 words, thin-content pages (under 200 words) sit at an identical 19.6%, and internal links dominate at 79.1%, the highest of the series. The biggest sites (supermajors, primes) are also among the most heavily written.
| Investment | What machines get |
|---|---|
| 7,690 pages / 7.3M tokens of body text | 19.6% link-invisible on measurable sites |
| Typical page: 500–980 words | 29.3% orphans cohort-wide |
| Static HTML, 2.3% render dependency | 86% of schema is navigation/identity scaffolding |
| Open doors (14/15 crawled clean) | 4.1% agent-readiness, twelve sites at zero |
That contrast is the point. Industrial firms are funding extensive editorial pipelines whose downstream distribution fails at the graph layer. The 61% invisible tier on M5 and the 84% orphan share on M1 are not thin boilerplate an agent can skip. They are the sector’s actual published record, dark.
The B2B Paradox: Press Office, ESG Annex, Empty Product Surface
Healthcare organized around the newsroom, finance around jurisdiction, technology around the product line, retail around the investor. Industrial organizes around the newsroom and the sustainability report. The recurring sections are the newsroom (70% of pages on the two sites with a dedicated /newsroom path), sustainability (five of fourteen sites, the most consistent non-news section in the series), careers, and investor relations.
The sector’s public web is a press office with an ESG annex: exactly what its stakeholders asked it to be, and exactly what an AI assistant asked “what does this company make, and should I buy from them?” cannot use. Product catalogs, spec sheets, and dealer content live in subsidiary sites, distributor portals, and PDFs outside these corporate graphs.
That is the B2B paradox. When a procurement officer or enterprise buyer prompts an AI search engine for a capability, a program, or a product class, the agent often bypasses the manufacturer’s corporate domain entirely. It finds PR and ESG copy where it needed specs, and the referral loop goes to distributors, aggregators, or competitors whose product surfaces are natively indexed. The companies that built the open doors surrender commercial authority in the product categories they dominate offline.
What This Means for AI Search Readiness
For an AI agent that discovers and quotes content by following links, three implications follow.
1. Crawlability is necessary and nowhere near sufficient. This cohort wins every access metric (crawl success, render independence, static HTML, sitemap coverage) and still delivers a fifth of its content into a void, because access and legibility are different investments and only one of them was ever made. The sector’s openness makes it the cheapest to fix and the easiest to audit. It does not make it ready.
2. The template architecture is the bottleneck. Every extreme in this cohort (84% orphaned, 61% invisible, 70% of pages concentrated in the newsroom) is the mechanical output of a CMS pipeline that links new content once from a chronological index and never links to it again. The solution is architectural rather than editorial: related-content links on the press-release layout, topic hubs that route back into the archive, and a de-orphaning pass across existing inventory.
3. Early movers establish default authority. Twelve of fourteen sites score zero on every agent affordance, and the entire sector shares just three answer-shaped annotations among twenty-four thousand. In a field this uniformly unprepared, the first supermajor or defense prime that structures its data and wires its archive becomes the primary citable authority for models querying energy transition, defense programs, or industrial specifications. The competitive threshold could hardly be lower.
The fix is structural, and every piece of it is measurable: re-link the newsroom archive into the editorial graph, flatten the deep trees, spread betweenness off the careers and corporate hubs, and write answer-shaped markup for the pages meant to be quoted. This is the entire premise of the Digital MRI service.
Methodology
Fourteen anonymized Fortune 500 energy and manufacturing sites (oil and gas supermajors and independents, defense primes, aerospace, heavy equipment, and diversified manufacturers) each run through the same pipeline: an HTTP-first adaptive crawl, main-content extraction (navigation, header, and footer links filtered out), dual-graph construction, and the five-lens topology analysis (Skeleton, Circulation, Organs, Health, Nervous System). Fifteen candidates were crawled; fourteen produced clean measurable graphs and one was excluded (crawler blocked). Three additional candidates were dropped at the pre-flight probe on hard anti-bot grounds. This is a curated cohort, not a random sample, so aggregate figures are exploratory benchmarks for F500 industrial topology rather than point estimates for the sector.
Reachability and invisibility use only the twelve sites whose crawl captured a usable sitemap; topology, content, and the scorecard use all fourteen; fragility uses the thirteen sites dense enough to compute. One site (M2, a heavy-equipment maker) resisted body-content extraction: its pages render text through client-side templates our extractor largely missed, so its word-count, orphan, and navigation-only figures reflect a near-empty content graph and are excluded from prose claims; its node/edge topology and scorecard are retained. Two sites initially produced zero-page crawls due to configuration issues (a subdomain redirect and an over-broad exclusion), were diagnosed, re-crawled clean, and are included; the failure modes and fixes are documented in the series’ public methodology notes.
The dual-graph model. Each page is classified against two graphs from the same crawl: the full graph (navigation included) for reach, and the content graph (in-content editorial links only) for authority. A page is reachable via editorial links if it has an inbound content link, navigation-only if reachable in the full graph but not the content graph, and link-invisible if nothing links to it in either, surviving only in the sitemap. This matches the model used in IR-002 through IR-004; per-site rates are not directly comparable across cohorts, but the shared finding now holds across five sectors: a measurable share of F500 content sits outside any link path an unaided agent can walk.
Disclaimers:
- Methods. PageRank, Louvain community detection, and betweenness centrality over crawled page structure.
- Robots & ethics. Site-level robots directives respected; disallowed pages never fetched. Publicly accessible page structure only. No content, metadata, or user data stored.
- Anonymization. Codenames M1-M14; the codename-to-domain mapping is intentionally not published. A site-level data appendix is available on request.
- Navigation exclusion. Any link target appearing on more than 80% of a site’s pages is treated as global navigation and dropped from the content graph; the full graph keeps it for reachability only.
- Token counts. Character-based estimates (page length divided by four), not tokenizer output.
- Betweenness. Reported as the share of total betweenness centrality carried by each site’s top 1% of pages; not computed for one site too sparse to support it.
- Scope. Statistical patterns for educational purposes only; not advice about any specific site or company.