The Wall Goes Up

The last three briefs looked at what machines read and quote. This one is about the doors being closed in their faces, including ours.

Twice this year we set out to crawl a consumer-facing Fortune 500 sector the way a link-following AI agent would: politely, honoring robots directives, reading public pages, no tricks. Twice the sector’s biggest names returned essentially nothing.

What we actually hit

In our retail cohort, ten of the sixteen consumer domains we probed failed a basic crawlability check, returning anti-bot challenge pages, hard 403 walls, or blank inventory shells before a single page could be measured. Several of the survivors passed the pre-flight probe and then blocked the full crawl anyway. Across three waves of attempts, not one consumer storefront produced a measurable site. The only properties that could be read were standalone corporate and investor-relations domains: small sites, managed by different teams, lacking the storefront’s perimeter defenses.

In telecom and media, the pattern repeated with greater severity. Major carriers, cable operators, and studios presented hard walls at both their consumer and corporate domains; one entertainment giant advertised a two-thousand-URL sitemap and returned nothing measurable from it. What survived was the sector’s back office, corporate media archives, tower REITs, and two carrier surfaces that leaked through, both exhibiting extreme structural disrepair.

The wall is rational and priced for the wrong decade

None of this is mysterious. Commercial anti-bot layers exist because storefronts face real adversaries: price scrapers, inventory bots, credential stuffers, sneaker bots. Classifying every non-browser reader as hostile was a sound default when every non-browser reader was.

The environment has moved. Bot traffic is now a majority of web traffic (we covered the measurement debate in Brief #001), and a growing share consists of agents acting on behalf of customers: assistants comparing plans, checking return policies, or researching purchases. Infrastructure vendors have begun engineering for that distinction: crawler verification schemes, per-crawler access controls, and protocols to meter AI crawlers rather than block them outright. The tooling to distinguish scrapers from verified assistants is emerging. The operational defaults, on the evidence of our crawl logs, have not caught up: edge defenses still treat a polite assistant and an inventory scraper as the same threat.

Behind the wall, the wiring is already broken

Here is the finding from our data that goes beyond “companies block bots,” and the reason this brief exists.

One might assume walled sites are well-maintained behind the barrier: coherent, densely linked, ready for whatever reader is eventually granted access. The two consumer telecom surfaces that did permit a crawl tell a different story. One was an active commerce funnel where 92% of pages had no inbound link of any kind. The other was an extensive support library, thousands of carefully written words per page, with two-thirds of its content unlinked.

Organizations that treat the open web purely as a threat surface appear to also stop maintaining it as an information architecture. The wall does not protect a garden. On the empirical evidence, it hides a warehouse with the shelving removed.

That correlation matters for prediction. When agent-mediated research becomes standard practice in consumer buying, walled enterprises face a compound deficit: assistants cannot get past the perimeter, and if the gate opened tomorrow, the content behind it is not structured to be traversed.

What this does not say

Stated plainly, so the limits of the crawl are clear.

Our cohorts are curated Fortune 500 samples, not exhaustive sector censuses. A wall against our identified crawler does not prove a wall against every system; major model providers negotiate private licensing agreements that bypass edge perimeters entirely. Furthermore, blocking is often a spectrum of throttling rather than absolute refusal.

The claim that survives scrutiny: on polite, robots-respecting terms, the protocol standard consumer agents rely on, the storefront web of the largest US retailers and carriers is effectively invisible today. And the rare exceptions that can be measured exhibit the most fragmented link architectures we have recorded.

The choice hiding in the default

Defaulting to “block everything” has quietly shifted from a conservative security posture into a commercial bet: a bet that agentic discovery will not matter to your category.

For some categories, that bet may hold. For categories where buyers already ask an assistant before checking out, it is an open invitation to any competitor who exposes a crawlable product and policy surface outside the wall, or verifies legitimate agent traffic at the edge.

Both adjustments begin with knowing what your web architecture looks like to an external machine reader. That is an empirical condition, measurable before a single page is analyzed. Sometimes the measurement reveals an intelligible topology. Sometimes it reveals that from the outside, there is nothing there at all.