October 16, 2026

Context Is Not Memory

Why bigger context windows will not solve enterprise AI: what the memory hierarchy tells us instead, and what we are building at Neuron Factory.

Zaid Kahn, Co-Founder & CEO

⋅

25 min read

A number that isn’t what it looks like

For three years the industry has kept score with a single number. 8K. 32K. 128K. A million. Ten million. Every model launch leads with the size of its context window, and every launch invites the same inference: that a model which can hold more can understand more, and that the remaining gap between a language model and a competent colleague is a capacity problem that the next generation will close.

I spent a good part of my career building the machines these models run on: the AI supercomputers at Microsoft, and before that the infrastructure at LinkedIn, and a number of years as Board Chair of the Open Compute Project, where the industry argues in public about racks, power and memory. That work leaves you with a particular instinct. When a software abstraction advertises a number, go and find where that number physically lives.

Do that with the context window and you find something clarifying. The context window is not a property of the model’s intelligence. It is a budget line in a memory hierarchy. Specifically, it is an allocation of high-bandwidth memory (HBM), the stacks of fast memory that sit on the accelerator’s own package, a few millimetres from the compute die, holding the model’s weights. The context window has a price per token, a bandwidth cost to read, and a hard ceiling set by what fits in that HBM. It behaves exactly like every other fast, small, expensive tier of memory that computing has ever built.

And we have known for sixty years how that story ends. You do not solve the memory problem by making the fast tier big enough to hold everything. You solve it with addressing, with caches, page tables, indexes, and schemas that decide what belongs in the fast tier at any given moment.

That is the argument of this essay. The interesting engineering is not in growing the window. It is in building the layer that decides what goes into it.

In construction, that layer is a knowledge graph. Construction is the right place to prove the point for two reasons, neither of them sentimental. It has the widest gap of any industry between the size of the corpus a decision depends on and the size of any window you could put that corpus in. And it already has standardized ontologies, built over thirty years, in public, by an international committee, sitting there waiting to be used.

What a context window physically is

Strip away the interface and a context window is, physically, a KV cache.

For every token it has seen, each transformer layer computes two vectors. The key is what later tokens match against when deciding how much attention to pay to this one. The value is the content that gets mixed into the output when they do. Both are tensors, learned numerical projections rather than words, and they are retained so that generating token n+1 does not require recomputing the attention state for tokens 1 through n. By default those tensors sit in HBM alongside the model weights, and in a standard dense-attention model they are read in full on every decode step: once for every single token the model generates.

The size is arithmetic, not mystery:

KV bytes = 2 × layers × kv_heads × head_dim × tokens × batch × bytes_per_element

Run it for a Llama-3.1-70B-class model with 80 layers, 8 key-value heads under grouped-query attention (GQA), 128-dimensional heads, bfloat16: 320 KiB per token, or 40 GiB for one 128K-token sequence at batch size one. Put that on a GB300, one of the most capable accelerators in production. The card carries 288 GB of HBM3e, and the model’s own weights claim 140 GB of it. What remains holds three of those conversations. Three users at full context, on a part that costs more than a house, with not quite enough left over for a fourth.

Now run the counterfactual. Had that model used classic multi-head attention, with 64 key-value heads instead of 8, the same single sequence would need 320 GiB, roughly 344 GB. That is more than an entire GB300 before a single byte of weights is loaded, and weights plus one sequence comes to 484 GB. No accelerator shipping today holds both.

Long context is not something the field achieved by scaling memory. It is something the field achieved by compressing the working set. Multi-query attention (MQA) divides the cache by the head count. Grouped-query attention divides it by eight and recovers essentially all of multi-head’s quality. DeepSeek’s multi-head latent attention (MLA) compresses the cache into a smaller learned representation, pushing a 671B model down to roughly 69 KiB per token, about 8.6 GiB at 128K. Every one of those techniques is a compression scheme for a memory tier. That is the first tell.

The second tell is the hierarchy itself, and what it costs to move a working set across it.

Figure 1. What it costs to read one 128K-token KV cache, once, at each tier of the hierarchy. Bandwidth figures are vendor-published; HBM4’s 22 TB/s is a stated specification rather than an independent measurement. On-die SRAM sits above this ladder at roughly 19 TB/s but holds only tens of megabytes, which is why the cache cannot live there. Note that the dominant cost is a crossing of millimetres inside the package, not a trip across the machine.

Note where the lines fall. The first three tiers are all inside the GPU package: SRAM on the compute die itself, HBM a few millimetres away. Only the last three leave it. The dominant cost in long-context inference is a crossing of millimetres, not metres.

This spread is why prompt caching is priced the way it is, why KV-offload tiering into host DRAM and flash has become a serious engineering discipline, and why the whole question of re-prefill versus fetch is a genuine tradeoff rather than an obvious win.

It is a cache hierarchy. It has always been a cache hierarchy. The “context window” is simply the name we give to its top tier when we are talking to people who don’t want to hear about HBM.

The economics bend in two places

Two distinct cost curves govern long context, and they bend in opposite directions for opposite reasons. Conflating them is the source of most bad intuition on this subject.

Prefill is compute-bound and quadratic. Processing the prompt requires dense weight matrix multiplications, or matmuls, which scale linearly with token count, plus the attention matrix, which scales with the square. For a 70B model the crossover, where attention floating-point operations equal weight FLOPs, sits at roughly 107,000 tokens. Below that, prefill is a linear matmul problem. Above it, the quadratic term takes over: at 128K, attention is already 55% of prefill compute; at a million tokens, 91%. The marginal cost of a prompt token is not constant. It more than doubles by 128K and grows an order of magnitude by 1M.

Figure 2. Attention’s share of prefill compute as the prompt grows, for a 70B-parameter model. Below roughly 107,000 tokens prefill is a linear weight-matmul problem. Above it the quadratic term dominates, and the marginal cost of each additional prompt token keeps climbing.

Decode is memory-bandwidth-bound and linear. Every generated token requires reading the entire KV cache. The arithmetic intensity of decode attention, meaning how much arithmetic the chip performs per byte it must fetch, works out to a startlingly simple expression: the ratio of query heads to key-value heads, which for a bfloat16 GQA-8 model is 8 FLOP per byte. The machine balance of the accelerators we run this on is 206 to 295 FLOP per byte. Decode attention therefore sits twenty to forty times below the hardware’s knee. The silicon is essentially idle, waiting on memory.

And here is the part that breaks the usual playbook: batching does not fix it. Batching rescues weight loading, because one set of weights serves many sequences. It does nothing for the KV cache, because every sequence carries its own. The arithmetic intensity of decode attention is independent of both sequence length and batch size. There is no throughput trick waiting to be discovered.

The market has already priced this

  • Google steps its Gemini Pro rates at 200K tokens: 2× input, 1.5× output above the line. OpenAI steps at 272K inside a nominally one-million-token window.
  • A cached input token costs 0.1× a fresh one at OpenAI and Anthropic, and as little as 0.02× at DeepSeek. That spread, a factor of ten to fifty, is the market’s revealed estimate of what prefill compute costs relative to storing and fetching a KV cache.
  • Most telling: Google charges $4.50 per million tokens per hour for context-cache storage on Pro-tier models. That is a published rent on HBM occupancy, which is only a sensible thing to meter if context is memory allocation, not intelligence.

Measured throughput tells the same story. NVIDIA’s own published TensorRT-LLM figures for a 70B model on an H200 pair show output throughput falling from 6,328 tokens per second at a 128-token prompt to 609 tokens per second at a 20,000-token prompt, roughly a tenfold collapse. And 20,000 tokens is nothing. It is a moderately long email thread.

The clearest signal of all is what the infrastructure is doing about it. DeepSeek’s published deployment splits prefill and decode across separate pools, with a minimum unit of 32 GPUs for prefill and 320 for decode, a ten-to-one ratio that tells you precisely where the money goes. NVIDIA has gone further and announced a distinct class of GPU specialized for the prefill phase, with high-capacity, lower-cost memory instead of HBM, because prefill is compute-bound and does not need the bandwidth.

When the hardware itself bifurcates along the seam of an abstraction, the abstraction is leaking. We are now spending extraordinary capital to make a linear, metered, physically bounded resource behave as though it were unbounded.

And it still doesn’t work

If the cost were the only problem, it would be a good problem, because costs fall. The harder finding is that the capacity does not deliver the comprehension it implies, and this is now one of the better-replicated results in applied machine learning.

The cleanest framing is nominal versus effective context. Adobe Research’s NoLiMa benchmark removes the literal-lexical-overlap shortcut that makes classic needle-in-a-haystack tests look so good, then measures the longest context at which a model retains 85% of its short-context score.

Figure 3. Claimed context window against measured effective context on the NoLiMa benchmark, where effective means the longest context at which a model keeps 85% of its short-context score. The Llama 4 Scout and GPT-4.1 figures come from the paper’s extended table of additional recent models. Measured in 2025; no 2026-generation models are included.

A claimed-to-effective ratio of ten thousand to one is not a rounding error. It is a different quantity being measured.

You do not have to take an outside benchmark’s word for it. OpenAI published, in the same post that announced a one-million-token window, that its model’s multi-hop graph-traversal score on Graphwalks, a breadth-first search task, reached 61.7% on prompts shorter than 128K tokens and 19.0% on prompts longer than that. NVIDIA’s RULER work found that models claiming 32K could not reliably hold 32K, and its 2025 follow-up found that every model tested, including the million-token claimants, degrades with length. Anthropic documents the phenomenon by name in its own developer documentation, calls it context rot, and in its engineering writing frames attention as a finite budget that every additional token depletes.

Three findings sharpen this past “more tokens means more noise”:

  1. Length itself is causal. A 2025 EMNLP Findings paper held retrieval perfect, so the relevant span was always found, and then removed the distraction entirely: first by replacing every irrelevant token with whitespace, then, in a separate experiment, by masking them out of attention altogether. Performance still degraded between 14% and 85% as input length grew. The damage is not distraction.
  2. More context can be actively worse. NVIDIA’s own retrieval work found an inverted-U: answer quality rises with retrieved context, peaks, and then falls. In their ∞Bench results, a 70B model with 48K tokens of well-ordered retrieved context scored 47.3 F1, a standard precision-and-recall accuracy score, against 34.3 for the same model given 117K tokens of raw context. Less context, better answer, one-third the cost.
  3. Agents degrade worse than chat. Across 200,000 simulated multi-turn conversations, models lost an average of 39% relative to single-turn. The decomposition is the interesting part: raw aptitude fell only 16%, while unreliability rose 112%. Long-running agentic work does not gently lose accuracy; it becomes erratic. Anyone shipping agents into a production workflow should read that as the central risk.

So: capacity is not comprehension. The window is a staging area, not a memory. Treating it as memory is the category error at the root of a great deal of disappointing enterprise AI.

The missing layer is addressing, not capacity

Here is where the systems instinct earns its keep.

Every layer of a computer’s memory hierarchy is paired with an addressing mechanism. Registers have names. Caches have tags and associativity. Virtual memory has page tables and a translation lookaside buffer. Databases have indexes. Filesystems have inodes. In every case, the fast tier is small by design, and the intelligence of the system lives in the machinery that decides what occupies it.

The context window is the only memory tier we have ever built with no addressing scheme at all.

What passes for addressing today is embedding similarity: convert the question to a vector, convert the corpus to vectors, retrieve the nearest k. That is content-addressable memory with a fuzzy comparator and no schema. It has no notion of identity, so two mentions of the same entity are two vectors rather than one thing. It has no notion of type, so a date, a deadline and a milestone are all just text. It has no notion of relation, so it cannot represent that this supersedes that, or that A governs B. It has no notion of time, so a revised document and its superseded predecessor are equally retrievable and equally confident. And it has no notion of permission.

A knowledge graph supplies exactly those things: durable identity, typed relationships, a governed schema, temporal validity, and access control. It is not a better retriever. It is the address space.

Being honest about the evidence

I want to draw a distinction here that the market routinely blurs, and that I think is the single most important thing to get right, because it determines whether you build an asset or a cost center.

GraphRAG builds a graph over a document corpus by having a large language model extract entities and relations from text chunks, then runs community detection and pre-summarizes every community, so that broad queries can be answered by map-reduce over those summaries. It is a real advance for a real problem: corpus-level sensemaking, where no single chunk contains the answer.

But be precise about what kind of graph that produces. In Microsoft’s reference implementation the default behaviour enforces no strict schema, and predicates are not predefined. Edges largely record that two entities appeared near each other in the same text; the semantics of the verb between them is neither understood nor managed. “A loves B” and “A hates B” both collapse into “A and B are related.”

That is a weak graph, an adjacency structure over text rather than a model of a domain. It will tell you that a subcontractor and a liquidated-damages clause appear together. It will not tell you whether that subcontractor is bound by the clause, exempted from it, or the party entitled to invoke it. In a tender, those three readings are three different numbers.

The published gains are correspondingly narrower than the enthusiasm suggests, and the costs are higher. Independent benchmarking puts the best graph methods at roughly +3 to +10 points on multi-hop and aggregation questions, approximately neutral on single-fact lookup, where plain vector retrieval often wins, and unreliable on summarization. HippoRAG 2, one of the strongest published graph systems, beats the best vector baseline by 2.8 F1 on average across seven question-answering datasets. Meanwhile Microsoft’s full GraphRAG global search has been measured at over 330,000 tokens per query against roughly 900 for vanilla retrieval-augmented generation, and Microsoft’s own follow-up work, LazyGraphRAG, reports comparable global-query quality at 0.1% of the indexing cost and over 700× lower query cost, which is, read plainly, an admission about the original’s economics.

I say all of this because a graph built the wrong way is a very expensive way to be slightly better at retrieval.

An ontology-grounded enterprise knowledge graph is a different object. Its schema is declared and governed rather than emergent per corpus. Its predicates are typed, and the type carries meaning: governs, supersedes and is-exempted-from are distinct relations a machine can reason over, not a single undifferentiated edge. Its entities have durable identifiers resolved once and referenced everywhere. It is persistent rather than rebuilt on every index run. It models time explicitly, so facts have validity intervals and a superseded fact is invalidated rather than deleted. And its purpose is to model the business, with actions and permissions, rather than to improve search over text.

  • Origin. GraphRAG: LLM extraction from documents. Ontology-grounded knowledge graph: Modeled against declared domain ontologies.
  • Schema. GraphRAG: Emergent, per-corpus, unstable. Ontology-grounded knowledge graph: Declared, versioned, governed.
  • Predicates. GraphRAG: Untyped; largely co-occurrence. Ontology-grounded knowledge graph: Typed; the relation itself carries meaning.
  • Lifetime. GraphRAG: Ephemeral; re-index on change. Ontology-grounded knowledge graph: Persistent; system of record.
  • Identity. GraphRAG: Heuristic, by name similarity. Ontology-grounded knowledge graph: Durable IDs, resolved once.
  • Time. GraphRAG: Snapshot. Ontology-grounded knowledge graph: Validity intervals; supersession.
  • Purpose. GraphRAG: Better retrieval over text. Ontology-grounded knowledge graph: A model of the domain.

And the asymmetry in the evidence is striking. The largest measured accuracy gains anywhere in this literature come from the second category, not the first. In a benchmark on enterprise question answering over a real insurance schema, GPT-4 answering directly against the raw relational schema was 16.7% accurate; the same model, same questions, querying an ontology-backed knowledge graph, reached 54.2%. A follow-on paper added a deterministic ontology-based query checker and a repair loop and pushed it to 72.6%, a four-fold improvement over the baseline.

The detail I find most instructive: roughly 70% of the repairs were triggered by ontology-based checks, meaning domain, range and domain-specific rule violations. The model was systematically asserting relationships the domain forbids, claiming that A contains B when the ontology says B contains A. That is precisely the class of knowledge that prose cannot encode and a schema can, and no amount of additional context window fixes it, because the constraint was never in the text to begin with.

Three more results worth having in mind:

  • Ontology-grounded retrieval (OG-RAG, Microsoft Research) reports +55% fact recall and +40% response correctness across four domains by retrieving semantically complete units rather than top-k chunks.
  • GraphReader, an agent navigating a graph of atomic facts with a 4K-token window, outperformed GPT-4 with a 128K window across input lengths from 16K to 256K. Structure beat capacity by a factor of thirty-two.
  • Zep’s temporal knowledge graph improved on a long-conversation benchmark in three dimensions at once. It cut token count roughly 70×, from 115,000 down to 1,600. It raised accuracy 11 points, from 60.2% to 71.2% on GPT-4o, an 18.5% relative gain. And it improved latency about 11×, from 28.9 seconds down to 2.6. Accuracy, cost and latency all moving the right way together is what a proper memory hierarchy is supposed to do.

That is the shape of the win. The graph does not replace the context window. It decides what goes into it. It is a compiler for the working set: given a question, resolve it against the ontology layer, traverse to exactly the facts that bear on it, and hand the model a few thousand tokens of typed, resolved, cited material. The design target is not to fill a million tokens. It is to need four thousand.

Why construction is the hardest case, and therefore the right one

Construction is where we at Neuron Factory have chosen to prove this, and not for sentimental reasons. It has three properties that make it the sharpest possible test.

The corpus is orders of magnitude past any window

Take a real, documented figure rather than one of the numbers that circulate in industry decks. Crossrail, a £16bn programme with sixty-plus major contractors, recorded over 650,000 asset tags, each carrying an average of ten key attributes, which between them generated roughly 10 million documents including as-built drawings.

Set that against a context window. A dense page of specification text runs somewhere around 500 to 800 tokens. A one-million-token window is therefore, generously, 1,250 to 2,000 pages. A single large tender package of specifications, drawings, bills of quantities, geotechnical reports and addenda routinely exceeds that on its own. A contractor’s portfolio of past projects, which is where the actually valuable knowledge lives, exceeds it by four orders of magnitude.

There is no version of “wait for a bigger window” that closes a gap of 10,000×. The gap is not closing; the corpus is growing faster than the window.

The pain is concentrated in exactly the operation a graph is good at

The honest version of construction’s productivity story is more complicated than the slides usually admit, and I would rather make the weaker true argument than the stronger false one.

The most-cited figure is McKinsey Global Institute’s finding that construction labour productivity grew about 1% annually over two decades, against 2.8% for the world economy. That figure is from 2017, using data through roughly 2015. It is nine years old, and it is the most misquoted statistic in the industry. More recent academic work from Harvard and the National Bureau of Economic Research argues the leading explanation is local land-use regulation, operating through constraints on project and firm size that suppress scale and discourage technology investment altogether. That is a serious argument, and it is not an information-management argument.

So let me be precise about the claim I am actually making. Information management is not the root cause of construction’s productivity history. But it is the largest addressable slice inside a single firm’s control, and the evidence for that slice is specific:

  • The Navigant Construction Forum’s synthesis of the Construction Industry Institute’s rework research puts median direct rework at about 4% of contract value, rising to 7.3% to 10.9% once indirect costs are counted, and notes that a substantial share of respondents believe their recorded rework captures only a fraction of what actually happens.
  • A 2018 industry survey of 599 professionals found they spend 5.5 hours per week searching for project data, plus 4.7 on conflict resolution and 3.9 on mistakes and rework. It is a vendor-sponsored survey and should be read as directional, but the shape of it matches every estimating floor I have ever walked.
  • The Construction Industry Institute’s work on front-end planning puts its cost at about 2.5% of total project cost, and finds that projects doing it well realize on average 10% cost savings and 7% schedule reduction, with an 8.6% cost advantage for intensive implementers over light ones.

Read those together and the mechanism is clear. Front-end planning is the highest-leverage 2.5% of spend on a project. The binding constraint on doing it well is estimator hours. And a large fraction of estimator hours is consumed by search and reconciliation: finding the relevant clause, checking the drawing against the specification, determining which addendum governs, and reconciling a supplier quote against a scope boundary. That is not a generation problem. It is an addressing problem.

The ontologies already exist

This is the asymmetry that makes construction the right first market, and it is the thing most AI companies entering the sector have missed. Most industries attempting ontology-grounded AI must first invent the ontology. Construction has spent thirty years building a whole layer of them, in public, by international committee:

  • Industry Foundation Classes (IFC) 4.3 is published as ISO 16739-1:2024 and defines, in buildingSMART’s own accounting, over 1,300 entities and roughly 2,500 properties organized into more than 750 property sets, and those are described as only the basic terms.
  • ifcOWL provides an official web-ontology-language serialization. The buildingSMART Data Dictionary hosts hundreds of interconnected dictionaries with immutable URIs, built on ISO 12006-3, 23386 and 23387.
  • ISO 19650 mandates a common data environment (CDE) with a formal state model of work-in-progress, shared, published and archived, meaning the industry already requires a governed information system of record.
  • MasterFormat gives fifty divisions of work results; UniFormat gives an orthogonal elemental classification; OmniClass gives multi-table faceting.
  • Information Delivery Specification defines machine-interpretable, automatically validatable exchange requirements: a contract to deliver the correct information. Construction built machine-verifiable data contracts before the analyst firms started predicting them.

And notice what that last set of standards implies. The same wall is simultaneously a work result under MasterFormat, an element under UniFormat, a physical object in IFC, a line in the bill of quantities, an activity in the schedule, a scope boundary in a subcontract, and a submittal in the CDE. Those are seven addressing schemes over one entity.

Figure 5. One wall, seven classifications. Each standard addresses the same physical thing for a different purpose, and each is maintained by a different party. Chunk retrieval can only find text that mentions the wall. A graph holds the identity, and the seven views are joins against it.

Flat chunk retrieval cannot represent that. It can only find text that mentions the wall. A graph represents it natively, because that is a join, and the joins are already standardized.

We could not find peer-reviewed work applying knowledge graphs specifically to tendering and estimating at corpus scale. The preconstruction phases are essentially unstudied. That is not a warning sign. That is the opportunity.

What we are building

Neuron Factory’s platform is built on a project-specific knowledge graph. It is a knowledge graph, deliberately and specifically, not a GraphRAG pipeline, and the distinction drawn above is the design. I will describe the architecture rather than any client engagement.

Ingestion aligns, it does not chunk. Documents pass through document understanding and knowledge extraction, and the output is not a pile of embedded fragments. It is entities and relationships aligned to construction ontologies, the standards above, extended where a firm’s practice demands it, with typed edges between them. A quantity is a quantity, attached to an element, governed by a specification section, priced in a bill line, assigned to a scope. The source text remains, indexed and linked, as evidence for every assertion. Both layers matter: the formal layer for reasoning, the text layer for provenance.

Conflicts are first class citizens. Tender documents are not a static corpus; they are a versioned negotiation. Addendum 3 supersedes a clause in the base specification. A drawing revision changes a dimension that a quote was priced against. In a vector index those are two similar chunks and the model picks one. In our graph a superseded fact carries a validity interval and an explicit supersession edge. It is invalidated, not deleted, because “what did we believe when we priced this, and what changed since” is a question every estimator and every claims process eventually asks.

Agents are graph traversals with a compiled working set. The multiple agents we run at the moment, covering prospecting, drawing intelligence, takeoff, conflict detection, and scope-of-work packaging into MasterFormat bid packages, are not prompts over a retrieved pile. Each is a traversal that resolves a question against the ontology layer and compiles a small, typed, cited working set for the model.

The conflict agent is the clearest illustration of why this architecture is not optional. The question “does this drawing contradict the specification?” is a join between two documents that never co-occur in any chunk, are not semantically similar to each other, and would frequently not both survive a top-k retrieval, precisely because each is individually a good match for the query and they are redundant with one another in embedding space. It is a graph query. The window only ever needs to see two nodes and the standard that governs them both.

Figure 4. The same question under two architectures. A drawing and the specification that governs it are each a good match for the query and are close to one another in embedding space, so a top-k retrieval tends to return one and discard the other as redundant. In the graph the relationship is an explicit typed edge, and the working set handed to the model is three nodes.

Everything carries provenance. Every fact the model sees arrives with its source document, section, and revision. This is not a nicety. A number in a bid is a contractual commitment. An estimator cannot act on an unsourced assertion, and a firm cannot defend one. The graph makes citation structural rather than an instruction we hope the model follows.

What this costs, honestly. Graph construction is more expensive than embedding a corpus. Ontology work is real engineering labour, not a one-afternoon schema exercise. Schema rigidity is a genuine failure mode, because a model of the world that cannot accommodate how a particular firm actually works is worse than no model. We accept those costs because they are paid once per domain and then amortized. The benefit of a graph is reuse: of the ontology, and of the business logic built on top of it. The second project on a given ontology layer is materially cheaper than the first, and the tenth is a different economic proposition entirely.

That compounding is the whole thesis. Cross-project intelligence, meaning how a subcontractor has performed, what we priced an assembly at last time, and which clauses have historically produced claims, is not a retrieval feature. It is what you get once entities have durable identity across projects. It is impossible without that, and close to automatic with it.

What changes

I will not put numbers on outcomes I cannot yet defend publicly, so let me describe mechanisms instead.

The constraint on a general contractor’s preconstruction function is estimator capacity: how many qualified people the firm has available to read a tender, price it, and stand behind the number. That capacity determines how many tenders the firm can pursue, and how thoroughly it can pursue each one. Both matter. Coverage drives the top of the funnel, and thoroughness drives both hit rate and margin, because an underexamined bid is either lost or won badly. When the search-and-reconcile fraction of estimator time compresses, both variables move at once, and they move against a base of 2.5% of project cost that the Construction Industry Institute’s data says governs 10% of the outcome.

The second-order effect is more interesting than the first. Once a firm’s tender history lives in a graph rather than in ten thousand folders, the estimating function stops being a per-project cost and starts being an accumulating asset. Every bid priced, every clause negotiated, every subcontractor’s performance becomes a queryable structure rather than institutional memory sitting in the head of someone two years from retirement. That is the transition the industry has been promised by a decade of document-management software and has not received, because document management stores documents, and what the work actually requires is a model of the project.

And none of this is really about construction. The pattern generalizes to any domain where documents must be reviewed, accepted by a regulator, and then acted on by whoever builds the thing, where what matters is not the volume of paper but the handover of semantics and business logic between parties who each have to trust it and act on it. Construction is simply where the gap is most extreme and where the ontologies already exist, which makes it the best place to build the thing and prove it works.

Even the silicon stopped betting on capacity

For most of this essay I have argued that waiting for a bigger window is a bad bet. The hardware roadmap has now started making that argument for me.

At GTC 2025, NVIDIA previewed Rubin Ultra at 1 TB of HBM per package, built from four compute dies carrying sixteen stacks of 16-high HBM4E. The configuration now reported for shipping silicon is 192 GB: two compute dies, 8-high HBM4. That is not a trim. It is below the 288 GB of Blackwell Ultra and standard Vera Rubin, which would make it the first time in recent memory that NVIDIA’s flagship ships with less memory than the part it replaces.

The reasoning is the part worth internalizing. Bandwidth does not come from stack height. Each HBM4 cube has 2,048 I/Os divided across its core dies, so roughly four layers already saturate the interface; stacking higher buys capacity and cost, not bandwidth. Yield losses compound with every die added. And because memory is supply-constrained, shorter stacks yield more usable cubes per wafer, with a 4-high stack delivering roughly three times the bandwidth per wafer of a 12-high one. Run the economics on a real inference workload and both 8-high and 12-high raise cost per token against 4-high. In SemiAnalysis’s modelling, a 26% system-cost premium for 12-high buys about 10% more throughput.

So the industry is converging on shorter stacks. Not because taller ones are impossible, but because the extra capacity has stopped paying for itself. Asked when the memory crunch eases, SemiAnalysis’s answer was that it does not happen within this decade.

Read that against the premise this essay opened with. The bet that the next generation closes the gap by handing you a bigger window is not merely slow. It is being unwound by the people who build the silicon, on the grounds that capacity is not worth what it costs relative to bandwidth. Meanwhile prefill is moving onto specialized chips, and KV caches are being tiered into host memory and flash with the same care we once gave to page-replacement algorithms. Every one of those moves is the industry conceding that the fast tier cannot hold everything, and building the hierarchy instead.

Because the history of the memory hierarchy has never been a history of making the fast tier large enough to hold everything. Every time we have tried, the working set has grown faster. What has actually worked, every single time, is better addressing: the cache tag, the page table, the B-tree, the schema. We have spent sixty years learning that the intelligence of a memory system lives in what decides what belongs in the fast tier right now, and almost none of it lives in the size of the tier.

The context window is the fast tier. It is L1 for reasoning. Treating it as though it were the disk has produced a generation of enterprise AI systems that are impressive in demonstration and unreliable in production, not because the models are weak, but because we skipped the addressing layer and hoped capacity would substitute for it.

It will not. So we are not waiting for a bigger window.

‍

Special thanks to Bogdan Arsintescu and Cristina Civili for their contribution to the article

Sources and notes

CONTEXT WINDOWS AS MEMORY

PRICING AND THROUGHPUT

LONG-CONTEXT DEGRADATION

GRAPHS, ONTOLOGIES AND RETRIEVAL

CONSTRUCTION

Other blogs

Why Construction Needs an AI Coworker — Not Just a Tool

Neuron Factory Achieves SOC 2 Type 1 Certification

Knowledge Graphs: Missing Link for AI in AEC?