From the free book Delta: Closing the Specification Gap (Dhuri, 2026). The PDF is the canonical edition — figures, tables, and code formatting are simplified online.
Tokens, statistical completion, context windows, and the mental model that unlocks better prompts
“You don’t need transformer mathematics to write excellent prompts. You need one mental model: the model is completing your document, not answering your question.” — If you write questions, you get question completions. If you write specifications, you get specification completions. Only one of those ships to production.
2.1 The Completion Engine The Mental Model That Changes Everything When you send a prompt to a frontier code-generation LLM, what reaches the model is a sequence of tokens. The model predicts the most likely next token, repeatedly, until the output looks complete. It is predicting. Not reasoning. Not retrieving from a knowledge base. Predicting based on statistical patterns across its training corpus. This means: if your document says ‘write a C# payment service’, the model predicts the tokens that most commonly follow that instruction in training data. Those tokens constitute a tutorial example — because most C# payment examples in training data ARE tutorials. They use double for amounts. They skip idempotency. They know nothing about your codebase.
12
ANTI-PATTERN — Expecting the Model to Infer Your Domain AVOID: ”Write a Spring Boot service that handles money correctly” USE: ”<domain_types>Money record: BigDecimal amount, Currency currency. Use Money.of(String, String).</domain_ types><constraints>Money value object for ALL amounts. Never double, float, or raw BigDecimal directly. Rounding: HALF_UP always.</constraints>” Why: The model can’t infer your domain definition of ‘correctly’. Statistically, ‘correct’ Java money handling uses BigDecimal. Your domain may require a richer Money value object. Specify it.
Cache?
500–2,000
Always
Version-control this. Cache every call. Cached reads ~90% cheaper.
50–200
No
Always fresh. Reflects current state.
200–600
Include the most domain-representative method you have.
Domain types (Money<T>, Result<T>, entities)
100–400
Copy from Appendix B. The model can’t infer these.
2,000–10,000
Use MCP filesystem instead. Ch 13.
106 +
Even million-token windows reason worse and cost real money when filled. Don’t stuff — give targeted access via the filesystem MCP server (Ch 13).
Minimum sufficient context is not about cost — it is about quality. Every irrelevant token pulls the statistical prediction in irrelevant directions. Thirty lines of the right context outperform 300 lines of loosely related code.
Measure your most common generation prompt in tokens (use tiktoken, your provider’s token-counting endpoint, or any online tokenizer). Identify the stable portion (system prompt, domain context). Cached reads of that stable portion cost ~90% less; because variable input and all output still bill at full rate, realistic wholeworkload savings are typically 50–80%. The implementation is in Chapter 12, and takes 30 minutes to set up.
0.0–0.2
Highly deterministic. Most probable tokens.
Production code generation, migrations, security reviews.
Anything requiring creative exploration.
0.3–0.6
Balanced. More variation, still coherent.
Documentation drafts, ADRs, technical writing.
Code that must be correct the first time.
0.7–1.0
Creative. Surprising output.
Architecture brainstorming, alternative design exploration.
Code that goes to production.
For enterprise code generation with a frontier LLM via API or IDE assistant: temperature 0.1. You want the highest-probability-correct tokens, not creative variation. Save creativity for the problem you are solving, not the code that implements it. Two practical notes: many agentic tools (Claude Code, Copilot) don’t expose the knob — their code defaults are already conservative — and when a model’s extended-thinking or reasoning mode is active, providers fix or ignore temperature entirely. The dial applies where you control the sampling call: direct API integrations, eval harnesses, and assistants that surface it.
Extended Thinking?
Why
No
Pattern is deterministic. Extended thinking adds tokens without quality gain.
Choose between event sourcing and CRUD for a HIPAA audit trail
Yes
Multiple competing constraints — correctness, auditability, query performance, team capability.
No
Mechanical. No complex trade-offs.
Yes
Hypothesis elimination requires step-by-step reasoning across multiple possible causes.
No
Clear criteria. Not reasoning-intensive.
Yes
Hidden costs, failure modes, and regulatory implications benefit from visible chain-of-thought.
2.5 The Typed Mental Model For engineers who think in strongly-typed terms, here is the most useful model for any LLM call: C# — The LLM Call as a Typed Method (Conceptual) // Think of every call to an AI coding assistant as calling this: public Task<string> Generate( string role,
// Who is the expert? What is their
,→ stack? string[] context,
// What MUST they know? (not inferred — you provide it) string task,
// What exactly do you want? (types +
,→ return values) string[] constraints,
// What is forbidden or required? (
,→ domain rules) float temperature = 0.1f, // 0.1 for code; 0.5 for docs; 0.8 for ,→ brainstorm bool extendedThinking = false // true only for complex trade-off ,→ analysis ); // CRITICAL: context[] is NOT your codebase unless you explicitly ,→ provide it. // CRITICAL: constraints[] does NOT enforce your domain rules unless ,→ you write them. // Defaults for both: whatever is statistically most common in ,→ training data. // For C# that means: double for money, field injection, no ,→ idempotency. // For Java that means: @Autowired fields, FetchType.LAZY N+1 risks, ,→ double for BigDecimal.
~90% cheaper cached reads on stable prompt prefixes using prompt caching (whole-workload savings typically 50–80%) — the highest-ROI infrastructure investment for any team making more than 500 AI-assisted code generations per month. Source: Anthropic API documentation (see docs. anthropic.com/prompt-caching) See Research & Statistics Notes (see Appendix J)
beginning of the document they complete. Write specifications, not questions. + Minimum sufficient context improves quality as well as cost — irrelevant tokens degrade output by pulling predictions in wrong directions. + Temperature 0.1 for code generation. Extended thinking only for complex multi-constraint trade-off analysis. + The typed mental model: context[] and constraints[] are empty by default. Statistical averages fill both. Those averages are wrong for your domain. + Prompt caching makes cached-prefix reads ~90% cheaper (typically 50–80% off the whole workload). Implement before scaling any AI workflow.
16 TRY IT NOW
UP NEXT — Chapter 3: The Specification Frame — Your Master Key
Chapter 3 is the most important chapter in this book. It contains the four-element structure that closes all three Specification Gap components simultaneously, plus the complete compilable Result<T> and Money<T> implementations that every subsequent recipe depends on.
17