On this page
Make the measurement answer the question
A useful comparison identifies the model, source revision, inputs, initial state, hardware and software. It records warm-up, sample count, run order, units, failures and uncertainty.
For a conversion, we first compare the required outputs and state with the source. For an efficiency change, we measure cost while checking the behaviour that must be preserved. Deployment decisions need quality and resource measurements on the intended workload.
Describe the level measured
- Component measurements time a named operation, such as plan construction, a neural kernel or a physics step.
- Systems measurements include the relevant surrounding work: data movement, routing, inference, updates, storage and recovery.
- Quality evaluations test the actual task on identified data, with selection and final evaluation roles recorded.
- Exploratory probes help choose the next experiment. Their results retain that limited purpose.
The current engineering summary contains exact-state runtime checks and component CPU comparisons reviewed on 20 September 2026. Each table links to a record with its original measurement conditions.
Earlier component results
The retained sparse-plan experiment reduced mean 65,536-token plan-construction wall time from 784.49 ms to 7.99 ms across 20 measurements per arm on one CPU host. It preserved the tested plan identity. The arms ran sequentially in one process, so the result applies to that component and host. Read the record.
The historical extreme-context ladder used synthetic all-zero K/V history, 20 decode trials per context and mode, 10 cache-transition trials and 10,000 bootstrap resamples. It measured storage representations, execution and resource admission. Language quality requires natural-language evaluation. Read the protocol and results.
Retained July findings
The 28 July handoff recorded a 50-replicate Tiny CPU test at 2,048K synthetic context. Full-vocabulary decode averaged 357.8 tokens/s. The faster Turbo path reached 441.9 tokens/s with 82% exact sequence agreement and differing first-step logits. Those semantics are part of the result.
Two sparse-geometry candidates also failed their frozen approximation tests. K8 missed the required threshold; at equal active width 1,152 both registered geometries failed. The finer geometry first passed at 2,880 of 3,072 active neurons.
Source-family CI passed, while final assembly jobs were blocked by an external billing limit. The terminal handoff retains that release status and its original date.
Reconstruction comparisons
The Qwen reconstruction record compares the complete expert bank with the pinned donor on its frozen engineering cases. It records logits, token ordering and greedy generation. Full-expert reconstruction establishes a reference for testing later sparse execution.
Further comparisons hold the donor, prompts, attention mode, runtime, hardware and measurement method constant. Each routing setting is checked for quality and token-level differences before its latency, throughput, memory and energy results are interpreted.
Reporting a result
We keep the source identity, method, quality checks and measured costs together. Negative results, interruptions and excluded samples remain visible. Changes to a threshold or protocol receive a new experiment identity.
Commercial savings are evaluated through the full workload and its operating costs. A faster component is useful engineering progress; the complete deployment still needs its own measurement.