On this page
Start with the decision
An experiment should answer a question that matters. Before running it, we identify the behaviour to measure, the comparison, the data and the result that would justify the next step.
For commercial work, we agree those details with the person who will use the result. Scope, price, data access and delivery expectations are written down before work starts.
Make the comparison useful
We record the source model and version, inputs, initial state, hardware, software and evaluation method. A conversion is checked against its source. An efficiency claim requires a quality comparison on the intended workload.
We measure the full cost relevant to the claim. That can include routing, memory movement, preprocessing, model execution, updates and checkpointing. Component timings identify the exact component measured. End-to-end results include the rest of the workflow.
Preserve the result
The record keeps the configuration, measurements, failures and decision together. Model and artifact hashes identify the exact material used. Checkpoint and restart tests establish which state can be recovered.
Failed and inconclusive experiments stay in the record. When a comparison is defective, we record the defect and run the correction under a new identity. The original result remains available for review.
Publish what can be checked
Public summaries identify their source and date. Tables, charts and downloads are generated from the same records. Historical results retain their original conditions and review dates.
Current implementation checks, live-service observations and model-quality evaluations each describe what was measured. The technical detail includes limitations that affect interpretation, such as a small sample, shared hardware or incomplete spatial coverage.
The public site carries reviewed summaries, measurements and provenance. Protected implementation details, weights, private data and credentials are handled separately for authorised technical review.
Decide what to do next
A useful result can support an upgrade, identify a repair or show that a proposed change should be dropped. We use it to decide where to put the next round of engineering time and compute.
Commercial releases also need operating evidence: installation, security, recovery, support and the ability for someone else to run the system. Each product page explains its current availability.
Read the latest evidence, benchmark method or recorded experiments.