Fast Tests, Benchmarks and Computer Chess Rating Lists

fast tests benchmarks computer chess rating lists

Computer chess produces many kinds of numbers. A benchmark may report nodes per second. A short engine test may report wins, losses, draws, score percentage or a provisional Elo. A live tournament may show current standings. A published computer chess rating list may contain hundreds, thousands or even millions of games processed through a rating tool. These outputs can all be useful, but they do not answer the same question.

This distinction matters because computer chess readers often see a fast result and treat it as if it were a mature rating conclusion. A new binary may run faster on one CPU. A development version may score well in a short match. A benchmark may show a large nodes-per-second advantage. None of those facts is automatically the same as a stable strength estimate inside a controlled rating list.

A benchmark asks: how fast does this binary, engine or machine search under a defined command and hardware condition?

A fast test asks: what early signal appears under limited time, limited games or limited opponents?

A computer chess rating list asks: what relative strength estimate emerges from a structured body of games, with defined conditions, an opponent pool, time control, opening policy, rating method and publication status?

The difference is not cosmetic. It is methodological.

What benchmarks measure

A benchmark is primarily a measurement of execution behavior. In chess engines, this often means speed, node count, nodes per second, compilation behavior, CPU scaling, architecture differences or a deterministic search signature. It can be extremely valuable. It can show whether a binary is using the expected instruction set, whether a compile is badly configured, whether one CPU architecture is faster than another, or whether a local build roughly matches the intended engine version.

Stockfish documentation is a useful reference point. Its bench command runs a standard search benchmark on a selected set of positions and prints total nodes searched and time taken. The documentation also explains that the default node count can serve as a version “signature” or fingerprint for the exact search algorithm in the binary, while nodes per second can provide a basic speed indication; for a more stable hardware benchmark, the documentation recommends speedtest.

That is a benchmark’s proper domain. It is not a tournament. It is not an Elo table. It is not a substitute for game evidence. A benchmark can tell the tester that one machine searches faster, or that one binary reaches a different node count than expected. It does not tell the reader how the engine will score across a serious opponent pool under tournament conditions.

This is why benchmark publication should always preserve context. The command used, thread count, hash size, binary architecture, CPU model, operating system, compiler and large-page status can all affect the interpretation. A number such as “nodes per second” is meaningful only if the reader knows how it was produced.

The same principle applies to public hardware benchmark pages. Ipman Chess, for example, separates testing lists from AMD/Intel chess benchmark material and reports hardware-specific information, engine core usage, concurrency, time control, tablebase use, hash and opening-book notes on its testing pages. Its AMD–Intel bench page explicitly describes a console bench command and then presents hardware-oriented nodes-per-second comparisons across many systems.

That kind of work is useful because it gives readers a hardware and speed reference. The problem begins only when a benchmark number is read as if it were a rating-list result.

Why speed is not the same as strength

A faster engine binary can be stronger, but the relationship is not automatic. Chess engine strength depends on search, evaluation, pruning, time management, move ordering, neural network behavior, tablebase interaction, stability and many other details. More nodes can help, but better nodes matter as much as more nodes.

A speed benchmark is therefore a diagnostic tool, not a competitive proof. It can reveal that an AVX2 build is slower than expected, that a BMI2 build is inappropriate for a given CPU, or that a compile is not using the intended architecture. It can also help engine authors and testers detect regression or verify that a binary belongs to a specific code path.

But speed alone cannot answer these questions:

• Does the engine convert advantages reliably?
• Does it avoid tactical collapses under tournament time controls?
• Does it scale well at longer time controls?
• Does it handle quiet endgames, fortress cases and tablebase transitions correctly?
• Does it perform equally well against different engine families?
• Does it retain strength when book exits are balanced and reversed?
• Those are game-evidence questions. They require played games, not only a benchmark.

This is especially important for Stockfish-derived engines, private builds, experimental binaries and heavily tuned configurations. A benchmark can confirm that a binary is fast. It cannot confirm that the engine’s competitive behavior is stable.

What fast testing can reveal

Fast testing still has a legitimate role. A fast test can reveal whether an engine crashes, whether UCI options behave as expected, whether time management fails, whether a new binary is grossly weaker than expected, or whether a build has obvious tactical problems. In engineering terms, fast tests are often excellent smoke tests.

A smoke test is not designed to establish final strength. It is designed to detect immediate failure. In computer chess, a smoke test may include a few short games, a fixed-position tactical probe, a bench signature, a UCI startup check, a time-loss check, or a short gauntlet against a stable reference engine.

Fast tests can also be useful for triage. If an engine loses every short game on time, refuses UCI commands, crashes on tablebase access or produces illegal moves, the tester does not need a thousand-game rating run to know that the binary is not ready. Short testing protects larger rating workflows from avoidable contamination.

However, fast testing becomes dangerous when its output is framed as stable Elo. A 20-game result can be informative, but it is not equivalent to a mature rating estimate. A 50-game blitz match can suggest a signal, but it still needs context. A 100-game gauntlet may be useful for development, but it can still be narrow if the opponent pool is small or stylistically biased.

The correct language is therefore cautious. Fast tests can suggest, detect, reject, verify or flag. They should not overclaim.

Why rating lists need opponent pools

A computer chess rating list is relational. An Elo value is not produced in isolation; it is inferred from results against other engines. The opponent pool shapes the number.

A list built from original UCI engines will not mean the same thing as a list dominated by Stockfish-derived engines. A list built from long classical games will not mean the same thing as a bullet list. A list built from one hardware profile will not mean the same thing as a list built from another. A list built from broad round-robin evidence will not mean the same thing as a narrow gauntlet.

CCRL gives a useful public example of why conditions matter. Its 40/15 about page describes the project as a group that runs thousands of games, collects them into a database and computes a rating list. It also publishes testing conditions, including equivalent time control, tablebase use, pondering off, tournament-format flexibility, hash guidance and opening-book constraints.

That is the difference between a rating list and a naked table. The rating number is surrounded by conditions. Those conditions tell the reader what the list is measuring.

The opponent pool matters because engines do not all fail in the same way. Some are tactically sharp but strategically unstable. Some scale well with time. Some are robust against a wide field but weaker against top neural-network opposition. Some are book-sensitive. Some are more vulnerable in fast time controls. A rating list needs enough opponent diversity to reduce the risk that a result merely reflects one narrow matchup profile.

This is why IJCCRL should preserve track separation. Original UCI Track and Derived Stockfish Track can both be published, but they should not be collapsed into one undifferentiated claim. Track-aware publication protects the reader from comparing unlike evidence.

Enough games does not mean infinite games

A rating list needs enough games to be useful, but the word “enough” depends on the purpose. A provisional event table may be useful during a live tournament with fewer games than a long-term rating list. A final event rating may be useful for that event without becoming the universal measure of an engine’s strength. A historical rating list needs deeper evidence and more stable conditions.

The key is to label status honestly.

A table can be:

• a benchmark result;
• a smoke-test result;
• a fast-test table;
• a provisional event rating;
• a final event table;
• an official rating list;
• or a long-term historical rating surface.
• Each category has value. The error is treating them as identical.

TCEC’s rules provide a useful example of status language. Its engine-rating section states that TCEC ratings are updated live after official games and that new engines receive temporary ratings based on testing until an official rating can be calculated after event participation. That distinction between temporary and official status is exactly the kind of language a serious engine publication should preserve.

IJCCRL should do the same. A provisional list should say that it is provisional. A fast list should say that it is a fast list. A benchmark table should say that it is a benchmark table. Precision increases trust.

Rating tools answer a different question

When a rating list is published, the rating tool becomes part of the evidence chain. BayesElo and Ordo are two common names in computer chess rating work, and both illustrate the difference between game evidence and rating calculation.

BayesElo is described by Rémi Coulom as a freeware tool that can read PGN game records and produce a rating list. Its documentation shows how PGN input is loaded, how ratings are produced and how uncertainty-related outputs can be displayed.

Ordo, by Miguel A. Ballicora, is described as a program designed to calculate ratings for chess engines or players. Its documentation explains that it considers all results at once, can read PGN input and can output ratings in text or CSV formats.

These tools do not turn weak evidence into strong evidence by magic. They calculate from the games supplied to them. If the PGN is narrow, biased, corrupted, duplicated or poorly documented, the output inherits those weaknesses. A good rating tool is necessary, but it is not sufficient.

A mature rating publication therefore needs two layers:

the calculation layer, including the rating tool, settings, anchors and output format;

and the evidence layer, including games, conditions, event rules, hardware notes, opening policy and exclusions.

A rating list without evidence is difficult to audit. Evidence without a clear calculation method is difficult to interpret. The serious publication connects both.

Hardware notes should accompany fast experiments

Hardware notes are especially important when publishing benchmarks and fast tests. A reader needs to know whether the result comes from a laptop CPU, a desktop CPU, a server CPU, a many-core system, a single-thread run, a multi-thread run, a specific compiler, a specific architecture target or a specific operating system.

This is not just a technical footnote. In engine testing, hardware changes can change observed behavior. A fast test at 10 seconds plus increment on one system may not reflect performance at 40 minutes plus increment on another system. A binary optimized for one instruction set may behave differently on another CPU. Large pages, thread binding, hash size and tablebase access can all change speed and sometimes stability.

Ipman’s benchmark-oriented pages are useful precisely because they foreground hardware. They do not pretend that all benchmark numbers float in a vacuum. Systems, cores, threads, tablebases, hash and time controls are part of the reported context.

IJCCRL can borrow that discipline without copying the format. For every fast experiment, the publication should show at least:

• engine name and version;
• binary architecture;
• CPU model;
• threads;
• hash;
• time control or bench command;
• opening policy if games are played;
• tablebase use;
• game count;
• opponent pool;
• and status label.

That is the minimum context needed to avoid confusion.

Recommended IJCCRL taxonomy

The cleanest solution is to publish a fixed taxonomy and use it consistently.

Benchmark

A benchmark measures engine or hardware behavior under a command or synthetic search condition. It may report nodes searched, total time, nodes per second, binary architecture, compiler, thread count and hash. It should not be presented as an Elo result.

Recommended label:

Benchmark — hardware/search-speed evidence only

Smoke test

A smoke test checks whether a binary is operational. It may include UCI startup, isready, a short search, a small number of games, crash detection, time-loss detection and basic legality. It should not be presented as a strength estimate.

Recommended label:

Smoke test — operational validation, not a rating

Fast test

A fast test is a short competitive experiment. It may use fast time controls, a limited opponent pool or a limited number of games. It can reveal early signals, but the uncertainty remains high.

Recommended label:

Fast test — early competitive signal

Provisional rating

A provisional rating is a temporary estimate produced from incomplete or still-active evidence. It can be useful during a tournament, but it must not be framed as final.

Recommended label:

Provisional rating — active or incomplete evidence

Official rating

An official rating is a published estimate based on a defined dataset, controlled conditions, sufficient game volume, declared rating method and closed publication status.

Recommended label:

Official rating — closed dataset and declared method

This taxonomy would help IJCCRL readers understand exactly what kind of evidence they are looking at.

How fast tests should feed rating work

Fast tests should not be discarded. They should be placed in the correct part of the pipeline.

A healthy workflow looks like this:

• first, benchmark the binary;
• second, run UCI and smoke tests;
• third, run a fast competitive check;
• fourth, decide whether the engine is stable enough for a larger event;
• fifth, include the engine in a controlled tournament or rating pool;
• sixth, publish provisional status while evidence is still incomplete;
• seventh, close the event and preserve PGN, downloads, archive and rating output.

This chain gives every test a role. The benchmark protects the build. The smoke test protects the tournament. The fast test protects the rating pool. The tournament produces game evidence. The rating tool turns that evidence into a table. The archive preserves the historical record.

Problems occur when the chain is compressed. If a benchmark is treated as a rating, the publication overclaims. If a fast test is treated as an official list, the uncertainty is hidden. If a rating list is published without games or method notes, the evidence chain is incomplete.

IJCCRL should therefore keep its internal surfaces separated:

• chess engines ratings lists should remain the project-level gateway;
• current ratings hub should host rating surfaces;
• rules and audit should explain methodology;
• downloads should preserve final packs and evidence material;
• archive should store closed event history;
• and live broadcasts should remain the real-time layer, not the final evidence layer.

Why this matters for engine-release audiences

Engine-release audiences often move quickly. A new binary appears, a benchmark is posted, a short match is run, and the community wants an immediate conclusion. That is understandable. Computer chess is technical, competitive and fast-moving.

But quick publication can distort interpretation. A new compile may look stronger because it is faster on one CPU. A private build may win a small match because the opponent selection favors its style. A test may overrepresent one opening family. A gauntlet may compare one engine against an outdated or inappropriate pool. A benchmark may show impressive speed while the engine has no demonstrated tournament stability.

The correct editorial answer is not to reject fast information. The correct answer is to label it.

A reader can use a benchmark to decide which binary to run. A developer can use a smoke test to catch a broken build. A tester can use a fast test to decide whether a version deserves deeper testing. A rating editor can use provisional tables to track a live event. But only a controlled rating workflow should be presented as a rating-list conclusion.

What not to merge without context

Three comparisons are especially dangerous.

First, do not merge benchmark speed with Elo. A 20 percent speed improvement is not automatically a 20 percent strength improvement, nor does it automatically imply a fixed Elo gain.

Second, do not merge fast-test Elo with official rating-list Elo. A short match and a long-term list may both output numbers, but they do not share the same evidential weight.

Third, do not merge different hardware lists without explanation. A result obtained on one system at one time control cannot be treated as equivalent to another result obtained under different hardware and timing conditions unless the relationship is explicitly defined.

This is why rating tables need method notes. Readers should not have to guess whether they are looking at a hardware table, a fast-testing table, a provisional tournament table or an official rating list.

A practical reader checklist

Before trusting a number, ask:

• Is this a benchmark, a fast test, a provisional rating or an official rating?
• How many games were played?
• Which opponents were included?
• What time control was used?
• What hardware was used?
• Were openings controlled or reversed?
• Were tablebases used?
• Was the rating method stated?
• Is the PGN or download evidence available?
• Is the status final or provisional?
• If those answers are missing, the number may still be interesting, but it is not yet a fully interpretable rating-list result.

Conclusion

Fast tests, benchmarks and computer chess rating lists belong to the same ecosystem, but they do not have the same evidential status. A benchmark measures speed, search behavior or binary identity. A smoke test checks whether an engine is operational. A fast test provides an early competitive signal. A provisional rating helps readers follow an active or incomplete dataset. An official rating list requires controlled game evidence, a declared rating method, sufficient context and clear publication status.

The most responsible computer chess publication is not the one that produces the fastest number. It is the one that labels each number honestly.

For IJCCRL, the correct path is clear: benchmarks should remain hardware and build diagnostics; fast tests should remain early signals; rating lists should remain evidence-based publications connected to games, downloads, rules, archive and audit trails. That separation protects readers, protects engine authors and protects the credibility of the rating surface itself.

A benchmark is not a rating list. A fast test is not automatically a stable strength estimate. The value of each number depends on the question it was designed to answer.

Sources and references

• Stockfish documentation — UCI commands, bench and speedtest.
• Ipman Chess — engine testing lists and hardware/test-condition examples.
• Ipman Chess — AMD / Intel chess benchmark page.
• CCRL — 40/15 testing conditions and rating-list context.
• Rémi Coulom — BayesElo documentation.
• Miguel A. Ballicora — Ordo documentation.
• TCEC rules — engine ratings, temporary ratings, event rules and publication context.

Jorge Ruiz Centelles

Filólogo y amante de la antropología social africana

SÍGUEME