Rules for a Computer Chess Rating List
Inclusion rules decide what a computer chess rating list actually measures. A rating list is not defined only by its Elo table, game count or published rank order. It is defined first by the engines it allows into the pool, the versions it accepts, the status it assigns to each participant, and the categories it uses to separate different kinds of evidence.
A computer chess rating list that mixes public releases, private builds, original UCI engines, Stockfish-derived engines, experimental forks and outdated versions without clear labels may still produce numbers, but those numbers become difficult to interpret. The central question is not only “which engine is strongest?” The better question is: strongest among which engines, under which inclusion rules, and inside which publication category?
This article defines an IJCCRL-style framework for engine inclusion rules. It is not a replacement for established resources such as CCRL or TCEC. CCRL already publishes rating lists with conditions, categories and comments on inclusion, and its About page explicitly states that CCRL does not decide engine originality and that inclusion or non-inclusion should not be treated as a judgement on an engine’s status. TCEC, in turn, defines event participation, stage structure, temporary ratings, engine updates, hardware, adjudication and archive logic inside its own competition framework.
IJCCRL should therefore publish its own inclusion policy for its own system: live broadcast, PGN evidence, downloads, archive, winners, rules and rating lists. The purpose is not accusation. The purpose is interpretability.
Why inclusion rules matter
A rating list is a measurement system. Before it measures performance, it defines the population being measured. If that population is unclear, the table becomes ambiguous.
For example, a list of original UCI engines does not answer the same question as a list of Stockfish-derived engines. A list of public releases does not answer the same question as a list that allows private development builds. A list that accepts every new commit immediately does not answer the same question as a list that freezes versions for a complete tournament cycle.
This distinction matters because readers often treat Elo as if it were portable across contexts. It is not. Elo belongs to a defined pool, a defined method and a defined dataset. If the inclusion rules change, the meaning of the rating changes with them.
A strict inclusion policy protects the list from three common problems:
First, it prevents category confusion. Readers should know whether an engine belongs to an original-engine pool, a derived-engine pool, a private-engine pool, or a provisional testing pool.
Second, it prevents version drift. A rating list should not quietly compare one engine’s stable public release against another engine’s private nightly build unless the difference is labelled.
Third, it protects public trust. When readers can see why an engine was included, excluded, labelled provisional, or moved to a separate track, the rating list becomes easier to audit.
Public releases versus private engines
The first inclusion distinction should be public release status.
A public release is an engine version that ordinary users can obtain, identify and test independently. It may be open source, freeware, commercial or otherwise publicly distributed, but the key point is that its identity can be verified outside the rating list.
A private engine is different. It may be legitimate, original and technically serious, but if the tested binary is not publicly available, the reader cannot independently reproduce the exact tested object. That does not automatically make the result invalid. It does mean the result should be labelled differently.
A serious computer chess rating list should therefore separate at least three statuses:
Public release means the engine version is publicly obtainable and identifiable.
Private or author-submitted build means the binary was provided for testing but is not generally public.
Internal IJCCRL test build means the engine is tested for development, infrastructure or event reasons, but should not be treated as a fully comparable public release unless the build later becomes public.
This distinction is especially important for readers who use rating lists to choose engines for their own testing. A public release rating helps them understand what they can download and reproduce. A private build rating may still be useful, but it is a different kind of evidence.
The policy should be simple: private engines may participate in events only if their status is declared. They may receive provisional or event-specific results, but they should not be silently merged into a public-release rating surface unless IJCCRL intentionally defines such a surface.
Original UCI engines versus Stockfish-derived engines
The second major distinction is engine family.
The Universal Chess Interface gives engines a common protocol for communicating with GUIs and tournament managers. But UCI compatibility does not say whether an engine is original, derived, experimental, private, commercial or forked.
For IJCCRL, the most important public separation should be:
Original UCI Track
This track should focus on independent engine families and should be the main international-facing competition surface. The purpose is diversity of engine design, independent authorship and public comparability across distinct chess-programming traditions.
Derived Stockfish Track
This track may still be interesting and useful, but it answers a different question. It measures performance among engines derived from or strongly influenced by the Stockfish ecosystem, including forks, tuned builds, book-integrated variants or derivative experimental branches.
The point is not to attack derived engines. A derived-engine tournament can be technically interesting. It can reveal tuning differences, compile differences, NNUE update discipline, book policies and engineering choices. But it should not be merged casually with an original-engine list.
A Stockfish-derived engine and an original UCI engine may both be valid UCI competitors, but they do not necessarily belong in the same primary rating interpretation. If they are mixed, the list must say so clearly. If they are separated, readers can understand each table more honestly.
For IJCCRL, separation is the stronger editorial choice. It allows the Original UCI Track to represent independent engine competition while the Derived Stockfish Track remains a secondary but documented branch.
Version naming and architecture naming
Inclusion rules must also define version identity.
A rating list should not publish only “Engine X” if multiple builds exist. It should identify enough of the tested object to make the result traceable. Useful naming elements include:
- engine name;
- version number or date;
- commit hash where applicable;
- CPU architecture or binary target;
- thread count if relevant;
- NNUE or network file identity where relevant;
- public/private/provisional status.
For example, an engine tested as an AVX2 binary should not be silently treated as identical to an SSE4.1/POPCNT binary if architecture affects performance. Similarly, a Stockfish-derived engine using a newer default NNUE network should not be collapsed into an older build without documenting the network identity.
Architecture naming is not cosmetic. It helps readers understand reproducibility. A rating surface that says “64-bit” may be too vague when practical performance differs by AVX2, BMI2, AVX512, VNNI, SSE4.1/POPCNT or other targets.
The policy should therefore state that IJCCRL may normalise display names for readability, but it must preserve technical identity in the record. The public table can remain clean; the archive, download pack or audit note should keep the full technical details.
Update policy during events
A rating list also needs an update policy.
If an engine is updated during a tournament, the rating evidence becomes more complex. Is the engine still the same participant? Did the update fix a crash? Did it change playing strength? Did it change network files, evaluation, search, book behaviour or time management?
TCEC publishes explicit engine-update logic in its rules, including conditions around updates, division deadlines and risk when authors submit new versions. IJCCRL does not need to copy that structure, but it should have its own explicit rule.
A practical IJCCRL update policy could be:
For a closed rating event, the engine version should be frozen before the start of that event.
Emergency updates should be allowed only for serious technical failures such as crashes, UCI communication problems, illegal moves or reproducible startup failures.
If an emergency update is accepted, the archive entry must say so.
If a strength-changing update is submitted, it should normally wait for the next event or rating cycle.
If an engine changes major architecture, network, book integration or evaluation system, it should be treated as a new version and not silently merged with the prior one unless the rating policy explicitly allows version aggregation.
This protects the list from invisible version drift. It also protects authors, because every participant knows the same rule before the event begins.
When an engine should remain provisional
Not every engine should immediately receive a final rating status.
An engine should remain provisional when its evidence is incomplete, unstable or not yet comparable with the main pool. Provisional status is not a punishment. It is an honesty label.
An engine should be provisional if:
- it has too few games;
- it has not completed a mirrored opening cycle;
- it has not completed the event stage assigned to it;
- it entered as a private or author-submitted build;
- its public release identity is unclear;
- its version changed during the event;
- it suffered repeated crashes or UCI communication failures;
- its book, network or tablebase behaviour is under audit;
- its track assignment is unresolved.
TCEC uses temporary rating logic for new engines before official ratings are calculated after event participation. That general idea is useful: temporary or provisional ratings can help readers follow an event, but they must not be confused with final ratings.
For IJCCRL, a provisional label should be visible in three places: the rating table, the event page and the archive entry. If a provisional rating later becomes official, the transition should be documented.
Engine inclusion and tournament evidence
Inclusion rules should not live separately from the evidence chain.
An engine should not be included in a rating list merely because its name is interesting or its claimed Elo is high. The event record should support the inclusion. That means the list should connect to:
- the event where the games were played;
- the PGN pack or game archive;
- the rules and audit framework;
- the winners page if the event is closed;
- the archive entry for historical preservation;
- the rating list where the result is interpreted.
This is where IJCCRL has a useful editorial advantage. Its workflow already separates live broadcast, events, downloads, winners, archive and ratings as different publication surfaces. The TXT context explicitly defines those surfaces as separate functions rather than one mixed feed.
That separation should be preserved. The rating list should not become a blog. The blog should not become the official rating table. The archive should not become a duplicate downloads page. Each surface should answer a specific question.
Originality claims and editorial restraint
Engine originality is a sensitive topic. A public rating list should not casually make legal or moral claims about originality unless it has evidence, expertise and a defined process.
CCRL’s restraint is instructive: it states that its role is not to decide originality and that viewers must make their own judgements. IJCCRL should adopt the same caution.
This does not prevent IJCCRL from separating tracks. Track separation is not an accusation. It is a measurement design decision.
A safe editorial formulation is:
“Engines are assigned to IJCCRL tracks for publication clarity. Track placement is not a legal judgement on originality, licensing, copyright or authorship. It defines the competitive context in which IJCCRL interprets results.”
That sentence protects the project. It says what IJCCRL controls: its own categories. It avoids claims IJCCRL cannot prove.
Recommended IJCCRL inclusion checklist
Before an engine enters an IJCCRL rating surface, the following checklist should be completed.
1. Engine identity
Name, version, author or maintainer where known, public/private status, binary architecture and release date should be recorded.
2. Protocol
The engine must support the required protocol for the event, normally UCI unless a specific event allows another protocol.
3. Track assignment
The engine must be assigned to Original UCI Track, Derived Stockfish Track, or another explicitly defined category.
4. Release status
The engine must be labelled as public release, private author build, internal test build or provisional.
5. Version freeze
The exact tested version must be frozen for the event or stage unless an emergency update is approved and documented.
6. Configuration
Threads, hash, tablebase access, book policy, contempt settings, learning settings and relevant UCI options should be controlled or recorded.
7. Opening policy
The engine must play under the same opening policy as the event pool. If mirrored openings are used, both sides of each opening unit should be tracked.
8. Stability
The engine should pass startup, UCI readiness, legal move and time-management checks before entering a rating-valid event.
9. Evidence
PGN, result files, scheduler state, event metadata or equivalent support material should be preserved.
10. Rating status
The engine should be marked provisional until the minimum evidence threshold is met.
11. Archive status
When the event closes, the engine’s participation should be traceable through Archive, Downloads, Winners where applicable, and Rating Lists.
12. Non-claim rule
IJCCRL should avoid legal originality claims unless supported by explicit documentation. Track labels are publication categories, not court judgments.
How inclusion rules protect readers
Readers need to know what they are reading.
If a table says that Engine A is above Engine B, the reader should be able to ask: Were both engines public? Were both tested under the same time control? Were they from the same track? Were they stable? Were they updated during the event? Were their games mirrored? Were tablebases enabled? Was a book used? Were the games downloadable?
Inclusion rules make those questions answerable.
Without inclusion rules, the rating list becomes a scoreboard without a constitution. With inclusion rules, the list becomes a controlled publication object.
How inclusion rules protect engine authors
Engine authors also benefit from clear inclusion rules.
A public policy prevents arbitrary treatment. Authors know whether private builds are allowed, whether emergency updates are possible, how derived engines are labelled, how architecture names are displayed, and when ratings become official.
It also reduces unnecessary conflict. If an engine is labelled provisional, the reason should be procedural, not personal. If it is placed in the Derived Stockfish Track, that should reflect IJCCRL’s category system, not a public accusation. If it is excluded from a rating surface, the exclusion should be tied to documented criteria.
A serious rating list should be strict, but it should also be fair.
How inclusion rules protect IJCCRL
For IJCCRL, inclusion rules are a credibility shield.
The project already operates multiple surfaces: live events, rating tables, downloads, archive entries, winners pages and methodology pages. That structure is powerful, but only if the categories remain clean.
If original engines, derived engines, private builds and unstable test builds are all mixed without labels, the site becomes vulnerable to confusion. If every table, event and archive entry carries clear inclusion status, the site becomes easier to defend publicly.
This matters especially for OpenChess, TalkChess-style discussions and international computer-chess readers. These audiences are technical. They will notice unclear labels. They will also respect a policy that states limits honestly.
Recommended IJCCRL policy statement
IJCCRL can publish the following policy statement inside Rules & Audit and link to it from rating posts:
“IJCCRL rating lists are track-specific publication surfaces. Engine inclusion is governed by release status, protocol compatibility, version identity, event stability, track assignment and available evidence. Public releases, private author builds and provisional test builds may be treated differently. Original UCI engines and Stockfish-derived engines are separated where necessary to preserve interpretability. Track assignment is an editorial and methodological classification for IJCCRL events; it is not a legal judgement on originality, copyright or authorship. Ratings become official only after the required event evidence, PGN records and audit checks are complete.”
That paragraph is strict enough to be useful and restrained enough to avoid overclaiming.
Conclusion
Engine inclusion rules decide what a computer chess rating list actually measures. Elo values are only meaningful inside a defined population of engines, versions, tracks and evidence. Without inclusion rules, a rating list may still look precise, but its interpretation becomes fragile.
For IJCCRL, the correct policy is separation, traceability and restraint. Public releases should be distinguished from private builds. Original UCI engines should be separated from Stockfish-derived engines when the purpose of the list requires it. Version identity should be preserved. Emergency updates should be documented. Provisional status should be visible. Rating publication should be connected to Events, Rules & Audit, Downloads, Archive, Winners and Live Broadcast.
This does not replace CCRL, TCEC or any established computer-chess resource. It defines how IJCCRL should explain its own lists. A rating table is stronger when readers know exactly which engines were allowed into it, why they were allowed, and what kind of evidence supports the result.
Sources / References
- CCRL 40/15 index and About page: CCRL publishes testing summary, games, Bayeselo computation, category legend and comments on engine inclusion; its About page also states that CCRL does not decide engine originality.
- TCEC archive and rules information: TCEC presents live/archive surfaces, event structure, stage rules, temporary ratings, engine updates, hardware, adjudication and engine-specific configuration.
Jorge Ruiz Centelles
Filólogo y amante de la antropología social africana