Skip to content
Portada » Methodology

Methodology

Methodology — How IJCCRL Builds Its Chess Engines Rating Lists

At IJCCRL we publish transparent, reproducible chess engines rating lists across multiple time-control divisions (e.g., Bullet and Blitz/Extended). This page documents the hardware, software, tournament format, openings policy, and data output used to compile the lists you see on pages like our Blitz and UltraBullet divisions.

1) Hardware & Software Environment

  • Host: HP ProLiant DL360p Gen8 (consistent across events)
  • OS: Windows 10 Pro Workstation (stable, no foreground load during runs)
  • Tablebases: Syzygy 5-piece (-tb C:\Syzygy -tbpieces 5)
  • GUI/Runner: CuteChess-CLI
  • Build family: We standardise on SSE3/SSE4/POPCNT binaries per engine where available, so that all engines in a given list share a comparable instruction-set baseline.
  • Threads & Hash: 1 thread (option.Threads=1) and 16 MB Hash (option.Hash=16) for every engine in the list, ensuring fairness and stable variance across pairings.
  • Parallelism: -concurrency 2 (two games in parallel) on the same host; no CPU pinning or NUMA affinity tricks are used, so conditions remain identical across engines.

Why this matters: fixing threads, hash, tablebases, and instruction set removes common sources of rating noise. Concurrency is modest to avoid thermal throttling and shared-resource contention.

2) Tournament Format (Round-Robin, Balanced Colors)

We use round-robin tournaments in CuteChess with balanced colors and multiple passes to increase statistical reliability.

  • Mode: -tournament round-robin
  • Color balance: -games 2 (each pairing plays two games, one as White and one as Black)
  • Passes: -rounds 7 -repeat 2 → multiple cycles of the schedule to reduce variance
  • Recovery: -recover (automatically resumes after interruptions)
  • Move cap (used in ultra-fast divisions): -maxmoves 200 to prevent pathological marathons

Effective sample depth: with -rounds 7, -repeat 2, and -games 2, each pair accrues up to 28 color-balanced games per division (7 rounds × 2 games × 2 repeats), providing a solid base for rating stability in fast time controls.

3) Time Controls (per Division)

We publish separate lists per TC to respect how engines scale with time:

  • Ultra-fast / Bullet division
    • Example config: tc=30+0.1 (30 seconds + 0.1s increment)
    • Rationale: highlights tactical sharpness, latency handling, and stability under clock pressure.
    • We also set -maxmoves 200 here to cap rare endgame stalls.
  • Blitz / Extended division
    • Current season config: tc=120+0.1 (120 seconds + 0.1s increment) under the “Blitz/Extended” umbrella.
    • Some historical posts refer to “180+2” in titles; season presets may vary by cycle, but the methodology (equal hardware, openings, threads=1, hash=16, round-robin, etc.) is unchanged.

Each rating list only aggregates results from its own TC, never mixing Bullet, Blitz, or Extended. This prevents cross-TC contamination.

4) Openings Policy (UHO — Uniform & Reproducible)

We use fixed UHO opening suites with 8-move books to ensure diverse, balanced, and repeatable starting positions:

  • Openings files (examples from our configs):
    • UHO_2022_8mvs_big_+100_+129.pgn
    • UHO_2024_8mvs_big_+100_+119.pgn
  • Invocation:
    • -openings file=..\Openings\PGN\UHO_XXXX_8mvs_... .pgn format=pgn order=random
  • Why UHO? Uniform coverage of ECOs, fixed depth (8 moves), and randomness in order (not content) prevent over-fitting to a tiny opening subset and keep the lists comparable year-to-year.

5) Engines, Builds & Fairness

  • Each engine is added via CuteChess engine definitions (-engine conf="EngineName ...") pointing to SSE3/SSSE3/SSE4/POPCNT builds when available, with identical Threads=1 and Hash=16.
  • NNUE nets (when applicable) are those shipped by the engine authors for the chosen build or the season’s published default.
  • No engine receives a custom book or special parameters other than the global constraints listed above.

6) Output, Checkpoints & Datasets

  • PGN export: -pgnout ..\Games\nnue_league_2025.pgn (complete game records)
  • Rating checkpoints: -ratinginterval 10 (we record incremental standings every 10 games to monitor drift)
  • Tablebases: only probing up to 5 pieces (-tbpieces 5) keeps runs uniform and avoids build-specific 6/7-man discrepancies.
  • Draw/Resign adjudication: we rely on game outcomes as played; hard adjudication flags are not forced in the current configs, minimising adjudication bias in fast TCs.

Where possible, we publish CSV/JSON extracts alongside the lists so that third parties can filter by engine, version, and division and reproduce Elo aggregates externally.

7) Reproducibility — Reference CuteChess Configs

Below are sanitised versions of the scripts we actually use to run current season divisions. Replace engine names with your local engine configs and keep the global options identical to match IJCCRL conditions.

7.1 Bullet / Ultra-fast Division (SSE3)

@echo off
set opening=UHO_2022_8mvs_big_+100_+129
@cutechess-cli.exe ^
  -event "Cup season Ijcrl 2025" -site "HP Proliant DL360P Gen8 Server" ^
  -engine conf="Alexandria 8.1.0" ^
  -engine conf="Clover.9.0-old" ^
  -engine conf="integral_sse41_popcnt" ^
  -engine conf="RubiChess-20240817" ^
  -engine conf="seer_v2.8_x64" ^
  -engine conf="stormphrax_7.0.0" ^
  -engine conf="Titan-x64" ^
  -engine conf="velvet-v8.1.1" ^
  -tournament round-robin ^
  -each tc=30+0.1 option.Hash=16 option.Threads=1 -tb C:\Syzygy -tbpieces 5 ^
  -openings file=..\Openings\PGN\%opening%.pgn format=pgn order=random ^
  -concurrency 2 -rounds 7 -games 2 -maxmoves 200 ^
  -repeat 2 -recover ^
  -pgnout ..\Games\nnue_league_2025.pgn ^
  -ratinginterval 10

7.2 Blitz / Extended Division (SSE3)

@echo off
set opening=UHO_2024_8mvs_big_+100_+119
@cutechess-cli.exe ^
  -event "Cup season Ijcrl 2025" -site "HP Proliant DL360P Gen8 Server" ^
  -engine conf="stockfish 17.1" ^
  -engine conf="berserk_20250606_64_ja_sse4" ^
  -engine conf="caissa-1.23-x64-sse4-popcnt" ^
  -engine conf="dragon-3.3-64bit" ^
  -engine conf="Ethereal-14.25-ssse3" ^
  -engine conf="Clover.9.0-old" ^
  -engine conf="RubiChess-20240817_x86-64-sse3-popcnt" ^
  -engine conf="stormphrax_700_64_ja_sse4" ^
  -engine conf="Obsidian 16.14" ^
  -tournament round-robin ^
  -each tc=120+0.1 option.Hash=16 option.Threads=1 -tb C:\Syzygy -tbpieces 5 ^
  -openings file=..\Openings\PGN\%opening%.pgn format=pgn order=random ^
  -concurrency 2 -rounds 7 -games 2 ^
  -repeat 2 -recover ^
  -pgnout ..\Games\nnue_league_2025.pgn ^
  -ratinginterval 10

Important: If you alter Threads, Hash, openings file, TC, or tbpieces, your results will no longer be directly comparable to IJCCRL lists. For cross-checking, keep these constants identical.

8) How We Aggregate and Publish Ratings

  • We use the round-robin results emitted by CuteChess to produce Elo standings per division.
  • Ratings are reported with error margins (±), games count, and LOS when available; lists are updated after each full pass so engines can move up or down based on fresh evidence.
  • We do not merge results across different TCs or across fundamentally different hardware. Each division is a clean slice with identical constraints.

9) Quality Control & Anti-Bias Guidelines

  • Identical conditions per division: same host, same TC, same Hash/Threads, same openings file.
  • Multi-pass scheduling (-rounds + -repeat) lowers variance, especially at fast TCs.
  • No engine-specific books/flags beyond the global constraints.
  • Crash-safe: -recover ensures we don’t lose data to power or session interruptions.
  • Open data: PGNs are exported and made available; CSV/JSON datasets accompany rating tables so others can re-compute and verify.

TL;DR for Readers

  • Fairness: one thread, 16 MB hash, identical tablebases (5-man), same UHO openings.
  • Format: round-robin, color-balanced, multi-pass schedule.
  • Divisions: Bullet (e.g., 30+0.1) and Blitz/Extended (e.g., 120+0.1).
  • Outputs: full PGNs plus rating snapshots; CSV/JSON when available for independent analysis.