Chess Engine Tournaments in cutechess-cli

Introduction

Creating and managing automated chess engine tournaments is a vital skill for developers, researchers, and enthusiasts aiming to evaluate engine strength, tune parameters, or simply enjoy high-quality engine-versus-engine play. The cutechess-cli tool, part of the Cute Chess suite, offers a robust command-line interface for orchestrating complex tournaments with minimal manual intervention. This tutorial guides you through every step—from installing the tool on both Windows and Linux to configuring tournament formats, engines, time controls, managing the engines.json file, and automating your tournaments with batch scripts.

Cute Chess comprises three main components:

• cutechess, a graphical user interface,
• cutechess-cli, the command-line interface for automated engine matches,
• libcutechess, a library for embedding Cute Chess functionality into other applications.

Since its inception in 2008, Cute Chess has been under active development, with contributions from Ilari Pihlajisto, Arto Jonsson, and the open-source community. The command-line utility cutechess-cli specifically caters to users who prefer scripting and automation over a graphical front end. Its feature set enables:

• Multiple tournament formats, including Round-Robin, Gauntlet, Knockout, Pyramid, and more.
• Flexible engine configuration, supporting UCI and XBoard protocols.
• Advanced time controls, spanning classical, blitz, and increment formats.
• Adjudication options, such as draw by repetition, resignation thresholds, tablebase adjudication, and SPRT stopping criteria.
• Integration with opening books, PGN/EPD input, and Polyglot book access.
• Result output in PGN, EPD, or custom result formats.

In this article, you will learn:

1. How to install cutechess-cli on Windows and Linux (Debian/Ubuntu).
2. Which tournament scenarios are available and how to configure them.
3. Engine options and time control settings to fine-tune engine behaviour.
4. Managing the engines.json file for preconfigured engine definitions.
5. Creating batch scripts to launch tournaments with a single command.
6. Search Engine Optimisation (SEO) best practices integrated throughout.

By the end of this comprehensive guide, you will be equipped to set up sophisticated chess engine tournaments entirely from the command line, fully leveraging cutechess-cli’s capabilities to generate reliable, reproducible, and extensible match data.

Installation

Before running tournaments, you must install cutechess-cli. We cover two primary platforms:

• Windows (binaries or build from source via MinGW/Visual Studio),
• Linux (Debian/Ubuntu via package manager or build from source).

Version Reference

We will use cutechess-cli version 1.3.1, released on 30 July 2023. This version includes important fixes for Windows release builds and crash handling with SIGINT.

Installing on Debian / Ubuntu

For Debian-based distributions, you have two options: use the system package (if available) or build from source.

Option 1: System Package (if available)

1. Update repositories: sudo apt update
2. Install cutechess-cli: sudo apt install cutechess-cli
3. Verify installation: cutechess-cli --version You should see output similar to cutechess-cli 1.3.1.

Note: The packaged version in your distribution may lag behind the latest release. If 1.3.1 is not available, proceed to Option 2.

Option 2: Building from Source

1. Install build dependencies: sudo apt install git build-essential cmake qtbase5-dev qtbase5-dev-tools libqt5svg5-dev
2. Clone the repository: git clone https://github.com/cutechess/cutechess.git cd cutechess
3. Create a build directory and compile: mkdir build && cd build cmake .. make sudo make install
4. Optional (generate manuals): sudo apt install mandoc mandoc -Tascii ../docs/cutechess-cli.6 | col -b > ~/cutechess-cli.6.txt mandoc -Thtml -Ostyle=man-style.css ../docs/cutechess-cli.6 > ~/cutechess-cli.6.html
5. Verify: cutechess-cli --version

This process installs the cutechess-cli binary in /usr/local/bin, making it available system-wide .

Installing on Windows

On Windows, you can either download precompiled binaries or build from source using MinGW or Visual Studio.

Option 1: Precompiled Binaries

1. Download the ZIP archive for version 1.3.1 from the Releases page on GitHub (look for assets named cutechess-cli-win.zip).
2. Extract the archive to a directory, e.g. C:\cutechess.
3. Add to PATH:
   • Search for Environment Variables.
   • Edit the Path system variable and add C:\cutechess.
4. Verify in Command Prompt: cutechess-cli --version

Option 2: Build from Source via MinGW

1. Install Qt (download the Qt Online Installer and select MinGW component).
2. Install MinGW and Git if not already present.
3. Clone the repository: git clone https://github.com/cutechess/cutechess.git cd cutechess
4. Open CMakeLists.txt in Qt Creator (installed with Qt).
5. Configure and Build:
   • Use the default MinGW kit.
   • Click Build → Build Project “cutechess”.
6. Copy cutechess-cli.exe from the build output directory to your desired location.
7. Add that location to your PATH for easy use .

Tournament Scenarios

cutechess-cli supports several tournament types, each suited to different evaluation needs:

• Round-Robin: Every engine plays against every other engine in a fixed number of games. The default format, ensuring full pairwise comparison.
• Gauntlet: One or more “seed” engines face the rest in one-way matches—ideal for testing a challenger against a field of opponents.
• Knockout: Single-elimination bracket; losers are eliminated each round until one champion remains.
• Pyramid: Each engine challenges all of its predecessors, creating a hierarchical competition.

You specify the format via -tournament <type>:

cutechess-cli \
  -engine cmd=stockfish proto=uci \
  -engine cmd=crafty proto=xboard \
  -tournament round-robin \
  -games 2 \
  -rounds 5

This example will run a round-robin with 2 games per encounter (one with white, one with black) over 5 rounds.

Gauntlet

In a Gauntlet:

cutechess-cli \
  -engine seeds=1 cmd=stockfish proto=uci \
  -engine cmd=komodo proto=uci \
  -engine cmd=crafty proto=xboard \
  -tournament gauntlet \
  -games 4

• seeds=1 designates stockfish as the seed.
• Each of the other engines will play 4 games against stockfish.

Knockout

For a Knockout bracket:

cutechess-cli \
  -engine cmd=stockfish \
  -engine cmd=komodo \
  -engine cmd=crafty \
  -engine cmd=leela \
  -tournament knockout \
  -games 6 \
  -rounds 1

• rounds=1 indicates single elimination (each pairing plays 6 games).

Pyramid

To run a Pyramid:

cutechess-cli \
  -engine cmd=stockfish \
  -engine cmd=komodo \
  -engine cmd=crafty \
  -engine cmd=leela \
  -tournament pyramid \
  -games 2

• Engine 2 plays Engine 1, Engine 3 plays Engines 1 and 2, Engine 4 plays Engines 1, 2, and 3.

Each format can be fine-tuned with additional options:

• -reverse to swap colours in the schedule,
• -repeat n for repeated openings with colour swaps,
• -site and -event to tag PGN output,
• -sprt for SPRT termination criteria in two-engine matches,
• -ratinginterval and -outcomeinterval to control console output frequency.

Configuring Engines and Time Controls

Each engine is added with -engine engine-options. Key options include:

Time controls (-tc) can be expressed in either:

• moves/time+increment, e.g. tc=40/300+5 for 40 moves in 5 minutes plus 5-second increment,
• infinite: tc=inf.

To apply the same time control or options to all engines, use -each:

cutechess-cli \
  -engine cmd=stockfish proto=uci \
  -engine cmd=komodo proto=uci \
  -each tc=40/120+2 \
  -each option.Threads=4

Managing the engines.json File

Rather than specifying each engine on the command line, you can define them in an engines.json configuration file. This file follows a simple structure:

[
  {
    "name": "Stockfish 15",
    "cmd": "stockfish",
    "proto": "uci",
    "option.Threads": "8"
  },
  {
    "name": "Komodo 14",
    "cmd": "komodo",
    "proto": "uci",
    "option.Threads": "4"
  }
]

Steps to use engines.json

1. Create engines.json in your working directory.
2. Populate with engine definitions as shown above.
3. Invoke cutechess-cli with -engine conf=<name>: cutechess-cli \ -engine conf="Stockfish 15" \ -engine conf="Komodo 14" \ -each tc=40/120+2 \ -tournament round-robin \ -rounds 10 \ -games 2
4. Benefits:
   • Cleaner command line—no long engine definitions.
   • Reusability—share engines.json across projects.
   • Version control—track engine parameter changes over time.

For advanced setups, you can reference multiple JSON files or override JSON options with command-line flags.

Creating Batch Scripts

Automate tournament execution by wrapping your command in a batch file (Windows) or shell script (Linux/macOS). This ensures repeatability and integrates with schedulers.

Windows Batch File

Create run_tournament.bat:

@echo off
REM Tournament: Stockfish vs Komodo
set CLI="C:\cutechess\cutechess-cli.exe"
set ENGINES="C:\tournaments\engines.json"
set OUT="C:\tournaments\results\sf_vs_komodo.pgn"

%CLI% ^
  -engine conf="Stockfish 15" ^
  -engine conf="Komodo 14" ^
  -each tc=40/120+2 ^
  -tournament round-robin ^
  -rounds 5 ^
  -games 2 ^
  -pgnout %OUT% ^
  -site "MyLab" ^
  -event "Benchmark 2025"

• Save the file and double-click to run.
• Logs appear in the console; PGN saved to %OUT%.

Linux Shell Script

Create run_tournament.sh:

#!/usr/bin/env bash
# Tournament: Stockfish vs Komodo

CLI="cutechess-cli"
ENGINES="./engines.json"
OUT="./results/sf_vs_komodo.pgn"

$CLI \
  -engine conf="Stockfish 15" \
  -engine conf="Komodo 14" \
  -each tc=40/120+2 \
  -tournament round-robin \
  -rounds 5 \
  -games 2 \
  -pgnout "$OUT" \
  -site "MyLab" \
  -event "Benchmark 2025"

Make it executable:

chmod +x run_tournament.sh

Run it:

./run_tournament.sh

Integration Tips

• CRON Jobs: Schedule periodic tournaments.
• CI Pipelines: Integrate into GitLab CI, GitHub Actions, or Jenkins.
• Logging: Redirect stdout/stderr to timestamped log files.
• Parameterisation: Pass configuration variables as script arguments.

Conclusion

By following this guide, you have learned how to:

• Install cutechess-cli on both Windows and Linux, leveraging system packages or building from source.
• Choose from multiple tournament formats—Round-Robin, Gauntlet, Knockout, Pyramid—to suit your evaluation goals.
• Configure engines with protocol, time controls, book options, and UCI parameters for precise control over match conditions.
• Manage an engines.json file to centralise engine definitions and streamline command-line invocations.
• Craft batch scripts to automate tournament runs, integrate with scheduling tools, and ensure reproducible results.

The ability to script chess engine tournaments empowers you to conduct extensive engine testing, parameter tuning, and performance benchmarking with ease. Whether you are a researcher seeking statistical significance, a developer fine-tuning engine options, or an enthusiast setting up friendly engine matches, cutechess-cli delivers robust features and flexibility.

For further reading and advanced topics, consult the following resources:

Bibliography

• Cute Chess Repository: The source code and issue tracker for Cute Chess.
• Cute Chess Releases (v1.3.1): Binary downloads and release notes for cutechess-cli 1.3.1.
• Cute Chess Wiki – Building from Source: Detailed build instructions for Debian/Ubuntu, macOS, and Windows.
• Chess Programming Wiki – cutechess-cli: Overview of cutechess-cli design and requirements.

Jorge Ruiz Centelles

Filólogo y amante de la antropología social africana

SÍGUEME