Predicting Chess Outcomes: A Disciplined ML Walkthrough

Most machine-learning tutorials pick a problem where a model wins. This one is more useful because the model barely does. The task is to predict the result of a chess game from its opening, using the public Lichess database, and the answer is that the first ten moves add little to what the players’ ratings already say.

That result is only worth something if it is honest, and making it honest is the actual work. This post rebuilds the whole study in nine scripts and two shared modules: an audit of the data, a leakage contract that tests enforce, baselines strong enough to lose to, seven different splits, diagnostics that say why the model stops where it does, and one experiment with a chess engine that shows what kind of information was missing.

Table of contents

The concept: four questions and one rule

The task is small and fixed in advance. From a pool of real games, predict four things:

1. White’s expected score before the game, from the ratings and the time control only.
2. White’s expected score after 3 moves, adding the opening.
3. White’s expected score after 10 moves, adding the early middlegame.
4. Both players’ ratings after 10 moves, from the time control and the moves, without ever seeing a rating.

The first three are one question asked at three points in the game, which makes them an experiment: how much does watching more of the game help? The fourth reverses it, predicting strength from play.

The expected score counts a win as 1, a draw as 0.5 and a loss as 0. One rule governs all four models: a model may use only what would be known at the moment it predicts, and is scored only on games it never trained on. The pre-game model may not see a move; the rating model may not see a rating. That rule is the leakage contract, and below it becomes code.

The data: Lichess, and the audit before the model

Every choice about the data is fixed before any model runs: one month (January 2015), one time control (games Lichess labels Blitz), rated games only, the first 100,000 in file order. The month comes from the Lichess dataset on Hugging Face, a parquet copy of the database dumps, downloaded at a pinned revision. Everything runs on a CPU; on an Apple M4 laptop the whole study, in the README’s order, takes about 16 minutes. data.py keeps the games and prints an audit:

January 2015: 1,497,237 rated games, 648,250 blitz; kept the first 100,000
they span 2014-12-31 23:00:06 to 2015-01-05 21:02:21 UTC: 4.9 of the month's 31 days
results: White wins 50.0%, draws 3.8%, Black wins 46.2%; White's mean score 0.519
ratings missing: 0

how games ended, and how many moves they lasted
     termination   games  median moves  no moves  under 3 moves
       Abandoned   1,891             0       990         100.0%
          Normal  70,079            31       740           2.1%
Rules infraction       6             5         0           0.0%
    Time forfeit  28,024            39        25           2.0%
abandoned games: White's mean score 0.476, decisive 100.0%, moves played 901
9,626 players; the 1,000 most active play in 76% of games
games reaching move 3: 96.1%
games reaching move 10: 93.7%

Full script: data.py

White scores 0.519 on average, a small first-move edge, and draws are only 3.8% of blitz games. Three findings in the audit change what the models are allowed to do.

Games without play are not chess. 1,891 games ended as “Abandoned”, 990 of them with no move at all, and together they contain 901 moves, under one per game. Yet every one has a decisive result, with one side scored as the loser. Another 765 games labelled “Normal” or “Time forfeit” also have no move. These results record who failed to start, not who played better, and no feature of the board can predict them. So a game counts as played when each side has made at least one move, which features.py finds true of 96.9% of games and which removes every abandoned one. All models train and score on played games only. The rule changes only the pre-game model’s games, since the others need several moves anyway, and a deployed pre-game model could not know in advance that a game will never start: its real error would include these games.

The first 100,000 games are not a sample of the month. They span 4.9 of January’s 31 days, starting late on 31 December UTC. Any effect of weekends, holidays or tournaments in that window is part of the data, so the numbers below describe those days, not January. The same 9,626 players fill all 100,000 games, and the 1,000 most active play in 76% of them, which matters for how the games are split below.

The ratings are Glicko-2, not Elo. Lichess rates players with “the Glicko-2 rating method developed by Mark Glickman”, according to its FAQ, and the dataset card says the rating columns “contain Glicko2 ratings”. The Elo formula used as a baseline below is an approximation for these numbers, which is one more reason to refit it.

Not every game reaches every point. Games that end earlier have no position to describe, so a model for move $k$ is trained and scored only on games that reach it, and abstains on the rest: 96.0% reach move 3 and 93.4% reach move 10. These are slightly below the audit’s 96.1% and 93.7% because the audit counts a move as reached once White has started it, while features.py below counts plies and needs both sides to have finished it.

Features: replaying the opening

Board features come from replaying each game with python-chess. A ply is one move by one side, so move 3 is 6 plies and move 10 is 20. features.py records material (pawns 1, knights and bishops 3, rooks 5, queens 9), captures, checks, castling, developed minor pieces and queen moves for each side, after ply 6 and after ply 20:

def replay(movetext):
    """Board features after 6 and 20 plies, the first move, and the move-10 FEN."""
    sans = [t.rstrip("?!") for t in re.sub(r"\{[^}]*\}", "", movetext).split()
            if not re.match(r"^(\d+\.+|1-0|0-1|1/2-1/2|\*|\$\d+)$", t)]   # drop numbers, results, NAGs
    board = chess.Board()
    counts = {k: {chess.WHITE: 0, chess.BLACK: 0} for k in ("cap", "chk", "castle", "queen")}
    out = {"first_move": sans[0] if sans else None, "plies": len(sans), "m3": None, "m10": None,
           "fen10": None}
    for ply, san in enumerate(sans[:20], start=1):
        move = board.parse_san(san)
        side = board.turn
        counts["cap"][side] += board.is_capture(move)
        counts["castle"][side] |= board.is_castling(move)
        counts["queen"][side] += board.piece_type_at(move.from_square) == chess.QUEEN
        board.push(move)
        counts["chk"][side] += board.is_check()
        if ply == 6:
            out["m3"] = snapshot(board, counts)
        if ply == 20:
            out["m10"] = snapshot(board, counts)
            out["fen10"] = board.fen()

Full script: features.py

100,000 games, 37 columns
games with a move by each side (2+ plies): 96.9%; abandoned among those: 0
move-3 features present: 96.0%; move-10: 93.4%
first moves: e4 59.2%, d4 25.0%, other 3.3%, c4 3.2%, Nf3 2.7%

The first move is kept as a category, with moves outside a list of ten common first moves grouped as “other”. The position after move 10 is also stored in FEN, a one-line text description of a board, for the engine experiment at the end.

Encoding the leakage contract as code

Data leakage is information reaching a model during training that it would not have when predicting. It produces scores that look excellent and collapse in use, and the usual defence, care, does not survive the next person adding a column. The defence here is structural: each model’s allowed columns live in one place, and the training code takes a model’s name and looks its columns up there; it has no other way to receive columns.

FEATURES = {
    "win_pregame": RATINGS + TIME,
    "win_move3": RATINGS + TIME + M3,
    "win_move10": RATINGS + TIME + M10,
    "win_move10_engine": RATINGS + TIME + M10 + ["m10_engine"],  # Stockfish on the move-10 board
    "elo_move10": TIME + M10,                      # predicts elo_w and elo_b: no rating may enter
    "elo_time_only": TIME,                         # the same target, to see what each part adds
    "elo_moves_only": M10,
}
TARGET = {"win_pregame": "score", "win_move3": "score", "win_move10": "score",
          "win_move10_engine": "score", "elo_move10": ["elo_w", "elo_b"],
          "elo_time_only": ["elo_w", "elo_b"], "elo_moves_only": ["elo_w", "elo_b"]}
NEEDS = {"win_pregame": None, "win_move3": "m3_material_diff", "win_move10": "m10_material_diff",
         "win_move10_engine": "m10_material_diff",
         "elo_move10": "m10_material_diff", "elo_time_only": "m10_material_diff",
         "elo_moves_only": "m10_material_diff"}    # a game is eligible when this is not NaN
CATEGORICAL = ["first_move"]


def eligible(df, model):
    """Games a model is trained and scored on: played games that reach its horizon.

    Played means each side made at least one move; this drops every abandoned
    game and the games that ended with no move at all."""
    keep = df.plies >= 2
    if NEEDS[model]:
        keep &= df[NEEDS[model]].notna()
    return df[keep]

Full script: contract.py

As a table, a mark means the model may see that group:

The win models form a staircase: each horizon adds one block of move features. Each horizon is scored on a slightly different set of games, so it is the gain over a baseline on the same games that isolates what the newly visible moves add. The rating model has an empty ratings column, because ratings are its target; two more entries give it the time control alone and the moves alone, to see what each part contributes. The rules are tests, so they hold after any edit:

def test_pregame_sees_no_moves():
    assert not any(c.startswith(("m3_", "m10_")) or c == "first_move" for c in FEATURES["win_pregame"])


def test_move3_sees_nothing_after_move3():
    assert not any(c.startswith("m10_") for c in FEATURES["win_move3"])


def test_horizons_only_add_information():
    assert set(FEATURES["win_pregame"]) < set(FEATURES["win_move3"]) < set(FEATURES["win_move10"])


def test_engine_sees_only_the_move10_board():
    assert set(FEATURES["win_move10_engine"]) - set(FEATURES["win_move10"]) == {"m10_engine"}


def test_elo_model_sees_no_rating():
    for model, cols in FEATURES.items():
        if TARGET[model] == ["elo_w", "elo_b"]:
            assert not set(cols) & set(RATINGS), model

ok  test_elo_model_sees_no_rating
ok  test_engine_sees_only_the_move10_board
ok  test_horizons_only_add_information
ok  test_move3_sees_nothing_after_move3
ok  test_move_groups_are_disjoint_from_ratings
ok  test_no_model_sees_its_target
ok  test_no_outcome_or_identity_columns
ok  test_pregame_sees_no_moves
8 rules hold

Full script: test_contract.py

What the tests protect against is easy to show. leak.py makes the edit a well-meaning person might make: it adds elo_diff to the rating model’s entry. That column is not a rating, but with the difference between the two ratings the model only has to estimate one rating to know both. The test fails on the edited contract, and the script trains the leaked model anyway to show what it would have reported:

test_elo_model_sees_no_rating: FAILED on the edited contract
rating model as contracted  MAE  162.1 rating points
rating model with elo_diff  MAE  143.1 rating points

Full script: leak.py

The leaked model reports a 19-point better error, which would read as an improvement on any leaderboard. Run before training, as part of the workflow, the test stops it before the number exists.

The analysis: a table that makes a decision

An exploratory chart is worth making when its result would change what happens next. Here the question is narrow. Ratings will dominate any prediction, so a move feature earns its place only if it says something within a rating band: do two equally rated players’ positions predict the result? diagnose.py answers it on training games only, so that no decision is made by looking at the games used for scoring:

White's mean score by rating gap (rows) and material after move 10 (columns), training games
mat         <=-2    -1     0    +1  >=+2
gap                                     
Black +150  0.16  0.19  0.25  0.32  0.43
Black +50   0.34  0.38  0.41  0.48  0.57
even        0.45  0.49  0.52  0.57  0.70
White +50   0.54  0.61  0.62  0.68  0.81
White +150  0.72  0.75  0.79  0.84  0.91
games per cell:
mat         <=-2    -1      0    +1  >=+2
gap                                      
Black +150  1958  1636   7993   486   284
Black +50   1709  1618   9459   683   409
even        2432  2376  14418  1236   707
White +50   1543  1562   9655   945   600
White +150  1354  1205   8515  1060   982

Full script: diagnose.py

The strongest gradient runs down the table: the rating gap. Along the outlined row of equal ratings, though, the score still rises from 0.45 to 0.70 as material shifts, a signal the ratings cannot explain, so material after move 10 belongs in the model. Most games sit in the middle column, with material level after ten moves. The same question asked of castling gets a flat answer:

even ratings (|gap| <= 50): White's mean score by who had castled by move 10
                              mean  size
m10_castled_w m10_castled_b             
0.0           0.0            0.517  6536
              1.0            0.501  3587
1.0           0.0            0.527  4789
              1.0            0.524  6454

At equal ratings, White’s score is between 0.50 and 0.53 whoever has castled by move 10. Castling stays in the feature set, because the boosted trees can ignore a useless column, but it is not worth engineering further.

The training: strong baselines and honest metrics

Each target gets its own gradient-boosted tree model, scikit-learn’s HistGradientBoostingRegressor, given exactly the columns its contract allows. Boosted trees are a strong default for tabular data: they handle non-linear effects and interactions without hand-built features. The hyperparameters are fixed in advance and early stopping uses a slice of the training set, so the validation games are never used to choose anything.

The win models are scored with the Brier score, the mean squared difference between the predicted probability and the outcome, lower being better:

A model is only as good as the baseline it beats, and this problem has a strong one. The Elo expected score turns a rating difference into a probability (Glickman & Jones give it as their equation 1):

It has no parameters, which makes it a fair first baseline and an unfair last one. It assumes equal ratings mean an even game, while White scores 0.519 here, and the ratings in this data come from Glicko-2, for which the formula is only an approximation. The second baseline therefore keeps the same curve and fits its two numbers to the training games, a logistic regression on the rating difference:

def elo_formula(df):
    """Expected score for White from the rating difference alone, no parameters."""
    return 1 / (1 + 10 ** (-df.elo_diff / 400))


def elo_refit(train, valid, model):
    """The same curve with its scale and White's edge fitted: logistic regression on elo_diff."""
    tr, va = eligible(train, model), eligible(valid, model)
    w = np.column_stack([tr.score, 1 - tr.score]).ravel()           # a draw counts half a win
    x = np.repeat(tr[["elo_diff"]].to_numpy() / 400, 2, axis=0)
    y = np.tile([1, 0], len(tr))
    lr = LogisticRegression().fit(x, y, sample_weight=w)
    return lr.predict_proba(va[["elo_diff"]].to_numpy() / 400)[:, 1]

Full script: models.py

train.py splits the games 80/20 at random, trains the models, and reports each win model’s gain over the refitted curve with a 95% bootstrap interval (the validation games resampled 1,000 times). ECE, the expected calibration error, is the gap between the mean prediction and the mean actual score in each of ten probability bins, averaged with each bin weighted by its size; small means games predicted at 0.7 score about 0.7 on average.

split: 80,079 training and 19,921 validation games; each model then keeps the played games that reach its horizon

model         games Brier: mean     Elo Elo refit   model  gain over Elo refit (95% CI)    ECE
win_pregame  19,304      0.2391  0.2053    0.2043  0.2048      -0.0005 (-0.0008, -0.0003)  0.007
win_move3    19,107      0.2390  0.2051    0.2041  0.2045      -0.0005 (-0.0008, -0.0001)  0.010
win_move10   18,585      0.2388  0.2054    0.2044  0.2025      +0.0018 (+0.0011, +0.0026)  0.007

ratings from play: mean absolute error in rating points, both players
  training mean             187.1   (18,585 games)
  elo_move10                162.1   correlation of the pair's mean rating 0.53
  elo_time_only             173.9   correlation of the pair's mean rating 0.39
  elo_moves_only            174.5   correlation of the pair's mean rating 0.38

Full script: train.py

Read against the textbook formula, all three win models improve on it. Read against the refitted curve, the picture changes. The pre-game and move-3 models are slightly worse, with intervals that exclude zero: they found nothing the refitted curve misses, and their gain over the formula was White’s edge and a better scale. Only the move-10 model gains, 0.0018 Brier, and all three are well calibrated (ECE 0.007 to 0.010).

The rating model improves on guessing the average by 25 points, but the ablation shows the moves are only part of that: the time control alone reaches 173.9 and the moves alone 174.5. Players who choose longer games are, on average, rated differently, and the model uses that as much as the board.

One split could be lucky, and a random split of games is not how a model meets new data, so seeds.py repeats the comparison seven times: five random splits; a split by time, training on the first 80% of games by start time and scoring on the last 20%; and a split by player, which puts each player on one side at random and keeps only games whose two players are on the same side, so no validation game involves a player seen in training:

       split   train  valid   pregame    move 3   move 10  rating MAE: mean -> model
   random 42  80,079 19,921   -0.0004   -0.0002   +0.0019             187.1 -> 162.0
    random 1  80,045 19,955   -0.0001   -0.0001   +0.0027             187.6 -> 162.9
    random 2  80,019 19,981   -0.0001   -0.0001   +0.0022             188.0 -> 162.5
    random 3  79,738 20,262   -0.0002   -0.0002   +0.0025             188.9 -> 162.6
    random 4  79,980 20,020   -0.0005   -0.0004   +0.0022             187.6 -> 163.1
 random mean                  -0.0003   -0.0002   +0.0023
        time  80,000 20,000   +0.0001   -0.0000   +0.0027             189.2 -> 164.0
     players  43,050 11,606   -0.0011   -0.0009   +0.0015             179.1 -> 164.6

Full script: seeds.py

The win results hold on every split: no gain before the game or at move 3, a gain at move 10 of 0.0019 to 0.0027 on random splits, 0.0027 by time and 0.0015 by player. (Split 42 here differs slightly from train.py’s run because seeds.py seeds the booster with the split’s seed, 42, where train.py seeds it with 0.) The rating model is where the split matters. On random splits it gains about 25 points over the average; with players kept apart, 14.5 (179.1 to 164.6). Part of what looked like reading strength from play was recognising players already seen.

These checks narrow the study’s simplifications without removing them. There is no untouched test set: the diagnostics below, and the decision to try an engine, used the same validation games that later scored the engine. The engine’s gain is far from zero (95% CI +0.0049 to +0.0072), so that does not change the conclusion, but a stricter study would hold out a third set and touch it once. And all of it describes 4.9 days of blitz in one month.

Diagnose before enhancing

When a model stops improving, the reflex is to add features or a bigger model. The better first step is to ask why it stopped, because a model at the limit of what its inputs contain will only fit noise with more features. diagnose.py runs three checks on the move-10 model.

A learning curve. Training on growing subsets of the training games, three random subsets at each size, shows whether more data would help:

learning curve, win_move10: validation Brier (Elo refit on the full training set: 0.2044)
   games  mean of 3 subsamples (lowest, highest)
   2,500  0.2105 (0.2095, 0.2121)
   5,000  0.2070 (0.2067, 0.2075)
  10,000  0.2051 (0.2045, 0.2057)
  20,000  0.2038 (0.2036, 0.2041)
  40,000  0.2031 (0.2030, 0.2033)
  74,825  0.2026 (0.2025, 0.2027)

The model needs between 10,000 and 20,000 games just to match the refitted curve. After that, each step buys less: 0.0013 from 10,000 to 20,000 games, 0.0007 to 40,000, 0.0005 to 75,000, and the ranges of the three subsamples no longer overlap. The curve is flattening but has not stopped, so more games would help a little; they would not change the order of magnitude.

Permutation importance. Shuffling one column at a time across the validation games and measuring how much the Brier score worsens shows what the model relies on:

permutation importance (Brier increase when shuffled; share of the total)
  elo_diff               0.05950  90.0%
  m10_material_diff      0.00451   6.8%
  m10_developed_b        0.00046   0.7%
  elo_b                  0.00032   0.5%
  elo_mean               0.00025   0.4%
  m10_castled_w          0.00017   0.3%
  m10_developed_w        0.00016   0.2%
  m10_castled_b          0.00012   0.2%

The rating difference carries 90% of the importance and material 6.8%; every other board feature is under 1%. The moves are a thin layer on top of the ratings, and almost all of that layer is material.

Where the error lives. Splitting the validation error by segment shows whether the model fails somewhere specific:

validation Brier of the move-10 model, by |rating gap|
                  mean  size
gap                         
[0.0, 50.0)     0.2335  5081
[50.0, 150.0)   0.2238  7126
[150.0, 300.0)  0.1847  4188
[300.0, inf)    0.0956  2190
by material after move 10 (even vs not)
                     mean   size
m10_material_diff               
uneven             0.1951   6122
even               0.2062  12463

The error is largest in games between players rated within 50 points, 0.2335, close to the 0.25 of a coin flip, and smallest when the gap passes 300 points. Games with level material after ten moves are harder than games where someone is ahead. The model is not failing on a type of game it could learn; the remaining error sits in games that are close, which is what close means.

The one lever that helped

Material is a crude reading of a position, and the board features say nothing about who stands better when the pieces are level, or whether an apparent material edge is about to be traded back. A chess engine measures exactly that. engine.py asks Stockfish 18 for its evaluation of every move-10 position, at a fixed budget of 20,000 nodes with one thread, which makes each evaluation repeatable:

NODES = 20_000
CAP = 1_500                       # centipawns; a forced mate counts as a decisive +-1500


def evaluate(fens):
    engine = chess.engine.SimpleEngine.popen_uci(shutil.which("stockfish"))
    engine.configure({"Threads": 1, "Hash": 16})
    out = []
    for fen in fens:
        engine.protocol.send_line("ucinewgame")   # no hash carried over between games
        info = engine.analyse(chess.Board(fen), chess.engine.Limit(nodes=NODES))
        out.append(max(-CAP, min(CAP, info["score"].white().score(mate_score=100_000))))
    engine.quit()
    return out

Full script: engine.py

Stockfish 18, 20,000 nodes per position, 93,410 positions
evaluation after move 10, White's view: median +45 cp, |eval| > 200 cp in 22.2%

The evaluation describes the move-10 board and nothing later, so the contract gives it a model of its own, the move-10 model’s columns plus this one, with a test that nothing else is added. engine_ab.py trains both on the same five random splits:

split 42, 18,585 validation games: Brier Elo refit 0.2044, move-10 model 0.2025, with engine 0.1965 (ECE 0.007)
engine column's gain over the same model without it: +0.0060 (95% CI +0.0049, +0.0072)
  material even   12,463 games: Brier 0.2063 -> 0.2027
  material uneven  6,122 games: Brier 0.1948 -> 0.1838

split seed  gain over Elo refit: without  with engine  ratio
        42                       +0.0019      +0.0079   4.2x
         1                       +0.0027      +0.0089   3.3x
         2                       +0.0022      +0.0082   3.7x
         3                       +0.0025      +0.0083   3.3x
         4                       +0.0022      +0.0086   3.9x
      mean                       +0.0023      +0.0084   3.6x

Full script: engine_ab.py

One column raises the gain over the refitted curve from 0.0023 to 0.0084 on average, 3.6×, on every split, with the model still calibrated (ECE 0.007). It helps more where material is uneven (0.1948 to 0.1838) than where it is level (0.2063 to 0.2027; diagnose.py’s 0.2062 and 0.1951 come from a model with a different seed). A material count after ply 20 often catches an exchange halfway through, and the engine knows whether the extra piece will be recaptured. The lesson is not “use an engine”. When a feature set has reached its limit, a different kind of information helps more than more of the same kind. The engine stays an opt-in step, because it adds a binary dependency and minutes of compute.

Where this leads

• Measuring a Classifier Honestly covers the Brier score, calibration and threshold choice for classifiers in general.
• Features and Overfitting explains why more features on a fixed amount of data can make a model worse, the risk the diagnostics above guard against.
• Learning Is Optimization derives the logistic regression behind the refitted Elo baseline.

Takeaways

• Choose the baseline that is hardest to beat. Every model here beats the textbook Elo formula; against the same curve refitted to the data, the pre-game and move-3 models lose by 0.0003 and 0.0002 Brier, and the move-10 model gains 0.0023.
• Split more than one way before believing a small number. The move-10 gain appeared on all seven splits (0.0015 to 0.0027). The rating model’s gain fell from 25 points to 14.5 when players were kept apart.
• Make the leakage rules executable. One rating-derived column turned a 162-point error into a 143-point one; the contract test caught it.
• Audit before modelling. All 1,891 abandoned games had decisive results, and 765 others had no move at all; the first 100,000 games covered 4.9 days and 9,626 players.
• Diagnose before enhancing. The learning curve, a 90% importance share for ratings and error concentrated in close games all said the board features were near their limit. A different kind of information, an engine evaluation, then raised the gain 3.6×.

References

Start with Glickman and Jones for the Elo expected score that both baselines use, then the Lichess FAQ for how the ratings in this data are actually computed.

• Lichess open database: the monthly game dumps, and the Lichess/standard-chess-games parquet copy on Hugging Face (CC0) that data.py downloads.
• Lichess FAQ: ratings: Lichess’s Glicko-2 ratings.
• Glickman & Jones, Rating the Chess Rating System (Chance, 1999): the Elo system and its expected-score formula.
• Glickman, Example of the Glicko-2 System: the rating method Lichess uses.
• Brier, Verification of Forecasts Expressed in Terms of Probability (1950): the original Brier score.
• Ke et al., LightGBM: A Highly Efficient Gradient Boosting Decision Tree (NeurIPS 2017): the histogram-based boosting that scikit-learn’s HistGradientBoostingRegressor follows.
• scikit-learn, Permutation feature importance: the importance method used in diagnose.py.
• python-chess and Stockfish: the move replay and the engine evaluation.