Memory-constrained forecasting

When a forecasting system can keep only so many bytes in memory, what should those bytes be? Tested on spikes in electricity prices, where the rare worst hours are what matter.

  • Python
  • Polars
  • MLX
  • Parquet
Read the research report

Key numbers

  • 19%

    lower tail-forecast error at equal memory, best budget

  • 14×

    more raw history beaten, at the same bytes

  • 10 GiB → 91 MiB

    memory in the heaviest step after one fix

How it works

  1. Electricity prices329M rows, as published
  2. Fixed set of nodes499, present every year
  3. Memory budgetsame bytes either way
  4. Learned vectorsa few slots, 14 wide
  5. Raw history14× more slots
  6. Tail-risk readout99th percentile
  7. Pre-registered testswritten before scoring
Prices are stored with the time each version was published, over 499 nodes that exist in every year. A fixed memory budget is spent one of two ways: a few compact learned vectors, or 14 times as many raw past values. Both feed the same tail-risk readout, and the results are judged by tests that were written down before any test data was scored.

Decisions that matter

  • Prices as they were known

    Real-time electricity prices get revised after the fact. Every version is stored with when it was published, so a model only ever sees what was known at the moment it had to decide.

  • Learned vectors win every comparison

    Across six memory budgets and two ways of choosing what to keep, learned vectors had the lower tail-forecast error in all 12 comparisons. 11 of the 12 are significant on their own (Diebold-Mariano p < 0.05); the smallest-gap case, six vector slots, is not. The project flags its pooled p-value as too optimistic, so the claim rests on these per-budget tests.

  • Kill gates written down first

    Each hypothesis had a pass-or-kill test fixed before any data was scored. Two of the three failed, and they were reported as failures.

  • Memory problems traced to code

    The build kept running out of memory. The cause was three allocations that loaded far more than they needed. After fixing them it peaks at 2.8 GiB with no swap and gives bit-identical results. The raw data is 329M rows of 15-minute prices, stored as first published.

Results

Learned vector state lowers tail-forecast loss at every memory budget.
  • 1 vector slot vs 14 raw13.1 %
  • 2 vector slots vs 28 raw19.2 %
  • 3 vector slots vs 42 raw17.7 %
  • 4 vector slots vs 56 raw14.0 %
  • 5 vector slots vs 70 raw9.2 %
  • 6 vector slots vs 84 raw4.8 %

99th-percentile pinball loss, learned vectors against raw history at equal bytes, one bar per budget from 1 to 6 vector slots. Recency policy, test split.

Three allocations fixed, from gigabytes to megabytes.
  • Node census, before10,240 MiB
  • Node census, after91 MiB
  • Projection, before4,506 MiB
  • Projection, after788 MiB
  • Yearly pivot, before1,536 MiB
  • Yearly pivot, after146 MiB

Peak memory of each step before and after its fix, log scale. The 10 GiB figure was one short-lived allocation, not the steady state.

Limits. This is research on one market and one machine. Of the three ideas tested, only the memory result held up; the other two were negative and are reported that way.

Next projectMainstay