Research report

Memory-constrained forecasting

Finished: results below are from the final frozen test.

The question

If a forecaster can keep only a fixed number of bytes, is it better to keep a few learned summaries or many raw past values?

19%

lower tail-forecast error at equal memory, best budget

At equal memory the vector state cuts tail-forecast error by up to 19%, and is ahead at every budget.

Scalar history
Vector state

Tap a memory budget

Pinball loss at q = 0.99, recency policy, on the held-out test split (3,504 queries, 499 nodes). Lower is better. The axis starts at 0.008, not zero. Bytes match by construction, checked against the memory accountant.

Frontier as a table

Resident KB	Vector slots	Scalar slots	Scalar loss	Vector loss	DM p
61.9 KB	1	14	0.01161	0.01009	2.8 × 10⁻⁵
75.8 KB	2	28	0.01174	0.009486	7.7 × 10⁻¹⁰
89.8 KB	3	42	0.01173	0.009649	3.6 × 10⁻⁸
103.8 KB	4	56	0.01175	0.0101	3.6 × 10⁻⁵
117.8 KB	5	70	0.01176	0.01068	0.011
131.7 KB	6	84	0.01157	0.01101	0.20

Why it matters

Price spikes are rare, and they carry the risk, so the forecaster is scored on the 99th-percentile tail, not the average. Watching hundreds of grid nodes at once means deciding what to remember.

Method

Two forecasters get exactly the same bytes. One spends them on a few 14-wide slots: a node's latest price plus 13 summaries of what the whole grid is doing. The other spends them on 14 times as many slots of that node's own recent history.
The comparison was written down before the test data was scored. Nothing looks ahead in time, and the byte counts are checked against the memory accountant.
Each memory budget gets its own Diebold–Mariano test on the loss difference, corrected for correlated errors (lag 24), under two keep-policies: recency and a tail oracle.

Data

One year of real ERCOT electricity prices: 499 grid nodes and 35,035 intervals. The held-out test holds 3,504 queries.
The grid is very redundant. One factor explains 76.6% of price variance and five reach 90%; a synthetic generator needed 387, so it does not behave like the real grid.

Results

Every one of the 12 comparisons favours the vector arm. Each test is reported on its own, with its exact p-value.

The vector arm is ahead in all 12 comparisons; 11 clear p < 0.05 one at a time.

p below 0.0083 (0.05 ÷ 6 budgets)
p below 0.05 only
p at or above 0.05

Tap a row for its numbers

Each row is one Diebold–Mariano test on the scalar-minus-vector loss difference (HAC variance, lag 24), one test per budget and per comparison. Dot is the difference, bar its 95% interval; right of zero means the vector arm is better. The pooled p-value (p = 8.4 × 10⁻²¹) is flagged by the repo as too optimistic and is not used here.

Tests as a table

Policy	Vector / scalar slots	Difference	95% interval	DM p
Recency	1 / 14	0.00152	0.000808 to 0.00223	2.8 × 10⁻⁵
Recency	2 / 28	0.00225	0.00153 to 0.00297	7.7 × 10⁻¹⁰
Recency	3 / 42	0.00209	0.00134 to 0.00283	3.6 × 10⁻⁸
Recency	4 / 56	0.00165	0.000865 to 0.00243	3.6 × 10⁻⁵
Recency	5 / 70	0.00108	0.000245 to 0.00191	0.011
Recency	6 / 84	0.000557	-0.000303 to 0.00142	0.20
Tail oracle	1 / 14	0.0305	0.0118 to 0.0493	0.001
Tail oracle	2 / 28	0.0349	0.0157 to 0.0542	3.6 × 10⁻⁴
Tail oracle	3 / 42	0.0355	0.0163 to 0.0546	2.9 × 10⁻⁴
Tail oracle	4 / 56	0.0346	0.0157 to 0.0535	3.3 × 10⁻⁴
Tail oracle	5 / 70	0.0316	0.0137 to 0.0494	5.3 × 10⁻⁴
Tail oracle	6 / 84	0.0213	0.00811 to 0.0344	0.002

Why summaries work: a handful of factors carry most of what the whole grid is doing.

Real prices need 5 factors for 90% of their variance; the synthetic generator needed 387.

Price
Tail indicator

Tap a factor count

Cumulative variance explained by the first factors across 499 real grid nodes and 35,035 intervals of one year of real prices. Factors for 95% and 99%: 11 and 38 (prices), 29 and 87 (tail indicator).

Variance as a table

Factors	Price variance	Tail-indicator variance
1	76.6%	61.6%
2	83.3%	71.5%
3	87.5%	78.3%
4	89.5%	81.2%
5	90.9%	83.5%
6	92.0%	84.8%
7	93.0%	85.9%
8	93.7%	86.9%
9	94.3%	87.8%
10	94.8%	88.6%

A learned memory keeper did not beat a simple baseline at any of three scales. A negative result, kept.

A learned memory keeper never beat the simple H2O-style baseline, at any of three scales.

Keep all
H2O-style
Learned

Tap a scale

Mean q99 pinball loss on each run's frozen test split at a 25% state budget; lower is better. Keep-all stores everything and is the reference; H2O-style keeps the most-attended entries.

Scales as a table

Scale	Streams × context	Keep all	H2O-style	Learned
Small	32 × 48	0.06458	0.07692	0.1238
Medium	512 × 256	0.08054	0.06319	0.08072
Full	2,048 × 1,024	0.1117	0.1097	0.1119

An earlier pre-registered claim, that streaming scenario selection would beat batch selection, was not supported.

A pre-registered claim failed: streaming scenario selection did not beat batch selection.

Median error, with interval

Tap a method

Median absolute error of the 99% CVaR estimate, in percentage points, on the frozen test (k = 96 scenarios kept, 12 exposures, one touch of the test set); lower is better, and bars show 95% bootstrap intervals. The claim was that the streaming reservoir would beat the probability-metric batch method at equal bytes. It did not: its error is 36.3 points higher (interval [−14.9, 68.4]).

Methods as a table

Method	Median error (pp)	Interval low	Interval high
Streaming reservoir	60.4	10.5	91.9
Probability-metric	23.4	22.1	27.2
Sequential min-sum	22.4	8.9	23.8
CVaR-aware	23.4	22.1	27.2
k-means medoids	12.2	11.2	20

The honest limits

The vector arm is ahead in all 12 comparisons, but not every one is significant on its own. Raw per-budget p-values clear 0.05 in 11 of 12; recency at the largest budget does not (p = 0.20). The repository reports recency as significant at 4 of 6 budgets and the tail oracle at 6 of 6.
The pooled p-value (8.4 × 10⁻²¹) is too optimistic because six correlated series are pooled. The per-budget tests are the evidence.
Tail-oracle losses are inflated by a mismatch in how the readouts were fit, so only the comparison between arms is meaningful there, not the level.
One electricity market, one year of prices, one machine. Other markets are not tested.

What is next

The same idea of spending a fixed budget by decision value is applied to a third resource, test-time compute, measured in milliseconds instead of bytes.
Open question: why the learned keeper trails the simple baseline at every scale.

Next reportSatellite constellations for disaster response