No. 41Dispatch22 September 2026

Four hundred milliseconds

On 14 August a generator and an interconnector dropped off the grid within a second of each other. This is what our batteries did in the first half-second, and why they did no more than the curve asked.

ICInes CarvalhoControls engineer

At 16:52:22 on Friday 14 August, a gas generator in the Midlands tripped, and 0.8 seconds later so did a cable to the continent. Together they took 1.3 GW off the grid. Frequency, which the whole system holds at 50 Hz, began to fall at about 0.024 Hz a second.¹

Every one of our sites watches that number with its own meter, twenty times a second. Twenty megawatts of the fleet were contracted that afternoon to the grid operator’s fastest frequency service, which pays us to push power into the grid in proportion to how far the frequency has fallen — and to do nothing at all while it stays near 50.

The curve

The contract is a curve, not a threshold. Inside ±0.015 Hz we do nothing: that is noise. From there to ±0.2 Hz we deliver up to 5% of our contracted power, and from ±0.2 to ±0.5 Hz the rest, in a straight line. Each site runs the whole curve itself. Nothing waits for our servers.

Deadband: ±0.015 Hz · no response
Knee: ±0.2 Hz · 5% of contracted power
Full response: ±0.5 Hz · 100%, 20 MW
Required within: 1 s of leaving the deadband
Our own target: 400 ms
Measured at: 20 Hz, at each site’s meter

Table 1. The response curve we ran on 14 August, as the contract states it.

In code it is short on purpose. It runs every 50 ms on a controller the size of a paperback,² and every branch in it is a thing the operator can ask us to justify.

Rust · controller/src/response.rs

/// The share of our contracted power to deliver at a grid frequency,
/// from -1.0 (take it all in) to +1.0 (send it all out).
pub fn response(hz: f64) -> f64 {
    const NOMINAL: f64 = 50.0;
    const DEADBAND: f64 = 0.015; // Hz: inside this, do nothing
    const KNEE: f64 = 0.2;       // Hz: 5% of power here
    const FULL: f64 = 0.5;       // Hz: all of it here

    let off = hz - NOMINAL;
    let d = off.abs();
    let share = if d <= DEADBAND {
        0.0
    } else if d <= KNEE {
        0.05 * (d - DEADBAND) / (KNEE - DEADBAND)
    } else {
        (0.05 + 0.95 * (d - KNEE) / (FULL - KNEE)).min(1.0)
    };
    -off.signum() * share // a low grid: export; a high one: import
}

Listing 1. The whole curve. The figure below runs this function, ported line for line, on the readings from Frome.

The minute, replayed

Frequency left our deadband at 16:52:23.6 and reached its lowest, 49.73 Hz, ten seconds later. At the bottom the curve asked for 27% of our 20 MW: 5.4 MW. We sent 5.4 MW. Drag through the minute below and watch the two instruments: the needle is the grid, and the ring is what the curve asked of us at that moment.

Frome · 14 August 202616:52:33.5

Sent to the grid: —
Since the trip: —
Held in reserve: —

Move through the minute+13.5 s

Fig. 1. One minute at Frome, from the site’s own meter. The ring runs Listing 1 on the frequency 340 ms earlier, our median delay that day; the rest of the fleet’s 20 MW stayed in reserve in case the grid kept falling.

That is the point of a curve. An event this size needs the fleet’s help, not its heroics: had the frequency kept falling, the other 14.6 MW were there for it.³ A battery that sends everything at the first dip is a battery with nothing left for the second one.

Four hundred milliseconds

The number in the title is the one we hold ourselves to: from the moment a site’s meter sees the frequency leave the deadband to the moment that site is on the curve. The service allows a second. Across the fleet on 14 August our median was 341 ms, and here are the twelve Somerset sites nearest the meter in Fig. 1.

Fig. 2. Amber past our own 400 ms, red past the service’s second. Ten of twelve made it; Bridgwater missed by 38 ms, Minehead by 510.

Minehead took 910. Nearly all of the difference is its inverter, a 2021 model that sleeps between jobs and takes 560 ms to wake.⁴ It still made the second, but 90 ms is closer than we like. This is the site’s own log of that half-second:

Terminal

$ slack log minehead --from 16:52:23.6 --to 16:52:24.6
16:52:23.650  meter     49.9848 Hz  outside the deadband
16:52:23.700  plan      asking 0.1% (0.02 MW)
16:52:23.702  inverter  waking from standby
16:52:24.262  inverter  awake after 560 ms
16:52:24.560  on the curve, 910 ms after the meter

Listing 2. Every site keeps 72 hours of this at 20 Hz.

What we changed

Two things. During the hours we are contracted, the six sites with 2021 inverters now keep them awake, which costs about 30 W a site and took Minehead to 352 ms in the next weekly test. And we now raise an alarm on any site slower than 600 ms in a real event, not only in the weekly one, because the test’s synthetic fall is gentler than a real grid.⁵

The grid was back above 49.9 Hz 3 minutes 40 seconds after the trip. Nobody in the office noticed until the operator’s report arrived the next morning, which is how it should be.

1Frequency is the same everywhere on the grid at once, to within a few millihertz: it is the speed at which every large generator in Britain is turning. Take more power than is made and they all slow down together.
2A Cortex-A53 board running Linux with the real-time patches. The loop’s worst case last month was 11.2 ms of its 50; No. 34 is how it got there.
3At 49.5 Hz the curve asks for all 20 MW. The grid has fallen that far twice in the last ten years.
4An inverter turns the battery’s direct current into the grid’s alternating current. The ones we have fitted since 2023 idle warm and answer in 40 ms.
5The weekly test feeds each controller a fall of 0.1 Hz a second and times the answer. The fleet’s median in tests is 290 ms.

No. 40Incidents8 September 2026

The hour that moved

A price feed started speaking UTC on 1 September. For one evening, 64 of our batteries charged through the peak they were there to cover.

TATom AchebePlatform lead

Every afternoon at 14:00 the scheduler plans the next day for each battery: when to charge, when to sell, when to hold back for frequency service. It plans against prices, half an hour at a time, from a feed that one of our suppliers publishes.

On Tuesday 1 September the supplier moved that feed to a new version. The new version gives each half-hour’s start in UTC, with a Z on the end. The old one gave local time with no offset, and our parser had always assumed local.¹ So it read 16:00Z, which is 17:00 in September, as 16:00, and every price of the evening landed an hour early.

What the batteries did

The scheduler did the right thing with the wrong day. It sold into what it believed was the peak and charged in what it believed was the cheap hour after it, which was really the peak. Sixty-four sites on the dynamic tariff² followed that plan on the evening of Wednesday 2 September, and took 30.5 MWh from the grid at the most expensive hour of the day.

Fig. 1. Charging by site group and half-hour. Between 16:30 and 19:00 the fleet should take nothing; two columns say otherwise.

How we found out

Alert: 64 sites importing in the peakFrom the check that compares plans with the published peak18:04
The batteries are obeying their plansSo the plans are wrong, not the sites18:11
The feed’s timestamps end in ZEvery price an hour early since 1 September18:26
All 64 held at zero by handNeither charging nor selling until the peak ends18:31
Plans for 3 September rebuiltFrom the feed, read with its offset19:02

Fig. 2. The evening, from the on-call log. Twenty-seven minutes from the alert to the hold.

It cost us £9,460 at that evening’s prices, and it cost the grid a little of the headroom it was counting on at 18:00.

Why it took an evening

The feed changed on Tuesday; the plans went wrong on Wednesday. Nothing looked wrong on Tuesday night, because a plan made from shifted prices is still a sensible-looking plan: it charges overnight, sells in the early evening, and its totals are within a few per cent of any other September day. Our checks compared each plan with itself — does it respect every battery’s limits, does it keep back the reserve the frequency contract needs — and all of that was true.

The check that fired at 18:04 is the one that compares a plan with the world outside: the grid operator’s published peak hours. It had existed for three weeks. Before that, we would have found out from the invoice.

The fix

Three lines, and a rule. The parser no longer guesses: a timestamp without an offset is an error, and an error stops the plan and wakes a person.

Python · scheduler/feeds/prices.py

# Before: a timestamp without an offset was taken to be London time.
start = datetime.fromisoformat(slot["start"]).replace(tzinfo=LONDON)

# After: the feed says which zone it means, or we stop.
start = datetime.fromisoformat(slot["start"])
if start.tzinfo is None:
    raise FeedError(f"{feed.name}: slot {slot['id']} has no offset")
start = start.astimezone(LONDON)

Listing 1. The change, merged at 21:40 the same evening.

The rule: a plan that charges during the grid’s published peak now needs a person to approve it before it runs. It has fired once since, correctly, on a windy Sunday when the peak really was the cheapest hour of the day.³

We have also asked the supplier to publish a new version of a feed at a new address instead of changing the old one in place. They have agreed to, from November.

1The old feed’s documentation said “local time”. The new one’s says “ISO 8601”, which is true of both.
2The dynamic tariff follows the half-hourly price. The other 148 sites were on fixed contracts that evening and planned from a different feed.
3Approving it took Ines four minutes, most of them spent checking the wind forecast.

No. 39Hardware25 August 2026

Why the new cabinets have louvres

Two slots, two baffles and a new fan curve took six degrees off our hottest afternoons.

PNPriya NairHardware engineer

Our first cabinets were sealed boxes with a heat exchanger in the door. They were quiet and they kept the rain out, but on a still July afternoon the air inside reached 41 °C, and the battery management system began to hold the cells back to protect them. Held-back cells are power we were paid to have and could not use.

The cabinets we have fitted since June have two louvred slots, low on the north side and high on the south, with a baffle behind each, so rain that finds its way past a louvre runs down the baffle and out rather than into the cabinet. The fan now follows the hottest cell’s temperature instead of the air’s.¹

In July	Sealed, 2024	Louvred, 2026
Hottest air inside	41.0 °C	35.2 °C
Hours held back	37 h	2 h
Fan energy a day	1.9 kWh	1.4 kWh
Noise at 1 m	38 dB(A)	41 dB(A)

Table 1. Two cabinets side by side at Street, July 2026.

The new fan is louder, by three decibels at a metre. At the site fence it is below the hum of the transformer, which we checked with the neighbours before we checked it with a meter.

1Cells care about their own temperature, not the air’s. Driven by the hottest cell, the fan runs less on cool nights and harder for the one warm string.

No. 38Firmware11 August 2026

Updating 212 batteries without a van

Firmware 4.2 reached 211 of our sites in nine days and four waves. The 212th rolled itself back, which is what we built it to do.

SOSam OkaforFirmware lead

A site controller that stops answering is a van, a drive and most of someone’s day, and some of our sites are an hour down lanes on Exmoor. So every update is built not to need a van, and we update often: 4.2 is the fifth release this year.

Four waves

Each controller holds two copies of its firmware.¹ An update writes the copy that is not running, reboots into it, and then has ten minutes to pass its health checks. If it does not, a watchdog boots the old copy again and reports why. Nobody has to be awake for that.

Fig. 1. The waves are the steps. The one-site dip on 4 August is Lynton going back to 4.1.

Wave	Sites	Started	Soak	Rolled back
1 · the depot	2	29 July	48 h	0
2 · canaries	20	31 July	48 h	0
3 · a third	80	3 August	72 h	1
4 · the rest	110	6 August	—	0

Table 1. The waves of 4.2. Sort by any heading.

What a wave waits for

A wave does not start on a date; it starts when the one before it has been quiet for long enough. Quiet means no rollback, no site that has restarted more than once, no change of more than 5% in the weekly frequency test, and no gap in readings we cannot explain. The soak times in Table 1 are the least we wait: wave 3 sat for 72 hours rather than 48 because it held Lynton’s rollback, and a rollback always buys a third day, even when we already know the cause.

Most of a release’s risk is in the first two waves, which are small on purpose. The depot’s two controllers sit on a bench beside real inverters and a simulated grid, and we try to break them there: pulled cables, a full flash disk, a clock set to 1970. The twenty canaries are real sites chosen for being unlike each other — the oldest inverters, the weakest signal, the largest battery — so that the odd ones meet a release first.

The one that came back

Lynton is our most remote site, and it reaches us over a mobile link that falls back to 2G when it rains. One of 4.2’s health checks is that it has spoken to our servers within five minutes of booting. On 4 August, in the rain, that took eleven. The watchdog rolled it back to 4.1 at minute ten, exactly as designed, and Lynton carried on with its job on the old firmware.²

TOML · fleet/rollout/4.2.toml

[health]
window_s = 600          # pass every check within ten minutes of boot
checks = [
  "meter.frequency_ok",   # 20 Hz readings, none stale
  "bms.cells_reporting",  # every string of cells answers
  "inverter.handshake",
  "uplink.checked_in",    # reached our servers at least once
]

[health.uplink]
# Lynton, 4 August: 2G in the rain took eleven minutes.
checked_in_within_s = 900

Listing 1. The checks every site must pass before it keeps new firmware.

We gave the check-in fifteen minutes and updated Lynton again on 9 August, in sunshine.³

1The second copy costs 64 MB of flash per controller. A van costs about £180 a visit before anyone opens a cabinet.
2A controller on old firmware is still a working controller. For five days Lynton simply missed 4.2’s faster frequency filter.
3The depot wave now also runs over a throttled link. We built it a very small Faraday cage.

No. 37Dispatch28 July 2026

Paid to charge

Twice this summer the wholesale price went below zero at lunchtime. Here is what the scheduler did with it.

ICInes CarvalhoControls engineer

On 12 July and again on 19 July, the day-ahead price for the half-hours from 12:30 to 14:00 went negative. Wind and sun together made more power than Britain wanted, and generators paid to keep running rather than stop and start again.

Lowest price−£38.20/MWh

Taken in41.0MWh

Sold that evening at£182/MWh

Fig. 1. The six negative half-hours of 12 and 19 July, together.

The scheduler treats a negative price like any other: it is the cheapest energy of the day, so it fills the batteries then and sells in the evening.¹ The only thing we changed was a guard that had never fired. It refused to plan any charge below −£20, on the theory that such a price was more likely a broken feed than a real one. It was real. The guard now asks a person instead of refusing.²

1Less the round trip: our cells give back about 88% of what goes in.
2Both days had been forecast negative the afternoon before, so the question it asks is short.

No. 36Hardware14 July 2026

A year of cell temperatures

Every day since last July, the hottest cell in the fleet, on one calendar. The three weeks that taught us most were in February.

PNPriya NairHardware engineer

Every cell in the fleet reports its temperature once a second, and there are 58,000 of them.¹ Each day we keep one number from all of that: the hottest reading from any cell, anywhere. It is a blunt measure and a useful one, because a battery ages fastest at its hottest cell.

Fig. 1. 14 July 2025 to 13 July 2026. Point at a day for its reading.

Summer is where you would look first, and July and August are bright. But look at the three weeks from 2 February. We changed the overnight schedule that month to buy more of the cheap hours, and a cell charged at its full rate warms by eight to twelve degrees in an hour.² Those are the hottest winter days we have, and they were hot at four in the morning.

We moved the overnight charge to start earlier and run at half the rate. February’s hottest cell fell from 37 °C to 31.

What the calendar is for

We look at it every Monday, and it has changed our minds twice. The first was February. The second is that the hottest days are not the days with the hottest weather. Heat builds up in a cabinet across a day of hard trading and leaves slowly when the air outside is still, so the fleet’s worst days last summer came after its busiest afternoons, not its sunniest ones.

The summer peaks are the cabinets themselves, and the ones we fit from June onwards are built differently. More on that when they have seen a summer.

1212 sites of 16 to 64 strings, 18 cells a string. We keep every reading at one-minute resolution for 90 days.
2Our cells are lithium iron phosphate. Charged at half rate they warm by three or four degrees, and a cold February night has room for that.

No. 35Incidents30 June 2026

Seventeen minutes in Bridgwater

A modem locked up, a rule was too patient, and the battery kept doing the right thing without us.

TATom AchebePlatform lead

On 24 June at 11:42 our Bridgwater site stopped reporting. For seventeen minutes we could neither see it nor send it anything.

Last reading receivedCharging at 180 kW, as planned11:42:10
Alert: silent for three minutesOn call starts looking11:45:10
The modem’s own port is silent tooThe network is up; the modem is not11:51:30
The controller power-cycles the modemAfter its fifteen-minute rule11:57:10
Readings arrive, none lostSeventeen minutes of buffer, in order11:59:40

Fig. 1. From the on-call log.

The controller never stopped. It held its plan for the day, answered two small frequency deviations on its own, and kept every reading until the link came back.¹ What failed was the modem, and what was slow was the rule that restarts it: fifteen minutes of silence before the controller power-cycles it. We have made it three.

1Each controller keeps 72 hours of readings. Our servers take them in order when the link returns, so the gap in our charts filled itself within a minute.

No. 34Firmware16 June 2026

From one second to fifty milliseconds

The site controller used to think once a second. Getting it to twenty times a second meant taking almost everything out of the loop.

SOSam OkaforFirmware lead

Until this spring each site controller ran a one-second loop: read the meter, read the battery, decide, write the inverter’s set-point, write the log, send to our servers. For the wholesale market that is plenty. For the fast frequency service, whose contract gives us a second for the whole answer, it spent the budget before we had started.¹

What was in the loop

We timed every step over 4,000 loops at the depot. Two steps were most of the second, and neither needed to be in it: writing the log to flash, and sending readings to our servers.

Fig. 1. 440 ms of a one-second loop, and 390 of them were bookkeeping.

What we took out

In 4.0 the loop does four things — read, read, decide, write — and hands its readings to a second thread through a ring buffer. That thread logs and sends at its own pace, and if it falls behind, the buffer holds 72 hours.

// 3.x: one loop does everything, once a second.
loop {
    let m = meter.read()?;
    let b = bms.read()?;              // ~38 ms
    let sp = decide(&m, &b, &plan);
    inverter.write(sp)?;
    log.write(&m, &b, sp)?;          // ~210 ms on flash
    uplink.send(&m, &b, sp)?;        // ~180 ms on 4G, seconds on 2G
    sleep_until(next_second());
}

// 4.0: the loop only reads, decides and writes, every 50 ms.
loop {
    let m = meter.read()?;
    let b = bms.cached(Duration::from_millis(500))?; // refreshed every 200 ms
    let sp = decide(&m, &b, &plan);
    inverter.write(sp)?;
    ring.push(Reading { m, b, sp });  // the log and uplink threads drain it
    sleep_until(next_tick(Duration::from_millis(50)));
}

Listing 1. The loop, before and after. decide did not change.

The battery’s own management system takes 38 ms to answer, most of a 50 ms tick, so it moved to a thread of its own too, refreshing a cached copy every 200 ms.² Cells do not change much in 200 ms; the grid can. The loop’s median is now 4.1 ms.

What it cost

Two things got harder. A log line and the reading it describes are now written by different threads, so each carries the loop’s tick number, and our tools sort by that rather than by arrival. And a crash can no longer be found by reading the last line of the log, because the last line may be 200 ms behind the loop: the loop now keeps its last hundred ticks in memory and writes them out on its way down.

In exchange the controller answers the grid within one tick instead of one second, and the fast frequency service became a contract we could sign.

1The service allows one second from the frequency leaving the deadband to full response. A one-second loop can spend all of it before it has read the meter.
2A cached reading is never more than 200 ms old in normal running, and the loop refuses one older than 500 ms: it holds the last set-point and raises an alarm instead.

The Slackwater Log

Four hundred milliseconds.

Entries

Four hundred milliseconds

The curve

The minute, replayed

Four hundred milliseconds

What we changed

The hour that moved

What the batteries did

How we found out

Why it took an evening

The fix

Why the new cabinets have louvres

Updating 212 batteries without a van

Four waves

What a wave waits for

The one that came back

Paid to charge

A year of cell temperatures

What the calendar is for

Seventeen minutes in Bridgwater

From one second to fifty milliseconds

What was in the loop

What we took out

What it cost