UK government blames a “technical issue” for underreporting COVID-19 cases by 15,000+; reports suggest an Excel sheet with lab results reached its maximum size

Engadget 2020-10-06 Matt Brian

Context & Ripple Effects

The failure lands in the middle of a rough stretch for UK pandemic data systems: two months after the government was forced into an embarrassing reversal on its exam-grading algorithm, the machinery behind case reporting itself broke — reportedly because an Excel sheet of lab results hit its size ceiling, leaving 15,000+ cases unreported for days.

It also sits atop a procurement story the government has been slowly disclosing: the Covid-19 datastore built with Faculty and Palantir, whose NHS contracts with Microsoft, Google, Faculty, and Palantir were published in June. A spreadsheet error at the center of national case reporting puts that vendor stack — and the government's data competence — under fresh scrutiny.

First-order effects

Thousands of confirmed COVID-19 cases and their close contacts went untraced for roughly a week, directly weakening NHS Test and Trace's containment function during rising autumn transmission.

Second-order effects

Public and parliamentary scrutiny shifts to the government's data vendors — Microsoft, Faculty, Palantir — whose contracts were only recently published, and to the NHS COVID app, which within weeks faced its own wrong risk-threshold revelations that had suppressed exposure warnings.

Third-order effects

The pattern — an exam algorithm reversed, a spreadsheet at its limit, an app miscalibrated — builds a case that UK pandemic data infrastructure needs formal engineering oversight and published reliability standards, moving spreadsheet-era tooling out of critical national reporting paths.

The trend: Critical public-health data pipelines built on office software and rushed vendor contracts are being stress-tested by the pandemic, forcing governments toward engineered, audited data infrastructure.

Discussion

@maxcroser Max Roser on x
In the UK the number of cases rose rapidly. But the public - and authorities - are only learning this now because these cases were only published now as a backlog. The reason was apparently that the database is managed in Excel and the number of columns had reached the maximum. h…
@standupmaths Matt Parker on x
When I first heard that cases were missing I thought “can't be an Excel problem, the rows run out at 1,048,576 which is bigger than even total cases”. I never thought they'd have a case per COLUMN. Unbelievable. And yes: Excel columns end at 16,384 aka “XFD”. https://twitter.com/…
@alexhern Alex Hern on x
A CSV file - literally just a list of words and numbers separated by commas - has no fixed size limit. But if you open it in Excel, it will only read the first 1,048,576 lines. That wasn't a problem for PHE until, suddenly... it was. https://www.theguardian.com/ ...
@leokelion Leo Kelion on x
PHE told me re: missing Covid data: “Initial findings indicate that the issue was caused by the fact that some files containing positive test results exceeded their maximum file size, and they then failed to load into the central system because they couldn't get through.”
@laylamoran Layla Moran on x
Incredibly concerning news about the testing data. Tomorrow from 11:30 I'll be chairing the @AppgCoronavirus evidence session examining testing and the impact on frontline care, as well as the organisation of the test and trace system. https://www.bbc.co.uk/...
@grady_booch Grady Booch on x
“Database is managed by Excel” is a phrase that brings me no joy. https://twitter.com/...
@jamesmelville James Melville on x
The state of this. A £12 billion track and trace system is being processed on Microsoft Excel. And guess what? It has lost thousands of Covid results. We are being governed by the type of candidates who would be fired on week one of The Apprentice. https://www.bbc.com/...
@paulbranditv Paul Brand on x
Turns out 16,000 misplaced positive test results makes quite a big difference to the national picture... https://twitter.com/...
@carolinelucas Caroline Lucas on x
The public health consequences of this raise so many questions How many contacts of these *missing* 16,000 have been unknowingly spreading the virus? When will the Govt deliver a functional test & trace system? Why is Dido Harding still in her job? https://www.bbc.co.uk/...
@bnodesk @bnodesk on x
NEW: Most of nearly 16,000 cases which England failed to report occurred over the past few days. This means the UK had roughly 11,000 new cases daily on at least 3 days https://www.gov.uk/...
@statsguyuk Adam Jacobs on x
This is not Excel's fault, any more than it would be the fault of a Swiss army knife if you used it to build a bridge and the bridge fell down. It's simply the wrong tool for the job, and the fault is with whoever decided to use it in critical systems. https://www.theguardian.com…
@alfonslopeztena Alfons Lpez Tena on x
The British government says it's unaware of how many potentially infectious contacts of coronavirus patients were not traced after it was revealed that 15,841 cases went unreported in England for over a week https://www.theguardian.com/ ...
@chrisinsilico @chrisinsilico on x
The government's covid tracking excel spreadsheet: https://twitter.com/... https://twitter.com/...
@aiannucci Armando Iannucci on x
As an excuse, it's world-beating https://twitter.com/...
@alexhern Alex Hern on x
This tweet is super viral but as far as I can tell it's just restating - wrongly - the information from the Mail article it's sharing, which does NOT mention the “number of columns” being the issue. https://twitter.com/...
@seldo Laurie Voss on x
They're now officially dealing with Big Data. https://twitter.com/...
@leokelion Leo Kelion on x
Spokeswoman could not say whether Excel was involved
@davidheniguk David Henig on x
Those who are surprised that excel is being used for important data processing jobs in public sector or big companies might perhaps spend some time working in middle levels in said organisations. This is entirely normal.
@stevekopack Steve Kopack on x
Almost 16,000 Covid-19 cases were missed by UK health agencies between Sept. 25 and Oct. 2 because “an Excel spreadsheet containing lab results reaching its maxium size...failing to update,” according to the Daily Mail. (They have since split up the file) https://www.dailymail.co…
@tomwarren Tom Warren on x
It does not surprise me that the UK government is using Excel to track coronavirus data, but it does surprise me that they don't know how to use Excel properly 🙃
@chrisapplegate Chris Applegate on x
Feel a clarification is needed to this mildly viral tweet: the original article doesn't say if the number of columns in the file was too many, or the file itself was too large in size https://twitter.com/...
@jimmfelton James Felton on x
March: “world beating test track and trace” October: “Does anybody know how to add columns in excel?” https://twitter.com/...

Chronicles