Nothing Is Unstable

I've always been skeptical of “Big Data”, not just because of the hype but some intuitive concerns that using more data would solve problems better. I could never put my finger on it, but over the years got confirmation a few times, e.g. when building or validating models and checking that using just one percent of the data would work just as well for model building. Actually, better, since reducing training time from half a day to minutes enables you to actually try out ideas.

So with joy I discovered a 2018 paper by Xiao-Li Meng: Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election that adds a statistical underpinning to the fact more data is not necessarily better. Now I can't say I've understood the paper in detail, but it is well written and I have a bunch of important takeaways I want to share!

If you poll voters before an election, or take a sample of your population for which you have manual labels, the central question is how well you represent the full population. In statistical terms, you are asking about the difference between sample average and population average. Deciding the sampling or selection process is the R-mechanism. It might be a random sample, but especially in “Big Data” it might be self-reported or recorded data.

The key insight is that the difference between sample and population average is a sum of three errors: Data quantity + Problem Difficulty + Data Quality.

The data quantity error is very familiar. It encapsulates the idea that with increasing sample, the difference between sample and population average shrinks. The problem difficulty is related to the variability of responses. The crucial and often neglected error though is the data quality, which is investigated in this paper. Data quality is measured by the so-called data defect correlation as the correlation between the selection mechanism R and the response variable. This means that the chance of a record being selected depends on the response itself. It is related to what is known as a “design effect”. If we sample at random, the data defect correlation will be zero. However, in a “Big Data” setting where we recorded, say, 80% of the population, it can become a dominant error as it grows with the square root of N, given a population of size N. The impact becomes apparent when translating the error into effective sample size. The effective sample size is inversely proportional to the square of the data defect correlation. The paper shows that even with an “extremely modest” data defect correlation of 0.05, the effective sample size of half the US population (e.g. the registered voters in the USA) becomes ... 400! The paper has an extensive example on why polling failed to predict Trump's win in 2016 that I won't discuss here, but that is enlightening to read.

And so Meng poses the Big Data Paradox:

The bigger the data, the surer we fool ourselves.

And as much as that was true in the Big Data hype ten years ago, with Internet-sized training sets for LLMs it has becomes extinction-level foolishness!

In an audit of a database written by the chatbot, one of the issues had as a consequence that SELECT * FROM t WHERE 4 = 5 returned a row...

So I now propose that we call people who think “vibecoding” will replace people thinking “inhabitants of the land where 4 equals 5”.

I'm trying to actually make at least one recipe from each cookbook we have. I started with Noma's Guide To Fermentation,

and made lacto-fermented plums. That worked out well, but I didn't realize it would also create some alcohol. A small amount, but unfortunately any amount is a no-no for me, so my fermentation experiments are on ice for now.

In a few moments I will assemble the Mapo Tofu from Kenji López-Alt's “The Wok”: Have to leave out the Chinese cooking wine though...

/edit:

Our memory experiences time non-linearly for sure, but I think also discontinuously. At least for me, it feels like a folder of dossiers on memorable life situation with the disconcerting feeling that you will go back to those soon, or even that you should be able to move between them.

Had this big guy drop two days ago to eat some apples, chase the does and then ruminate.

In Serbia, a leaderless student party just toppled the fascist. Way underreported, as student protests leading to a truly democratic process of engaging with issues challenges ruling classes everywhere.

On the other hand, one should not idealise the situation. The list comprises ideologically opposed blocs, from rightwing adherents to leftwingers and liberals. Most of those elected have never before held political office, and the parliamentary system will require them to nominate a prime minister and, in future presidential elections, a presidential candidate. 

Yes, but the achievement of just getting to that state where the real work can even start should also not be understated.

In the discussions over whether the chatbot is “intelligent” or “super intelligent” or “artificially intelligent” (No.), let's not overlook the actual conclusion: “Intelligence” has always been a mostly useless concept, since it is so poorly defined and overwhelmingly a social construct of white supremacy. We should have gotten rid of it long ago, and everyone using “intelligence” as a concept should be viewed with suspicion.

Of all the bullshit innovation the idealess tech industry has given us over the last decade, the endless scroll must be one of the most evil. The obvious reason is the addiction of the never ending feeds of social media, but one subtler reason that I loathe is that it defeats any effort to find the long tail. Whenever you get a ranked list of “(thing|music|movies|books|search results) you should be interested in” that is actually just the “list of the most average stuff we are trying to shove into your face”, a good trick was to jump to result page 4 or 10 and poke around there. Way higher chance to find interesting things if you're willing to filter by trial and error. Except you can't do that anymore. There is no page 4. There is just the endless scroll. Maybe you can open items further down in new tabs. But more often, you're in an app, scroll scroll scroll, tap one item, go back and can't even find it again because the feed refreshes. This concentration of gate-keeping power instead of the promised personalisation is deeply destructive.

I was happy with setting up a simple blog as a static website on Vercel fed from GitHub, but I never wrote much. And then the Vercel CEO came out as hot for genocide and Vercel went “Agentic AI” and Github is turning into a LLM-boosting garbage show so it's time for another reset. I'm trying this as a simple text-based WriteFreely blog now to enable faster publishing and lower quality.