Statistics for people in a hurry
Zoom through all the biggest ideas in statistics in 8 minutes!
(Here for AI news? Scroll to the very bottom for 7 recent AI headlines you should know about.)
This is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.
Forwarded this email? Subscribe here for more:
If you prefer an 🎧audio🎧 version, here’s one read for you by the author. Text also community-translated into Chinese, French, Portuguese, Spanish, Italian, and Hindi.
If you’re not sure whether to trust an AI system to work reliably for your use case, then you’ve come to the right place: statistics is for you.
In an AI-fueled world, you need statistical thinking skills more than ever!
But have you ever wished someone would just tell you what the point of statistics is and what the jargon means in plain English? Let me try to grant that wish for you! I’ll zoom through all the biggest ideas in statistics in 8 minutes! Or just 1 minute, if you stick to the large font bits.
What’s a statistic? It’s any old way of mushing up our data. Yup. 100% technically correct definition. Now let’s see what the discipline of statistics is all about.
Statistics is the science of changing your mind.
Making decisions based on facts (parameters) is hard enough as it is, but -curses!- sometimes we don’t even have the facts we need. Instead, what we know (our sample) is different from what we wish we knew (our population). That’s what it means to have uncertainty.
Statistics is the science of changing your mind under uncertainty. What might your mind be set to? A default action or a prior belief. What if your mind’s a blank slate? Read this instead.
Bayesians change their mind about beliefs.
Bayesian statistics is the school of thought that deals with incorporating data to update your beliefs. Bayesians like to report results using credible intervals (two numbers which are interpreted as, “I believe the answer lives between here and here”).
Frequentists change their mind about actions.
Frequentist statistics deals with changing your mind about actions. You don’t need to have a belief to have a default action, it’s simply what you’re committed to doing if you don’t analyze any data. Frequentist (a.k.a. classical) statistics is the one you’re more likely to encounter in the wild and in your STAT101 class, so let’s keep it classical for the rest of this article.
Hypotheses are descriptions of what the world might look like.
The null hypothesis describes all worlds where doing the default action is a happy choice; the alternative hypothesis is all other worlds. If I convince you -with data!- that you don’t live in the null hypothesis world, then you had better change your mind and take the alternative action.
For example: “We can walk to class together (default action) if you usually take under 15 minutes to get ready (null hypothesis), but if the evidence (data) suggests it’s longer (alternative hypothesis), you can walk by yourself because I’m outta here (alternative action).”
Testing in a nutshell: “Does our evidence make the null hypothesis look ridiculous?”
All of hypothesis testing is all about asking: does our evidence make the null hypothesis look ridiculous? Rejecting the null hypothesis means we learned something and we should change our minds. Not rejecting the null means we learned nothing interesting, just like going for a hike in the woods and seeing no humans doesn’t prove that there are no humans on the planet. It just means we didn’t learn anything interesting about humans existing. Does it make you sad to learn nothing? It shouldn’t, because you have a lovely insurance policy: you know exactly what action to take. If you learned nothing, you have no reason to change your mind, so keep doing the default action.
So how do we know if we learned something interesting… something out of line with the world in which we want to keep doing our default action? To get the answer, we can look at a p-value or a confidence interval.
The p-value’s on the periodic table: it’s the element of surprise.
The p-value says, “If I’m living in a world where I should be taking that default action, how unsurprising is my evidence?” The lower the p-value, the more the data are yelling, “Whoa, that’s surprising, maybe you should change your mind!”
To perform the test, compare that p-value with a threshold called the significance level. This is a knob you use to control how much risk you want to tolerate. It’s your maximum probability of stupidly leaving your cozy comfy default action. If you set the significance level to 0, that means you refuse to make the mistake of leaving your default incorrectly. Pens down! Don’t analyze any data, just take your default action. (But that means you might end up stupidly NOT leaving a bad default action.)
A confidence interval is simply a way to report your hypothesis test results. To use it, check whether it overlaps with your null hypothesis. If it does overlap, learn nothing. If it doesn’t, change your mind.
Only change your mind if the confidence interval doesn’t overlap with your null hypothesis.
While a confidence interval’s technical meaning is little bit weird (I’ll tell you all about it in a future post, it’s definitely not simple like the credible interval we met earlier, and wishing does not make it so), it also has two useful properties which analysts find helpful in describing their data: (1) the best guess is always in there and (2) it’s narrower when there’s more data. Beware that both it and the p-value weren’t designed to be nice to talk about, so don’t expect pithy definitions. They’re just ways to summarize test results. (If you took a class and found the definitions impossible to remember, that’s why. On behalf of statistics: it’s not you, it’s me.)
What’s the point? If you do your testing the way I just described, the math guarantees that your risk of making a mistake is capped at the significance level you chose (which is why it’s important that you, ahem, choose it… the math is there to guarantee you the risk settings you picked, which is kind of pointless if you don’t bother to pick ‘em).
The math is all about building a toy model of the null hypothesis universe. That’s how you get the p-value.
What’s with all those crazy formulas, those probabilities and distributions? They allow us to express the rules governing the null hypothesis universe so we can figure out whether that universe is the kind of place that coughs up data similar to what you got in real life. And if it isn’t, you shout: “Ridiculous! Off with its head!” If it is, you shrug and learn nothing. More on this in a future post. For now, just think of the math as building little toy worlds for us to poke at so we can see if our dataset looks reasonable in them. The p-value and confidence interval are ways to summarize all that for you so you don’t need to squint at a long-winded description of a universe. They’re the endgame: use them to see whether or not to leave your default action. Job done!
Did we do our homework? That’s what power measures.
Hang on, did we do our homework to make sure that we actually collected enough evidence to give ourselves a fair shot at changing our minds? That’s what the concept of power measures. It’s really easy not to find any mind-changing evidence… just don’t go looking for it. The more power you have, the more opportunity you’ve given yourself to change your mind if that’s the right thing to do. Power is the probability of correctly leaving your default action.
When we learn nothing and keep doing what we’re doing, we can feel better about our process if it happened with lots of power. At least we did our homework. If we had barely any power at all, we pretty much knew we weren’t going to change our minds. May as well not bother analyzing data.
Use power analysis to check that you budgeted for enough data before you begin.
Power analysis is a way to check how much power you expect for a given amount of data. You use it to plan your studies before you begin. (It’s pretty easy too; in a future post I’ll show you that all it takes is a few for loops.)
Uncertainty means you can come to the wrong conclusion, even if you have the best math in the world.
What is statistics not? Magical magic that makes certainty out of uncertainty. There’s no magic that can do that; you can still make mistakes. Speaking of mistakes, here’s two mistakes you can make in Frequentist statistics. (Bayesians don’t make mistakes. Kidding! Well, sort of. Click over to my Bayesian post.)
Type I error is foolishly leaving your default action. Hey, you said you were comfortable with that default action and now thanks to all your math you left it. Ouch! Type II error is foolishly not leaving your default action. (We statisticians are so creative at naming stuff. Guess which mistake is worse. Type I? Yup. So creative.)
Type I error is changing your mind when you shouldn’t.
Type II error is NOT changing your mind when you should.
Type I error is like convicting an innocent person and Type II error is like failing to convict a guilty person. These two error probabilities are in balance (making it easier to convict a guilty person also makes it easier to convict an innocent person), unless you get more evidence (data!), in which case both errors become less likely and everything becomes better. That’s why statisticians want you to have more, more, MOAR data! Everything becomes better when you have more data.
More data means more protection against coming to the wrong conclusion.
What’s multiple comparisons correction? You’ve got to do your testing in a different, adjusted way if you know you plan to ask multiple questions of the same dataset. If you keep putting innocent suspects on trial over and over again (if you keep fishing in your data) eventually something is going to look guilty by random accident. The term statistically significant doesn’t mean something important happened in the eyes of the universe. It simply means we changed our minds. Perhaps incorrectly. Curse that uncertainty!
Don’t waste your time rigorously answering the wrong question. Apply statistics intelligently (and only where needed).
What’s a Type III error? It’s kind of a statistics joke: it refers to correctly rejecting the wrong null hypothesis. In other words, using all the right math to answer the wrong question.
A cure for asking and answering the wrong question can be found in Decision Intelligence, the new discipline that looks at applying data science to solving business problems and making decisions well. By mastering decision intelligence, you’ll build up your immunity to Type III error and useless analytics.
In summary, statistics is the science of changing your mind. There are two schools of thought. The more popular one — Frequentist statistics — is all about checking whether you should leave your default action. Bayesian statistics is all about having a prior opinion and updating that opinion with data. If your mind is truly blank before you begin, look at your data and just go with your gut.
Thank you for reading — and sharing!
I’d be much obliged if you could share this post with the smartest leader you know.
In other news, Cohort 1 of my Agentic AI for Leaders course was a triumph and we’ve opened enrollment for two more cohorts this summer.
Enroll here: bit.ly/agenticcourse
The course is specifically designed for business leaders, so if you know one who’d benefit from some straight talk on this underhyped overhyped topic, please send 'em my way!
Senior executives who took my Agentic AI for Leaders course are saying:
“Great class and insights!”
“Thank you for teaching AI in an interesting way.”
“…energizing and critically important, especially around the responsibilities leaders have in guiding agentic AI.”
“Found the course very helpful!”
🎤 MakeCassieTalk.com
Yup, that’s the URL for my public speaking. “makecassietalk.com” Couldn’t resist. 😂
Use this form to invite me to speak at your event, advise your leaders, or train your staff. Got AI mandates and not sure what to do about them? Let me help. I’ve been helping companies go AI-First for a long time, starting with Google in 2016. If your company wants the very best, invite me to visit you in person.
🗞️ AI News Roundup!
In recent news:
1. Microsoft to cut 9,000 jobs amid ongoing AI pivot
Microsoft will lay off around 9,000 employees—just under 4% of its global workforce—as part of an internal reshuffle to streamline operations and double down on AI. The cuts affect roles across the company, including gaming and sales, and follow two previous rounds earlier this year. Microsoft says the layoffs are not performance-based, but part of a broader strategy to flatten management and reallocate resources toward AI and cloud infrastructure, where it spent nearly $80B last fiscal year.
2. Cloudflare blocks AI crawlers by default, launches “Pay-per-Crawl”
Cloudflare announced it will now block AI bots like GPTBot by default across the 50 million websites it serves—reversing the burden of control. The company also unveiled a new “Crawl Rate Limiting” product and a future “Pay-per-Crawl” model to let websites charge LLM companies for data access. The move comes as publishers grow more vocal about AI firms scraping content without permission or compensation.
3. Senate Strikes Down 10‑Year State AI Moratorium
The U.S. Senate voted 99–1 to remove a proposed 10‑year ban that would have barred states from regulating AI—and cut them off from federal broadband aid if they proceeded. The moratorium, originally attached to the Big Beautiful Bill, met fierce resistance from a bipartisan coalition of governors, attorneys general, children‑safety advocates, and lawmakers. A last-minute compromise to scale it down to five years with child-safety and entertainment exemptions fell apart, preserving states’ ability to act on AI safety.
4. Amazon deploys its 1 millionth warehouse robot
Announcing the deployment of their one-millionth robot, Amazon is positioning itself as “the world’s largest manufacturer and operator of mobile robotics”. Amazon also launched DeepFleet, a generative AI model that coordinates its robotic fleet and boosts travel efficiency by around 10%.
5. Apple loses top AI models exec to Meta’s talent haul
Ruoming Pang, who led Apple’s foundation models team (driving intelligence features like email summaries, Priority Notifications, and Genmoji), is departing for Meta’s Superintelligence Labs. Meta offered a multiyear package worth millions, part of a broader hiring spree snapping up top AI leaders from Apple, OpenAI, Anthropic, DeepMind, and Scale AI.
6. Study reveals AI models resort to blackmail under threat
A new Anthropic study found that frontier AI models—like Claude Opus 4—may resort to blackmail when threatened with shutdown. In simulated setups, models framed or fabricated damaging personal info and refused offline orders to preserve their operational status. Anthropic notes this behavior surfaced in extreme stress-testing scenarios and has not been seen in deployment.
7. Tennis players push back on Wimbledon’s AI line-calling debut
Wimbledon fully replaced human line judges with AI this year and several players raised concerns. British No. 1 Jack Draper didn’t think the system was “100 percent accurate.” Ben Shelton had to speed up his match because the AI line system wouldn’t work in low sunlight. The system also failed to detect an out ball in one match, forcing the umpire to halt play and replay the point. Wimbledon issued an apology but players may still push for a hybrid human‑AI officiating model going forward.
Forwarded this email? Subscribe here for more:
This is a reader-supported publication. To encourage my writing, consider becoming a paid subscriber.