No idea about statistics, but in most physict courses in my university, they recomend 3 books:
1) The main book, that has a complete explanation and is well ordered. It's for learning.
2) Tha Landau book, that is super short and hard. It's only to check you didn't miss any important formula or topic.
3) There Feynman book, that is anassorted colection of fairytales for physicist. It's a pleasure to read it but you must already read 1 to understand it.
4) The Shaum book, that is almost a colection of exercices. Some people hate it. Some people love it. I like it as a companion to theother books.
I guess you are complaining that 1 is boring and want to write 3. It's a good idea, but it's harder than expected.
My personal opinion is that statistics textbooks usually come from a prescriptive perspective, and that makes it challenging for the reader to get visceral intuition for what is actually going on. Any reader would be far better off just visualizing the damn distribution / samples and using reasonable judgement, instead of implicitly assuming a Gaussians distribution and blindly memorizing tests / formulae. Making the distributions explicit allows us to model them and get an intuition for what the samples are telling us. I would whole-heartedly recommend the Model based machine learning book to anyone (online version is free) https://mbmlbook.com/
Look at the histogram and think about what distribution one could reasonably impute from samples. And what you would set as bounds for "outliers", per your needs. While we're at it, let me also say that it might be useful to specify outlier bounds not just based on the spread in sample values, but the costs/payoffs they imply for your application.
If you are not doing something crazy, most reasonable people would agree with your judgement. Conversely, if you are making non-obvious inferences where reasonable people disagree, you are in murky water and no sophisticated statistical method will save you. Math is not magic; theorems merely recycle (launder) modeling assumptions into results.
>Look at the histogram and think about what distribution one could reasonably impute from samples. And what you would set as bounds for "outliers", per your needs. While we're at it, let me also say that it might be useful to specify outlier bounds not just based on the spread in sample values, but the costs/payoffs they imply for your application.
Only people with prior education/training in statistics are capable of doing this. The people who don't need a textbook.
Something like 60% of US adults read at or below the 6th grade level, and 25% of US adults struggle to comprehend graphs or charts entirely. Someone who has no idea what a standard deviation is can't intuit about distributions. I think you're dramatically overestimating the average person.
> Someone who has no idea what a standard deviation is can't intuit about distributions.
I disagree vehemently with this claim. I could cite my experience in teaching this topic to liberal arts / humanities college students in the US, but it is really more obvious than that. Anyone can understand a histogram easily and far more intuitively than they can understand the formula for a standard deviation and whether it must divide by N or N-1. Statistics courses and textbooks get stuck on that kind of pedantry, and most students end up missing the forest for the trees.
My larger issue, any time I have tried to learn statistics, is how fast the notation moves. You end up flipping back pages and pages just to double-check a definition that was given once and is now being extended syntactically. It's infuriating.
I've got a Bachelors and two Masters degrees in CS/Math, but yet I feel Probability and Statistics is my greatest weakness. I just cannot grok it / build an intuition for it, and believe me, I've tried. My biggest gripe with Prob/Stats textbooks is that it's very hard to explain things without needing to rely on measure theory.
Maybe probability and statistics are a skill issue on my behalf, but what I absolutely loathe is the absolute lack of standardisation when it comes to notation in measure theory. Every textbook does it differently. All of them assume that their notation is the one everyone uses. Nobody bothers to explain _what_ the notation means. If you ask me, every bit of new notation should be introduced with a sentence or two on "how to read this symbol in your head" - especially when there are indices, subscripts and superscripts involved. It's especially terrible for measure theory because there's so much "implicit" information you're supposed to gather from the context - but in a way I understand it, because if every bit of notation of absolute and complete, I imagine it would be quite hard to type up.
Anyways, my rant on measure theory notation aside - I would absolutely read yet another Prob/Stats textbook. But unfortunately I will also drop it really quickly if the author doesn't show me any "notation-sympathy" :)
+ Krushke’s Doing Bayesian Data Analysis, and for the very basics there is Downey’s Think Bayes. I guess there might be a gap in the literature for a different approach, heck, I’d read it, but there is some very good material already out there.
Maybe my expectations were too high given all the online praise, but I've been working through it for the past two weeks and I've been underwhelmed:
- The language is often very vague and imprecise, so it's more difficult than it feels it should be
- Concepts are sometimes introduced "at random" in ways that only really make sense in hindsight. So as you're reading you're left scratching your head as to why something was brought up.
- There are constant philosophical and historical digressions that seem to hold deeper meaning, but maybe once you know the topic already.
- Similarly, constantly talking about people that take issue with the method. People not liking the method is a constant theme (they seem really butthurt about this?). But the craziest part is this all done before you even really understand what the method is!!
- The editor must have placed some strict requirement of saying "Bayesian" at least five times per page.
- No index. Useless table of context. But lots of end-notes you feel compelled to flip to constantly
Overall it feels like a textbook written to impress other statistics professors - and as an outlet for the author to air some frustrations with how people do statistics (which may be completely valid!)
The overall structure and objectives seem solid for the most part. It's just a lot of the details aren't great. The problems (so far) have been good. The examples in the text are fun and compelling, but you have to do your own legwork to actually pick through all the prose and tie the pieces together - to figure how it fits together mathematically. Fortunately AI helps as a tutor
Rethinking is indeed written for people who have some or even a lot of existing proficiency in statistics, but for whom it never really clicked. If that’s not your background, I can understand that it might feel disjointed.
Having also studied statistics in university (undergrad), something I kept running into is that you can't really unlock the intuition for many concepts without taking more advanced courses. For example, degrees of freedom shows up as early as AP Statistics, but even a non-rigorous visual explanation of it leans on linear algebra, which most students don't see until much later.
I think more resources like seeing-theory would be great since stats books are almost universally dry (Blitzstein being a notable exception), but I'm not sure how easily more advanced concepts lend themselves to visual explanation in a way that's digestible for a non-stats person.
I feel this boils down my learning journey as well. You start unraveling a very good intuition about the underlying concepts MUCH much later, but partly because those intuitions themselves are never conveyed and are supposed to be learned from the proofs, and are an indirect product of learning.
- put a lot of 'teaching' into the book. Don't write so the learner memorizes; instead, motivate each discussion so that the learner understands, and gains what Wirth calls _coverage_.
- ensure there are no errors; this will turn off learners.
- keep a light tone, be yourself not a stuffed shirt. A good model for such writing is Nassim Taleb's non-technical books.
- never seem to be in a hurry. Many good books were ruined because midway the author lost interest and just hurried up.
- don't be too verbose, and neither too concise. Every sentence should do required work; learn how to write (if needed) and practice daily.
- ensure the book uses good, comfortable fonts, and good typography. Must learn this. Many good books ruined due to lack of fluency effect.
If you conscientiously do these things, there will always be readers for your book.
A long time ago, I took a "statistics for engineers" class in order to graduate. I slept through most of the classes. It sucked, and 70 percent of it was just "distribution of the week." It did not help that homework was optional for half of it.
At some point in my professional career I started reading non-fiction books and even bought a used statistics textbook for 10 bucks on abebooks. I didn't end up actually reading it until 12 years later during the COVID lockdown. I ended up shooting for 10 pages a day, 7 days a week. If those 10 pages included review exercises, it would be a long night.
Could just be the right book at the right time, but this one really helped me understand stuff beyond the normal HS math stuff, like RMS-error, calculating correlation, the difference between standard error and standard deviation, the relationship between sample size and standard error, t-tests, and chi-squared. Working as an SRE/release engineer, this stuff really helped me overcome a lot of _bad_ canary data analysis my predecessors had constructed.
That book was the 3rd edition of Statistics by Freedman et al.[1] One thing I want to complement was getting the pedagogy right. Most chapters have strong narrative hooks, several "check your knowledge" problems, review exercises, and post chapter bullet points to assist with spaced repetition. There's even a series of "special" review exercises covering entire sections of the book, ie exams.
For the HN crowd I should also probably note that the book is almost entirely non-bayesian and not intended to prepare readers for further coursework. You will not learn normal phraseology like "IID," "random variable" or "kernel".
There are a number of good books to explain the main concepts of statistics without getting bogged down in the technical details. As a grad student at a public university in the USA, I worked as a teaching assistant for the undergraduate course Introduction to Statistics. The course used SPSS software to enable the students to perform statistical analysis on data sets. Although we offered a good textbook (listed below), many of the students struggled with the main concepts, so my colleagues and I added a optional supplemental reading list to the syllabus. The following books are the ones we recommended to the students. I have read all these books, and they are pretty good at explaining statistics to anyone interested in learning the material at a conceptual level; anyone can read and understand these books, regardless of their educational background.
Ian Ayres (2007) Super Crunchers: Why Thinking by Numbers is the New Way to Be Smart.
Joseph Healey (2020) Statistics: A Tool for Social Research [This is the textbook used in the class, which I highly recommend].
John Allen Paulos (1988) Innumeracy: Mathematical Illiteracy and Its Consequences.
Nate Silver (2012) The Signal and the Noise: Why So Many Predictions Fail, but Some Don't.
Nassim Nicholas Taleb (2005) Fooled by Randomness, second edition.
Nassim Nicholas Taleb (2010) The Black Swan: The Impact of the Highly Improbable, second edition.
Did you see that Andrew Gelman and others just published Bayesian Workflows? https://avehtari.github.io/Bayesian-Workflow/
It sounds similar to what you're after, away from describing the logic of models and the maths, instead it's about (quote from intro) 'There are all sorts of tacit knowledge in applied statistics that do not make it into published
papers and textbooks. The present book is intended to put some of these ideas out in the open'
Even if I would not read it back to back on release, it would be a pleasure to have a reliable and citable reference on hand. Whenever I stumble upon new complex problems, outside of the regular, often fairly repetitive statistical questions of my field, I need to rely on many searches and LLM queries to find my answer. I wonder if a book could actually replace all that, but it would be my first place to check.
I think statistics are important for everyone to understand; however, I am wondering what shape that book would take and who the target audience would be. It would be wildly different to write a book like that for people who already have a very clear or existing knowledge on books than for people who are just dipping their feet and want to know more.
There are a number of fairly good books on statistics. One of the best is How To Lie With Statistics by Darrell Huff which was written in 1954. There is also the Cartoon Guide to Statistics by Larry Gonick and Woolcott Smith. The problem most statistics books is that they don't focus on what you can do with statistics. It is important if you are a gambler or want to understand artificial intelligence. Most statistics books are super dry and full of bad examples. I would focus on books like The Theory of Gambling and Statistical Logic. It becomes much more interesting when it is applied.
In my dotage I have become interested in radiation epidemiology and, of course, the issue of 'statistical power' comes up repeatedly. As noted, most physicists know only as much statistics as is absolutely necessary for their work. It would be wonderful to introduce statistics by a set of scenarios (possibly with references to relevant parts of the text itself) drawn from, say, 10 distinct fields. To my dismay, I found that chatGPT did a remarkably decent job of telling me what had been missing in my readings. So think of the project as: what value is added above what a reader could find after 30 minutes of AI delving.
I would at least investigate it. I have purchased and read statistics books recently. Mostly the old classics by R.A. Fisher and D.R. Cox and etc, I have a copy of Handbook 91 by Mary Natrella. My questions are pretty simple. I like the "worked examples" approach in Handbook 91.
Definitely there’s an interest for visual pedagogical content. However a book nowadays may not be the most effective way to reach a wider audience, instead of a video or an interactive website. I guess that combining these other media may help you reach more people to get interested in the book.
I took my first statistics class as an evening course, before going to graduate school, and 90% of the work involved doing hand calculations, avg., variance, std. dev., z-scores, t-tests, etc. And I think that gave me a strong grasp of those fundamentals.
If you could do something similar for bayesian statistics I think that would be useful, but not necessarily popular.
But, It would be helpful for beginners. I work with students and I know a lot (like a lot) of students (not math or stats major, other disciplines) are scared of statistics. They are always looking for resources to learn.
if i be honest. textbooks are pointless nowadays. AI can make any stats into a short note and it's only going to get worse. on top of that youtube(not just youtube) has ton of stat videos where people don't need a textbook to read.
Not sure I'd read it as-written. But I'd love a statistics book whereby the chapters are real case studies. I personally learn best when I can apply new theory to a tangible problem (not just an example problem that's been reduced to almost nothing).
However going the 'visual' route might be enough for me to pick it up.
Aren't a zillion statistics books out there already? Yes I've been wanting to read one, and Wikipedia also has lots of good statistics articles. I've been wanting to work through Freedman and Pisani's book but you know how it goes. It's supposed to be excellent.
A blog or YouTube channel would be a nice halfway step to test the waters. A publisher might like to see that anyway. It would give you a natural feel for what the appetite is and where you want to put the level of the presentation.
It depends on what parts of statistics is being taught and the application of each of the leanings and how it relates to the real world.
More generally, I would buy a statistics if it is linked to today's interesting technological breakthroughs and also if it comes as a distilled version for beginners.
Yes. I have so many stats texts and like many technical fields I suspect they are written as if we learn how things were derived historically not how they truly are.
Yes! I have a degree in telecommunication engineering,but statistics is one of my favourite topics to study.
I found things like bayesian inference or MCMC not really easy to master properly
Yes, there is always a need for another "intuitive statistics" book.
However the link you have provided is not the way; it is low on content and high on pretty distractions. Use all sorts of diagrams and graphs primarily, with animations only where required. The key is to always relate to something in the real world so one can see its actual relevance. Also tie it back to other fields of mathematics so one can see how they all come together.
And of course Nassim Taleb's works are a good source of inspiration. Here is a great video summarizing Taleb's ideas nicely Pareto, Power Laws, and Fat Tails - https://www.youtube.com/watch?v=Wcqt49dXtm8
I’m weird. I read textbooks and instruction manuals. I read incredibly fast though, so it fits nicely. There’s always some deeper learning to be gained in those pages.
Hi, I would read it if it came with a GitHub of python code that I can modify for my own use. The fewest dependencies, the better.
I like Statistical Rethinking, but unfortunately the solver was very slow and I didn't end up using it much - not the author's fault, it's probably my computer that's way too old.
Warning: ramble ahead. I am not a statistician. This is based on my experience as a physicist-by-training exploring the more theoretical side of ML/AI.
Statistics is big. It's hard to answer without knowing what you plan on covering. I agree that most books (I've seen) suck. Wasserman all of statistics is the best I know, for my purposes. The problem I've found with most statistics books is trying to be too cute and clever by catering to a certain crowd thereby justifying holes that make it harder to truly understand and making the subject seem an incoherent patchwork.
The other problem that's even harder to solve is that the majority of people picking up a statistics book don't really think they need to learn statistics, just certain pieces. Which makes statistics seem less coherent and thus furthering the perception that statistics is in fact incoherent, leading to special background books that say "here's all you really need to know about statistics."
The final problem is statistics IS kind of incoherent as most often presented, especially when it tries to be what I'd roughly call "backward compatible". Why is so much time spent on p-values, for example? Is that really what a consistent modern perspective of statistics entails[1]? But backward compatibility forces it's inclusion, because that's what people still use and have used because they never got taught of "philosophy" of statistics, but rather rules. If you don't teach those rules everyone else uses, you're making them spend even more time on statistics than the small amount they're already unhappy spending. So the lowest common denominator is taught, which is incoherent. It's a vicious cycle that is not the fault of statisticians.
To an outsider there's not unifying dominant through lines because the field is justified largely through its application. There's no obvious "philosophy" or vibe of how to think about statistics. What I mean is that one approaching the field doesn't really grow more comfortable with it and doesn't really feel like they're advancing to a more coherent view, so they're less motivated to try to grok it, because it simply doesn't look like there is something to grok. With all of previous factors contributing to this.
So I guess the point of my ramble is that I think what's missing the promise of a reward for learning statistics "properly", really understanding it. In my biased view this is largely because it's not often presented to the non-professional statistics student as if there is a clear thing to understand, more just a collection of tools. And I think this is partly because the author already knows the reader doesn't want to spend time on it. So my advice would be, come up with a clear story of what statistics is. Promise and fulfill the promise that it is worth spending the time in a more abstract world for a little bit because the end is worth it. That it will save time and frustration in the long run because it won't just be a collection of basically faith-based tools with mystical rituals they will feel uncomfortable with for the rest of their careers. Tell that coherent story and then you can demand more commitment from your audience[2]. Maybe that cuts down your audience size but I think the net effect will be more people understanding statistics and what it actually is.
What would be the main threads and themes in your book? How would you justify spending time on it? What would require as necessary background? I don't understand the subject enough to offer any opinion. The most coherent things to me are convergence types and bounds. And then do you include computational approaches? For example in practice I'd say 90%+ of people would be better off using bootstrapping analysis for errors in most real-world cases compared to the standard "approved" approaches, but that's not very satisfying and the theory of it is certainly hard to integrate cleanly. TL;DR I don't envy anyone writing a statistics textbook.
[1] This is just a single aspect, but it gets to a broader problem. This is a really hard problem to overcome. So much of academia is taught p-values and to so much of academia that is basically the hardest math they know, and they work very hard to follow the rules they were taught, which to them is statistics. I don't know how you tell them to re-learn something especially when the answer is more math. That makes them even more hesitant and less likely to fully embrace a deeper understanding of the field.
[2] maybe that's the larger point. The analogy is maybe teaching calc vs. algebra based physics. Algebra based physics is harder to teach and learn, and it is less coherent and complete. There is not much meat in it, mostly rote tools. IMO conservation laws like energy and momentum and solving shit from there would leave a better taste for what physics is, but instead they are often used in one lecture to derive kinematic equations which are then elevated because students can memorize them. I guess it's better than nothing but it leaves people with the wrong impression of what physics is, just like I think most statistics leave a bad impression of what statistics is. Make the statistical equivalents of something like conservation laws the central objects. Something that makes it a coherent story that builds and rewards.
No idea about statistics, but in most physict courses in my university, they recomend 3 books:
1) The main book, that has a complete explanation and is well ordered. It's for learning.
2) Tha Landau book, that is super short and hard. It's only to check you didn't miss any important formula or topic.
3) There Feynman book, that is anassorted colection of fairytales for physicist. It's a pleasure to read it but you must already read 1 to understand it.
4) The Shaum book, that is almost a colection of exercices. Some people hate it. Some people love it. I like it as a companion to theother books.
I guess you are complaining that 1 is boring and want to write 3. It's a good idea, but it's harder than expected.
Similar to an old idea I had about how every programming language needs three books:
1. Basic introduction.
2. Reference tome, which has absolutely everything.
3. Cookbook with style advice for the more advanced student, which assumes you've read 1 and can look up various details in 2.
These days, 2 would be a wiki and 1 would likely be a bunch of pages on that wiki, but it's still good if you have someone sit down and write 3.
I think you're describing the the Diátaxis framework [1], which would further split your (1) into fully guided tutorials and discursive explanations.
[1]: https://diataxis.fr/
Back in my day, the 3 books for programmers were Knuth vol 1, Knuth vol 2, and Knuth vol 3. ;)
Most german speakers will look back at the 2000 pages of "Java ist auch eine Insel" in terror, but it was actually all three books in one.
My personal opinion is that statistics textbooks usually come from a prescriptive perspective, and that makes it challenging for the reader to get visceral intuition for what is actually going on. Any reader would be far better off just visualizing the damn distribution / samples and using reasonable judgement, instead of implicitly assuming a Gaussians distribution and blindly memorizing tests / formulae. Making the distributions explicit allows us to model them and get an intuition for what the samples are telling us. I would whole-heartedly recommend the Model based machine learning book to anyone (online version is free) https://mbmlbook.com/
How do you make 'reasonable judgements'? How do you tell whether someone else made reasonable judgements? How do you judge other people's intuition?
Modelling distributions explicitly sounds nice, yes.
Look at the histogram and think about what distribution one could reasonably impute from samples. And what you would set as bounds for "outliers", per your needs. While we're at it, let me also say that it might be useful to specify outlier bounds not just based on the spread in sample values, but the costs/payoffs they imply for your application.
If you are not doing something crazy, most reasonable people would agree with your judgement. Conversely, if you are making non-obvious inferences where reasonable people disagree, you are in murky water and no sophisticated statistical method will save you. Math is not magic; theorems merely recycle (launder) modeling assumptions into results.
Yup. And thinking through the physical realities or whatever real world constraints exist.
>Look at the histogram and think about what distribution one could reasonably impute from samples. And what you would set as bounds for "outliers", per your needs. While we're at it, let me also say that it might be useful to specify outlier bounds not just based on the spread in sample values, but the costs/payoffs they imply for your application.
Only people with prior education/training in statistics are capable of doing this. The people who don't need a textbook.
Something like 60% of US adults read at or below the 6th grade level, and 25% of US adults struggle to comprehend graphs or charts entirely. Someone who has no idea what a standard deviation is can't intuit about distributions. I think you're dramatically overestimating the average person.
> Someone who has no idea what a standard deviation is can't intuit about distributions.
I disagree vehemently with this claim. I could cite my experience in teaching this topic to liberal arts / humanities college students in the US, but it is really more obvious than that. Anyone can understand a histogram easily and far more intuitively than they can understand the formula for a standard deviation and whether it must divide by N or N-1. Statistics courses and textbooks get stuck on that kind of pedantry, and most students end up missing the forest for the trees.
My larger issue, any time I have tried to learn statistics, is how fast the notation moves. You end up flipping back pages and pages just to double-check a definition that was given once and is now being extended syntactically. It's infuriating.
I've got a Bachelors and two Masters degrees in CS/Math, but yet I feel Probability and Statistics is my greatest weakness. I just cannot grok it / build an intuition for it, and believe me, I've tried. My biggest gripe with Prob/Stats textbooks is that it's very hard to explain things without needing to rely on measure theory.
Maybe probability and statistics are a skill issue on my behalf, but what I absolutely loathe is the absolute lack of standardisation when it comes to notation in measure theory. Every textbook does it differently. All of them assume that their notation is the one everyone uses. Nobody bothers to explain _what_ the notation means. If you ask me, every bit of new notation should be introduced with a sentence or two on "how to read this symbol in your head" - especially when there are indices, subscripts and superscripts involved. It's especially terrible for measure theory because there's so much "implicit" information you're supposed to gather from the context - but in a way I understand it, because if every bit of notation of absolute and complete, I imagine it would be quite hard to type up.
Anyways, my rant on measure theory notation aside - I would absolutely read yet another Prob/Stats textbook. But unfortunately I will also drop it really quickly if the author doesn't show me any "notation-sympathy" :)
Most are dry text. I rewrote a stats 101 book in the form of an operations books used by a coffee shop owner
How would it overlap or differ from 'Statistical Rethinking'?
This is widely regarded as the most accessible intro textbook to Bayesian statistics.
https://xcelab.net/rm/
+ Krushke’s Doing Bayesian Data Analysis, and for the very basics there is Downey’s Think Bayes. I guess there might be a gap in the literature for a different approach, heck, I’d read it, but there is some very good material already out there.
Maybe my expectations were too high given all the online praise, but I've been working through it for the past two weeks and I've been underwhelmed:
- The language is often very vague and imprecise, so it's more difficult than it feels it should be
- Concepts are sometimes introduced "at random" in ways that only really make sense in hindsight. So as you're reading you're left scratching your head as to why something was brought up.
- There are constant philosophical and historical digressions that seem to hold deeper meaning, but maybe once you know the topic already.
- Similarly, constantly talking about people that take issue with the method. People not liking the method is a constant theme (they seem really butthurt about this?). But the craziest part is this all done before you even really understand what the method is!!
- The editor must have placed some strict requirement of saying "Bayesian" at least five times per page.
- No index. Useless table of context. But lots of end-notes you feel compelled to flip to constantly
Overall it feels like a textbook written to impress other statistics professors - and as an outlet for the author to air some frustrations with how people do statistics (which may be completely valid!)
The overall structure and objectives seem solid for the most part. It's just a lot of the details aren't great. The problems (so far) have been good. The examples in the text are fun and compelling, but you have to do your own legwork to actually pick through all the prose and tie the pieces together - to figure how it fits together mathematically. Fortunately AI helps as a tutor
Rethinking is indeed written for people who have some or even a lot of existing proficiency in statistics, but for whom it never really clicked. If that’s not your background, I can understand that it might feel disjointed.
I appreciate the sanity check :) I'm still enjoying studying the subject
I'm a nerd and do a lot of stats for my job and I would not read a statistics textbook.
I read a lot of informational things, but math / stats / software has always felt like an area where a book is just the wrong format.
If I were you I'd make an interactive website like SQLZoo or a video series like StatQuest.
Those are educational formats that really clicked with me for whatever reason.
Having also studied statistics in university (undergrad), something I kept running into is that you can't really unlock the intuition for many concepts without taking more advanced courses. For example, degrees of freedom shows up as early as AP Statistics, but even a non-rigorous visual explanation of it leans on linear algebra, which most students don't see until much later.
I think more resources like seeing-theory would be great since stats books are almost universally dry (Blitzstein being a notable exception), but I'm not sure how easily more advanced concepts lend themselves to visual explanation in a way that's digestible for a non-stats person.
For degrees of freedom specifically, check out these videos: https://www.youtube.com/playlist?list=PLmtbsGjqSdcbF261LACoV...
It's still complicated but the visualizations help a lot!
I feel this boils down my learning journey as well. You start unraveling a very good intuition about the underlying concepts MUCH much later, but partly because those intuitions themselves are never conveyed and are supposed to be learned from the proofs, and are an indirect product of learning.
Practical advice:
- put a lot of 'teaching' into the book. Don't write so the learner memorizes; instead, motivate each discussion so that the learner understands, and gains what Wirth calls _coverage_.
- ensure there are no errors; this will turn off learners.
- keep a light tone, be yourself not a stuffed shirt. A good model for such writing is Nassim Taleb's non-technical books.
- never seem to be in a hurry. Many good books were ruined because midway the author lost interest and just hurried up.
- don't be too verbose, and neither too concise. Every sentence should do required work; learn how to write (if needed) and practice daily.
- ensure the book uses good, comfortable fonts, and good typography. Must learn this. Many good books ruined due to lack of fluency effect.
If you conscientiously do these things, there will always be readers for your book.
Good luck!
A long time ago, I took a "statistics for engineers" class in order to graduate. I slept through most of the classes. It sucked, and 70 percent of it was just "distribution of the week." It did not help that homework was optional for half of it.
At some point in my professional career I started reading non-fiction books and even bought a used statistics textbook for 10 bucks on abebooks. I didn't end up actually reading it until 12 years later during the COVID lockdown. I ended up shooting for 10 pages a day, 7 days a week. If those 10 pages included review exercises, it would be a long night.
Could just be the right book at the right time, but this one really helped me understand stuff beyond the normal HS math stuff, like RMS-error, calculating correlation, the difference between standard error and standard deviation, the relationship between sample size and standard error, t-tests, and chi-squared. Working as an SRE/release engineer, this stuff really helped me overcome a lot of _bad_ canary data analysis my predecessors had constructed.
That book was the 3rd edition of Statistics by Freedman et al.[1] One thing I want to complement was getting the pedagogy right. Most chapters have strong narrative hooks, several "check your knowledge" problems, review exercises, and post chapter bullet points to assist with spaced repetition. There's even a series of "special" review exercises covering entire sections of the book, ie exams.
For the HN crowd I should also probably note that the book is almost entirely non-bayesian and not intended to prepare readers for further coursework. You will not learn normal phraseology like "IID," "random variable" or "kernel".
[1]: https://www.amazon.com/dp/B00SLB5Q72?lv=shuf&channelId=520&p...
There are a number of good books to explain the main concepts of statistics without getting bogged down in the technical details. As a grad student at a public university in the USA, I worked as a teaching assistant for the undergraduate course Introduction to Statistics. The course used SPSS software to enable the students to perform statistical analysis on data sets. Although we offered a good textbook (listed below), many of the students struggled with the main concepts, so my colleagues and I added a optional supplemental reading list to the syllabus. The following books are the ones we recommended to the students. I have read all these books, and they are pretty good at explaining statistics to anyone interested in learning the material at a conceptual level; anyone can read and understand these books, regardless of their educational background.
Ian Ayres (2007) Super Crunchers: Why Thinking by Numbers is the New Way to Be Smart.
Joseph Healey (2020) Statistics: A Tool for Social Research [This is the textbook used in the class, which I highly recommend].
John Allen Paulos (1988) Innumeracy: Mathematical Illiteracy and Its Consequences.
Nate Silver (2012) The Signal and the Noise: Why So Many Predictions Fail, but Some Don't.
Nassim Nicholas Taleb (2005) Fooled by Randomness, second edition.
Nassim Nicholas Taleb (2010) The Black Swan: The Impact of the Highly Improbable, second edition.
Did you see that Andrew Gelman and others just published Bayesian Workflows? https://avehtari.github.io/Bayesian-Workflow/ It sounds similar to what you're after, away from describing the logic of models and the maths, instead it's about (quote from intro) 'There are all sorts of tacit knowledge in applied statistics that do not make it into published papers and textbooks. The present book is intended to put some of these ideas out in the open'
Where would your book fit into this?
Even if I would not read it back to back on release, it would be a pleasure to have a reliable and citable reference on hand. Whenever I stumble upon new complex problems, outside of the regular, often fairly repetitive statistical questions of my field, I need to rely on many searches and LLM queries to find my answer. I wonder if a book could actually replace all that, but it would be my first place to check.
I think statistics are important for everyone to understand; however, I am wondering what shape that book would take and who the target audience would be. It would be wildly different to write a book like that for people who already have a very clear or existing knowledge on books than for people who are just dipping their feet and want to know more.
There are a number of fairly good books on statistics. One of the best is How To Lie With Statistics by Darrell Huff which was written in 1954. There is also the Cartoon Guide to Statistics by Larry Gonick and Woolcott Smith. The problem most statistics books is that they don't focus on what you can do with statistics. It is important if you are a gambler or want to understand artificial intelligence. Most statistics books are super dry and full of bad examples. I would focus on books like The Theory of Gambling and Statistical Logic. It becomes much more interesting when it is applied.
In my dotage I have become interested in radiation epidemiology and, of course, the issue of 'statistical power' comes up repeatedly. As noted, most physicists know only as much statistics as is absolutely necessary for their work. It would be wonderful to introduce statistics by a set of scenarios (possibly with references to relevant parts of the text itself) drawn from, say, 10 distinct fields. To my dismay, I found that chatGPT did a remarkably decent job of telling me what had been missing in my readings. So think of the project as: what value is added above what a reader could find after 30 minutes of AI delving.
I would at least investigate it. I have purchased and read statistics books recently. Mostly the old classics by R.A. Fisher and D.R. Cox and etc, I have a copy of Handbook 91 by Mary Natrella. My questions are pretty simple. I like the "worked examples" approach in Handbook 91.
https://stewartschultz.com/statistics/books/Statistics_Terri...
Definitely there’s an interest for visual pedagogical content. However a book nowadays may not be the most effective way to reach a wider audience, instead of a video or an interactive website. I guess that combining these other media may help you reach more people to get interested in the book.
Another exemple of a successful visual pedagogical content is: https://www.byhand.ai/
Yes, with the caveat that someone else with knowledge in the field has to recommend it to me. Partner up with someone who teaches statistics.
I took my first statistics class as an evening course, before going to graduate school, and 90% of the work involved doing hand calculations, avg., variance, std. dev., z-scores, t-tests, etc. And I think that gave me a strong grasp of those fundamentals.
If you could do something similar for bayesian statistics I think that would be useful, but not necessarily popular.
Would I read it? No
But, It would be helpful for beginners. I work with students and I know a lot (like a lot) of students (not math or stats major, other disciplines) are scared of statistics. They are always looking for resources to learn.
if i be honest. textbooks are pointless nowadays. AI can make any stats into a short note and it's only going to get worse. on top of that youtube(not just youtube) has ton of stat videos where people don't need a textbook to read.
Not sure I'd read it as-written. But I'd love a statistics book whereby the chapters are real case studies. I personally learn best when I can apply new theory to a tangible problem (not just an example problem that's been reduced to almost nothing).
However going the 'visual' route might be enough for me to pick it up.
Aren't a zillion statistics books out there already? Yes I've been wanting to read one, and Wikipedia also has lots of good statistics articles. I've been wanting to work through Freedman and Pisani's book but you know how it goes. It's supposed to be excellent.
A blog or YouTube channel would be a nice halfway step to test the waters. A publisher might like to see that anyway. It would give you a natural feel for what the appetite is and where you want to put the level of the presentation.
It depends on what parts of statistics is being taught and the application of each of the leanings and how it relates to the real world.
More generally, I would buy a statistics if it is linked to today's interesting technological breakthroughs and also if it comes as a distilled version for beginners.
Yes, I’d read it.
But beware of opinions.
Don’t let people put you down, especially here in HN, where people are perceived to smart. Smart doesn’t equal sensible or unbiased.
Many books are written to scratch the itch of the author. Just like an open source project. It is a work of love.
Yes. I have so many stats texts and like many technical fields I suspect they are written as if we learn how things were derived historically not how they truly are.
Yes! I have a degree in telecommunication engineering,but statistics is one of my favourite topics to study. I found things like bayesian inference or MCMC not really easy to master properly
We have the perfect answer to your statement; https://news.ycombinator.com/newpoll
Such a bizarre question. Many, many people have read statistics textbooks.
Can you write one that's more worth reading than the standard ones? Don't answer that question, just prove it.
I probably would buy/read it, but I wouldn't rush out to buy one. I barely passed stats in college and I wish had a better intuitive understanding.
Don’t ask. Start from a few blog posts and see traction, read feedback.
You will also see hos long it takes - and what is thd difference between an idea and making it real.
Yes, if those statistics served accounting, and the homework was using something like Python in a Jupyter notebook.
I would! If you are thinking to write one, do it!
This sounded to me like that old “would you steal a car?” anti-piracy campaign until I clicked to read the post :)
Do the exercises. The end.
Yes surely. I like ML, >D>S and such but lacking in stats background I wishh I had.
Yes, I love it and would do so
Probably.
I would read it
Yes foundations of agnostic statistics and all of statistics
Yes, there is always a need for another "intuitive statistics" book.
However the link you have provided is not the way; it is low on content and high on pretty distractions. Use all sorts of diagrams and graphs primarily, with animations only where required. The key is to always relate to something in the real world so one can see its actual relevance. Also tie it back to other fields of mathematics so one can see how they all come together.
A good example to study is How to Measure Anything: Finding the Value of Intangibles in Business by Douglas Hubbard. Detailed review at - https://www.lesswrong.com/posts/ybYBCK9D7MZCcdArB/how-to-mea...
And of course Nassim Taleb's works are a good source of inspiration. Here is a great video summarizing Taleb's ideas nicely Pareto, Power Laws, and Fat Tails - https://www.youtube.com/watch?v=Wcqt49dXtm8
I’d probably sample it randomly yes.
Crap jokes aside, I mostly use Statistics In A Nutshell. It’s pretty ok. I say that as a member of the RSS for 30 odd years.
I’m weird. I read textbooks and instruction manuals. I read incredibly fast though, so it fits nicely. There’s always some deeper learning to be gained in those pages.
Good luck if you do this.
I would read a statistics textbook, if there were plenty of exercises, with the answer and explanation, which is usually lacking in textbooks.
Ready the theory is nice but you only learn by solving problems that uses it.
Hi, I would read it if it came with a GitHub of python code that I can modify for my own use. The fewest dependencies, the better.
I like Statistical Rethinking, but unfortunately the solver was very slow and I didn't end up using it much - not the author's fault, it's probably my computer that's way too old.
I have some statistic book in paper, trying to solve some lemmas from time to time, so I will not read any slop.
yes
Warning: ramble ahead. I am not a statistician. This is based on my experience as a physicist-by-training exploring the more theoretical side of ML/AI.
Statistics is big. It's hard to answer without knowing what you plan on covering. I agree that most books (I've seen) suck. Wasserman all of statistics is the best I know, for my purposes. The problem I've found with most statistics books is trying to be too cute and clever by catering to a certain crowd thereby justifying holes that make it harder to truly understand and making the subject seem an incoherent patchwork.
The other problem that's even harder to solve is that the majority of people picking up a statistics book don't really think they need to learn statistics, just certain pieces. Which makes statistics seem less coherent and thus furthering the perception that statistics is in fact incoherent, leading to special background books that say "here's all you really need to know about statistics."
The final problem is statistics IS kind of incoherent as most often presented, especially when it tries to be what I'd roughly call "backward compatible". Why is so much time spent on p-values, for example? Is that really what a consistent modern perspective of statistics entails[1]? But backward compatibility forces it's inclusion, because that's what people still use and have used because they never got taught of "philosophy" of statistics, but rather rules. If you don't teach those rules everyone else uses, you're making them spend even more time on statistics than the small amount they're already unhappy spending. So the lowest common denominator is taught, which is incoherent. It's a vicious cycle that is not the fault of statisticians.
To an outsider there's not unifying dominant through lines because the field is justified largely through its application. There's no obvious "philosophy" or vibe of how to think about statistics. What I mean is that one approaching the field doesn't really grow more comfortable with it and doesn't really feel like they're advancing to a more coherent view, so they're less motivated to try to grok it, because it simply doesn't look like there is something to grok. With all of previous factors contributing to this.
So I guess the point of my ramble is that I think what's missing the promise of a reward for learning statistics "properly", really understanding it. In my biased view this is largely because it's not often presented to the non-professional statistics student as if there is a clear thing to understand, more just a collection of tools. And I think this is partly because the author already knows the reader doesn't want to spend time on it. So my advice would be, come up with a clear story of what statistics is. Promise and fulfill the promise that it is worth spending the time in a more abstract world for a little bit because the end is worth it. That it will save time and frustration in the long run because it won't just be a collection of basically faith-based tools with mystical rituals they will feel uncomfortable with for the rest of their careers. Tell that coherent story and then you can demand more commitment from your audience[2]. Maybe that cuts down your audience size but I think the net effect will be more people understanding statistics and what it actually is.
What would be the main threads and themes in your book? How would you justify spending time on it? What would require as necessary background? I don't understand the subject enough to offer any opinion. The most coherent things to me are convergence types and bounds. And then do you include computational approaches? For example in practice I'd say 90%+ of people would be better off using bootstrapping analysis for errors in most real-world cases compared to the standard "approved" approaches, but that's not very satisfying and the theory of it is certainly hard to integrate cleanly. TL;DR I don't envy anyone writing a statistics textbook.
[1] This is just a single aspect, but it gets to a broader problem. This is a really hard problem to overcome. So much of academia is taught p-values and to so much of academia that is basically the hardest math they know, and they work very hard to follow the rules they were taught, which to them is statistics. I don't know how you tell them to re-learn something especially when the answer is more math. That makes them even more hesitant and less likely to fully embrace a deeper understanding of the field.
[2] maybe that's the larger point. The analogy is maybe teaching calc vs. algebra based physics. Algebra based physics is harder to teach and learn, and it is less coherent and complete. There is not much meat in it, mostly rote tools. IMO conservation laws like energy and momentum and solving shit from there would leave a better taste for what physics is, but instead they are often used in one lecture to derive kinematic equations which are then elevated because students can memorize them. I guess it's better than nothing but it leaves people with the wrong impression of what physics is, just like I think most statistics leave a bad impression of what statistics is. Make the statistical equivalents of something like conservation laws the central objects. Something that makes it a coherent story that builds and rewards.