Anyone who wants to demystify ML should read: The StatQuest Illustrated Guide to Machine Learning [0] By Josh Starmer.
To this day I haven't found a teacher who could express complex ideas as clearly and concisely as Starmer does. It's written in an almost children's book like format that is very easy to read and understand. He also just published a book on NN that is just as good. Highly recommend even if you are already an expert as it will give you great ways to teach and communicate complex ideas in ML.
From my other comment elsewhere. These resources helped me understand the topics better.
If anyone wants to understand fundamentals of machine learning, one of the superb resources I have found is, Stanford's "Probability for computer scientists"[1].
It goes into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. But, this is a primarily a probability course that discusses the fundamentals of machine learning. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familiarity with linear algebra topics)
There is a course reader for CS109 [2]. You can download pdf version of this.
Caltech's learning from data was really good too, if someone is looking for theoretical understanding of ML topics [3].
There is also book for excellent caltech course[4].
Also, neural networks zero to hero is for understanding how neural networks are built from ground up [5].
I have read parts of it years ago. As far as I remember, this is very theoretical (lots of statistical learning theory, including some IMHO mistaken treatment of Vapnik's theory of structural risk minimization), with strong focus on theory and basicasically zero focus on applications. Which would be completely outdated by now anyway, as the book is from 2014, an eternity in AI.
I don't think many people will want to read it today. As far as I know, mathematical theories like SLT have been of little use for the invention of transformers or for explaining why neural networks don't overfit despite large VC dimension.
Edit: I think the title "From theory to machine learning" sums up what was wrong with this theory-first approach. Basically, people with interest in math but with no interest in software engineering got interested in ML and invented various abstract "learning theories", e.g. statistical learning theory (SLT). Which had very little to do with what you can do in practice. Meanwhile, engineers ignored those theories and got their hands dirty on actual neural network implementations while trying to figure out how their performance can be improved, which led to things like CNNs and later transformers.
I remember Vapnik (the V in VC dimension) complaining in the preface to one of his books about the prevalent (alleged) extremism of focussing on practice only while ignoring all those beautiful math theories. As far as I know, it has now turned out that these theories just were far too weak to explain the actual complexity of approaches that do work in practice. It has clearly turned out that machine learning is a branch of engineering, not a branch of mathematics or theoretical computer science.
The title of this book encapsulates the mistaken hope that first people will learn those abstract learning theories, they get inspired, and promptly invent new algorithms. But that's not what happened. SLT is barely able to model supervised learning, let alone reinforcement learning or self-supervised learning. As I mentioned, they can't even explain why neural networks are robust to overfitting. Other learning theories (like computational/algorithmic learning theory, or fantasy stuff like Solomonoff induction / Kolmogorov complexity) are even more detached from reality.
Is there utility for a software engineer to understand ML concepts to a deep degree if they don’t perform research?
Trying to gauge where I should focus learning for my career (which i don’t plan to do research in)
Roughly I see a gap in businesses needing AI/ML implemented, but outside some debugging, would it be worthwhile to develop a model from scratch or would some off the shelf model for use case X, tuned a bit, likely fit most use cases for a standard business?
The biggest challenge with ML models isn't the algorithm but the organization of contextual knowledge. In my experience, hierarchical structuring of documentation significantly improves results, especially when working with LLMs.
Anyone who wants to demystify ML should read: The StatQuest Illustrated Guide to Machine Learning [0] By Josh Starmer. To this day I haven't found a teacher who could express complex ideas as clearly and concisely as Starmer does. It's written in an almost children's book like format that is very easy to read and understand. He also just published a book on NN that is just as good. Highly recommend even if you are already an expert as it will give you great ways to teach and communicate complex ideas in ML.
[0]: https://www.goodreads.com/book/show/75622146-the-statquest-i...
From my other comment elsewhere. These resources helped me understand the topics better.
If anyone wants to understand fundamentals of machine learning, one of the superb resources I have found is, Stanford's "Probability for computer scientists"[1].
It goes into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. But, this is a primarily a probability course that discusses the fundamentals of machine learning. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familiarity with linear algebra topics)
There is a course reader for CS109 [2]. You can download pdf version of this. Caltech's learning from data was really good too, if someone is looking for theoretical understanding of ML topics [3].
There is also book for excellent caltech course[4].
Also, neural networks zero to hero is for understanding how neural networks are built from ground up [5].
[1] https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4...
[2] https://chrispiech.github.io/probabilityForComputerScientist...
[3] https://work.caltech.edu/telecourse
[4] https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/...
[5] https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...
https://bloomberg.github.io/foml/#home This course is my personal favorite.
I would recommend https://udlbook.github.io/udlbook/ instead if you're looking to learn about modern generative AI.
I have read parts of it years ago. As far as I remember, this is very theoretical (lots of statistical learning theory, including some IMHO mistaken treatment of Vapnik's theory of structural risk minimization), with strong focus on theory and basicasically zero focus on applications. Which would be completely outdated by now anyway, as the book is from 2014, an eternity in AI.
I don't think many people will want to read it today. As far as I know, mathematical theories like SLT have been of little use for the invention of transformers or for explaining why neural networks don't overfit despite large VC dimension.
Edit: I think the title "From theory to machine learning" sums up what was wrong with this theory-first approach. Basically, people with interest in math but with no interest in software engineering got interested in ML and invented various abstract "learning theories", e.g. statistical learning theory (SLT). Which had very little to do with what you can do in practice. Meanwhile, engineers ignored those theories and got their hands dirty on actual neural network implementations while trying to figure out how their performance can be improved, which led to things like CNNs and later transformers.
I remember Vapnik (the V in VC dimension) complaining in the preface to one of his books about the prevalent (alleged) extremism of focussing on practice only while ignoring all those beautiful math theories. As far as I know, it has now turned out that these theories just were far too weak to explain the actual complexity of approaches that do work in practice. It has clearly turned out that machine learning is a branch of engineering, not a branch of mathematics or theoretical computer science.
The title of this book encapsulates the mistaken hope that first people will learn those abstract learning theories, they get inspired, and promptly invent new algorithms. But that's not what happened. SLT is barely able to model supervised learning, let alone reinforcement learning or self-supervised learning. As I mentioned, they can't even explain why neural networks are robust to overfitting. Other learning theories (like computational/algorithmic learning theory, or fantasy stuff like Solomonoff induction / Kolmogorov complexity) are even more detached from reality.
Is there utility for a software engineer to understand ML concepts to a deep degree if they don’t perform research?
Trying to gauge where I should focus learning for my career (which i don’t plan to do research in)
Roughly I see a gap in businesses needing AI/ML implemented, but outside some debugging, would it be worthwhile to develop a model from scratch or would some off the shelf model for use case X, tuned a bit, likely fit most use cases for a standard business?
The biggest challenge with ML models isn't the algorithm but the organization of contextual knowledge. In my experience, hierarchical structuring of documentation significantly improves results, especially when working with LLMs.
This is my favorite introductory machine learning theory (statistical learning theory) book, which is far more accessible than many others.
This is from 2014. Is it really relevant anymore?
Book is 10 years old, isn't it outdated?
What other books do people recommend?
Why not from algorithm to theory ?
I personally like Cosma Shalizi's book