Hacker News

I think this is an outcome of playing with systems that we don’t actually understand very well on a deep level. We have found recipes that work, and have used graduate descent (so many PhD theses) to understand and optimize these systems, but we’ve built a model of language that also encodes reasoning, and those things are extraordinarily complex. Providing proper mathematical rigor for their explanation requires, as you say, the level of commitment equivalent to a math degree. Also, because language is so complicated and subtle, you need highly precise statistical models of it, and that just takes a shitload of data; hence the scale. You can train toy models at home, but they will suck because you actually need massive amounts of data to capture the nuance that language requires. And that’s to say nothing of multimodal models.

Machine learning is more like biology, physics or law than like CS or math.

In the software engineering world, we're accustomed to the reality that to do great work, all you need is a sharp mind and a cheap laptop. Better tools help, but aren't really a necessity. You could probably become a top-notch Linux kernel engineer with nothing but dedication and self-study, a $300 laptop and a shabby apartment with internet access.

You just can't really learn ML that way, just like you can't do it with banking, professional trading or medicine.

> Practical/applied stuff seems to require levels of compute, training data, etc. that make it unapproachable for a home hacker for everything beyond basic exercises like a handwritten digit classifier.

Compute is only an issue if you want to train large foundational models from scratch. You can easily train non-neural models on CPU, and fine-tune smaller models (say <4B) on a commodity GPU.

For training data there's lots of datasets online (like Kaggle), but you can build your own and can label things yourself or get a language model to label them for you. Getting good data is the hardest part of ML so this is a good exercise.

Then do some hobby projects: classify whether your photos contain a photo of you or a particular friend/family member, predict the expense category from bank transaction exports, classify your emails into categories, extract book recommendations from Hacker News, ...

Get Sebastian's book and build an LLM from scratch. The problem with most textbooks (on any subject, it's not unique to ML/AI) is they usually try to build the theoretical foundations first. In many cases things are better taught top down.

I completely understand where you're coming from. While understanding the math and architectures of these large models can be edifying, it lacks some of the liberating/empowering spirit of traditional software. You can build a bespoke to-do app that is better suited to your needs than anything produced by some productivity software company, but you're totally beholden to one of the few big players for access to an effective LLM.

But there is so, so much interesting work and research to do at a much smaller scale. I work on drug discovery models, and none of my work requires data or infrastructure that are not generally accessible to the public (it's nice to have the computing resources of my institution but there are other ways to access a GPU).

So if you are really interested in this stuff, I would suggest looking at the endless supply of interesting problems in domains that are not dominated by huge frontier labs.

Several years ago I put together a classifier for putting my bank account statements into categories (groceries, housing etc). I use it for doing my accounts and budgeting. Maybe you have a similar problem? There are still lots of applications for small classifiers, you don't need a SOTA LLM for everything.

I have the books on my shelf for 5 years now and never encountered the need to build an AI so i haven't touched it yet. Don't feel bad about using AI to learn but not build one.

You can google stuff to learn without the need to build a search engine too.

I have however learned a lot about math! And other related stuff. I'd argue if you extend your low level skills in math and code when the need for an AI comes up, simple weather / sales forecast, who knows, you'll have an easier time building one.

In my experience, until you implement the math of a NN in numpy or something similar, it's easy to feel like a library is abstracting something away from you. Many people in ML focused academia stand on those libraries. They may conceptually understand chain rule but do not manually implement the math. The same feeling exists for encryption. There is a culture of shying away from rolling out your own ops, so it can easily feel like your understanding is segmented to some degree.

I think it's one of those things where you have to work in the field to really get it and be productive. You just don't have the time or resources otherwise.

if you are willing to put in the work you can devise a hyper specialization for which you can create an RL playground (which is likely absent in the frontier labs), then you can obtain cutting edge results on it by post training an open weights model using minimal resources.

Try making a smaller classifier model for some random problem you have that's immediately useful and doesn't exist yet.

But machine learning stuff feels like it could have room for learning-by-doing applications that aren't entirely pointless, so I wonder if there's some way to approach the topic I'm simply not finding.

That's what Karpathy's videos are for. You end up with a GPT-2.0 equivalent if you follow them to the end.

I feel the same. I feel machine learning is 1) you need strong math knowledge but also intuition. 2) You need actual production level data, a lot of the nuances of machine learning, applied scientists' roles are how to build strong models with real world, messy, inconvenient data. You need to apply various techniques and not just memorize a specific formula.

Compared to software coding, you build websites, apps etc. The barrier to entry is much higher.

I am trying to pivot into machine learning myself, so far my understanding is try to find a software engineer role where you deal with data, model infra, data pipeline, etc. Then within the same company, try to do more modeling work and build intuition and resume credentials in this area. Someone correct me if I am wrong.

Machine learning IS applied mathematics, especially statistics, probability, and optimization. Period.

The "baby steps" are not there just for intelectual satisfaction, the baby steps give you the building blocks to understand things that become progressively more complex. The baby steps are tools, that once you get familiar with them, you'll find new and new uses everywhere.

A good practical illustration of this: It is pretty hard to figure out how differential equations can be useful if you don't understand what they are. And yet, I bet that on january of 2020, there was at least one math teacher deep down in the amazonian jungle playing with a compartimental model based on a system of ordinary differential equations, to figure out the diverse scenarios this new epidemic could play out if that scary new virus escaped China and those few cruise ships.

Another example: take the concept of backpropagation: the underlying idea behind it is the chain rule from calculus, used to work backward through a computation and calculate how each parameter affects the error. Now, it is way easier to understand it in a small CNN that you are training to recognize handwritten digits than trying to understand it in the context of a full transformer model. A digit classifier is something that is way easier and affordable for you to fiddle with stuff and develop and intuition on loss functions, gradients, backpropagation and overfitting.

I did the same. I got "Build Your Own LLM From Scratch" last christmas and it is a great resource for learning how LLMs work, from tokenization to generating text. The author makes a great job explaining each new concept. If you know PyTorch (or at least know about tensors), the material will be approachable for you.

Sebastian, if you are reading this, this is an idea for another book: how to orchestrate agents for adversarial code review. I've been looking for learning resources to make something similar to this [1] at a smaller scale. I am aware of codex subagents, though. However, given that Sebastian already has a book for LLMs and another book on planning, I think adversarial code review is an interesting next step.

[1] https://bun.com/blog/bun-in-rust

[2] https://learn.chatgpt.com/docs/agent-configuration/subagents...

I don't like working with neural networks and I always found everything overcomplicated or oversimplified, sometimes both at the same time.

But this book is almost perfect. Every new concept is well described and every is well founded. And every step moves you forward (if you know, you know)

I read it the first time with a study group which was very helpful.

I like it too. Haven't got very far through it yet but just learning about how to use PyTorch and finally getting how back propagation works via autograd etc. was really fun and enlightening.

Karpathy 2.0!

You can go through it locally just fine; all you need to do is be able to run Qwen3 0.6B, which is a tiny LLM. A recent macbook (air or pro), or a PC with a good gaming GPU is perfect, but you can run the model and training examples on a PC CPU too; it will just be much slower. From his book (same examples as the video course):

> "the code examples have been carefully designed to run efficiently on a regular laptop, without the need for any special hardware. But if you do have access to a GPU, certain sections provide helpful tips on scaling up the datasets and models to take advantage of that extra power... The inference and evaluation chapters work well on CPUs and GPUs, while the training-oriented chapters benefit from access to a GPU if you want to reproduce the larger experiments."

So, consumer GPU helps but not essential. No need for expensive cloud GPU time.

You can take any open source model and continue training it. It's nothing like building a CPU.

They’re amazing for interview prep! ;)

he's a well known author on LLMs already.

You can follow his book and learn how to build an LLM from scratch. What is the issue?

It's also not a bad foundation for people who want to understand how these things are trained and work.

>apart from learning how the tech works that is.

some people enjoy learning how the tech works

Why teach people to play guitar once a week if they won't be a new Jimi Hendrix? Why teach people to run if they won't be a new Usain Bolt?

Maybe because they don't want to be an "emerging player" or something, they only want to understand how a new technology that is changing the world works and prefer to learn by doing.

If Sebastian is doing it by its personal brand, he really deserves it. I would trust him because I know he is a reference on his area of knowledge and I know he is because he shares his knowledge instead of only making fancy talks.

Let’s not be this negative, we might be going towards a new direction with technology that could allow small teams to build their own models. Maybe not frontier models. Maybe not close to the others in generalized benchmarks. But perhaps they could be really specialized into solving one particular problem better than the others.

You wrote so many words to say .. nothing of substance .. i still have no clue what you are criticizing.

If you're truly going to build your own model (and commit to the cost of that), then video tutorials will not cut it.

That makes no sense. It's like saying if you want an engineering degree, going to lectures will not cut it. Of course not, you have to do the homework and projects too.

People are going to be upset about this, but any learning material in this day and age is solely made for your own understanding when writing it. The LLMs can be better teachers and understand your level of need and engagement better than anyone else.