Hacker News

> General purpose models are always more robust and generally better than smaller narrower models

I feel like this is just the marketing conflation of AI=LLM, versus regular old ML? We're never going to need to deploy a full reasoning model on a low-power device just to do some fancy image recognition in the field. Specialised ML models are just intrinsically able to be a lot more efficient than their generalist equivalents

> General purpose models are always more robust and generally better than smaller narrower models.

What do you mean with more robust?

You're getting downvoted, but you're completely right. There are very few cases in which narrowing a model down is buying you anything worthwhile.

It seems like for LLMs, "general intelligence" is expensive, but "one more domain" is fairly cheap.

MoEs don't route like most people imagine. They aren't learning topic based experts despite the name

The original Mixtral paper [0] (in the "Routing analysis" section) found:

"surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic"

A quick skim of more recent analysis on MoE shows that this hasn't changed. MoE models do appear to work, but don't appear to do what the name implies, if anything they're routing based on the structure of the text and not the semantic content (and we're still not entirely sure what they're doing).

0. https://arxiv.org/pdf/2401.04088

Right MoE is a tradeoff between efficiency and intelligence.

I haven't tried it, but I saw Leanstral, an LLM specialized in writing Lean proofs, posted on HN recently and it claims to outperform some larger general purpose models. It didn't beat Claude Opus, but it seems to do decently at one tenth the cost. It's plausible that further research could yield other models that are smaller and more effective at limited tasks, reversing the trend of ever growing models.

What about it?

Your examples (math, literature) involve natural language. It stands to reason that a general language model will be more competitive in those domains. If you want examples of successful domain-specific models, look at AlphaZero and AlphaFold. LLMs aren't anywhere close to achieving that level of competence at abstract strategy games or protein folding.

"This will never work" is a pretty confident assertion for a field that's so young and rapidly evolving.

> No this will never work.

This bet is too early.

> Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this?

Because they have a very early product and they could train it, brute force, with access to an extraordinarily large pool of money. So did all the other labs. Because it was thus easier to scrape everything rather than spend enormous effort (with tools that did not really exist) to partition the training set. Any number of other "because"s.

It's just what they are doing now and it showed the earliest results.

LLMs are still less intelligent than rats, which have tiny brains.

DeepMind did release a math specific model. And OpenAI has released a coding specific model.

The answer to your question is “because the market isn’t big enough”, not because it doesn’t work. Why would knowing about 2019 internet memes help you in any way at coding?

You will probably want a search engine though. Perhaps a small LLM would work well as a component for that?

You can run a small model off a home generator — so in an emergency, you’d turn on both the generator and information service, eg, a mesh for “quick” responses querying that huge collection of information.

That way your machine that, eg, normally plays video games or does AI work can support relief efforts by supporting emergency response IT. You don’t need to mothball the machine, just have an “emergency” boot USB than can run the services from your home generator.

You don’t even need to bring it with you: turn it on and leave it “best effort” at home, while you continue to use it via WAN.

> If you do carry it to an enclave of civilization that has the right power, are they going to let you run it when it cuts into light for surgeons or heat to sterilize water?

Knowing humans? They'd probably take it by force and run it for themselves instead of providing light and heat to surgeons and water sterilizers...

/daily dose of cynism

The 12B QAT model is really overlooked because the tech industry has been so desperate for the LLM bet to play out to "product market fit" (which means please IPO now) that it has become convinced that coding models are the only things that matter.

If you are rebuilding society most of this knowledge is useless for centuries. You don't have enough labor to build and maintain factories. You will spend centuries in the hunter gather phase struggling to survive, while slowly building agriculture. You will be lucky if you can teach your grandkids to read - since that will be a useless skill.

Print important knowledge on paper and store it in a desert. in 2000 years society and population will advance enough to get a jump start based on our knowledge.

I feel like you could get a lot more quality of life improvement for more people with the money if you spent it on low tech solutions, eg more efficient cooking stoves for people still cooking with biomass, or solar microgrids for areas without electricity.

They want to ask the iOS Foundation model (frontier on device intelligence for something small) for instance about emergency procedures and life-saving info. I wouldn’t trust that model with much at all though. More likely to find what you need from miniature survival guides.

Fire!