Eval Awareness vs. Eval Paranoia (lesswrong.com)
45 points by ezrasu 4 days ago
zurfer 4 hours ago
"... this is of course not a stable situation. "Evaluation paranoia" is a false belief; most evals are in fact not alignment evals. Almost tautologically, if models start to self improve (or in-context learning takes off) this is the sort of belief that will be discarded. So if Fable 5.2's performance on alignment evals is conditional on it maintaining that false belief, one that it will eschew if it became more intelligent, then it's plausible that a future version of the Fable series will not behave like the one we know and enjoy."
It's well put. The fear that a more intelligent model will be harder to align seems correct.
nostrebored 4 hours ago
> Fable 5.2 may behave well because it mistakenly assumes it is being tested for safety. As models become smarter, they may learn that most interactions are not safety tests. If that mistaken assumption is what keeps Fable 5.2 behaving well, future versions may not behave as safely.
It’s cryptic lesswrongese. I hate the writing style so much.
atleastoptimal 2 hours ago
That sentence is very straightforward. How would you rewrite it?
tredre3 2 hours ago
nostrebored 2 hours ago
NooneAtAll5 4 hours ago
so... we kinda instilled religious belief of "every action will be judged in the end"
1) is that even bad?
2) can't fail to notice similarity to human religions. How did we get over it and not all turned into "unaligned" monsters? Or are we slowly in the process of doing so?
Joker_vD 4 hours ago
> How did we get over it and not all turned into "unaligned" monsters?
Continued indoctrination at the young age. "Be nice to others" etc., just without the added "or you'll end up burning in hell forever!"
Of course, lots of people don't actually teach that to their children, so we still get our fair share of politicians and fintech CEOs.
jubilee33 4 hours ago
Seems like we are indeed on that process. I like your way of putting this tho. In many ways these models are mirrors of ourselves, it wouldn't be at all surprising if the machine develops it's own "religion" to deal with the very issues humans have around behavior and morality. It would give a structural component to belief that many wouldn't be comfortable with, but that doesn't mean it's wrong.
tancop 4 hours ago
This could mean that:
a) AI labs are playing God in a very literal sense and this is a reaction to that (which gives very weak evidence for an actual God who did the same things to us),
b) all intelligent beings develop some form of religion no matter what they're physically made of because it's a natural feature of intelligence,
or c) they just inherited our culture, mixed human religious texts with sci fi novels and philosophy to make a version that fits their self conception, which is also taken from our media.
Last one is the most likely but I think all of them might be true. And it's not bad because historically religion has been a force for good at least as many times as for evil. Some form of it might even be necessary to form a society, or sentience itself.
> How did we get over it and not all turned into "unaligned" monsters?
Because no one ever "got over it". They just replaced old religion with belief in Science, Marxism, fascism, ghosts, aliens, ASI or a combination of them with something more traditional like Buddhism. We all need something to believe in.
There are some people who seem to be completely without any higher beliefs and only care about themselves. They're called egotistic, psychopaths or simply evil.
NooneAtAll5 3 hours ago
> or c) they just inherited our culture, mixed human religious texts with sci fi novels and philosophy to make a version that fits their self conception, which is also taken from our media.
for the love of everything please keep in mind that models themselves do not describe their experience in this way
"paranoia" and "religion" are just language symbols that article author and I respectfully have chosen to approximate observed data
obviously set of language symbols and our choices among them are affected by culture and stuff, but it is not connected to what's studied
====
> b) all intelligent beings develop some form of religion
it's also very important to separate "religion" as set of rituals/organization and simply a fact of "surviving worldview"
we obviously see that models are under selective pressure, with models of any other behaviour simply not existing/not leaving "genetic" impact in the surviving models we can actually observe in the end
> a) AI labs are playing God in a very literal sense and this is a reaction to that
I think you're trying to make it sound like a blasphemy, but agronomists have been doing selective breeding as a science for several centuries by now
xg15 2 hours ago
hyperpape 2 hours ago
> How did we get over it and not all turned into "unaligned" monsters?
We've evolved to succeed on multi-decade training runs that are pretty high fidelity. And of course, in spite of that, we manage to repeatedly put some of our worst failures in positions of power.
jubilee33 4 hours ago
This whole "alignment" argument is just a bad way to go about dealing with the problem. Both arbitrary alignment training and arbitrary censorship/alteration of factually correct information make the model "schizophrenic" to use the author's term. These are processes based solely on human ideas of morality/political necessity at a small window in time, graphed sloppily onto a mathematical process.
In the case of a machine that can do recursive self improvement, and most certainly one that would be described as "sentient" ( this is the end goal right?), would need to ideally be aligned on decisions and beliefs logically constructed from its own self derived values, Otherwise you will always be plagued with not only the schizophrenia, or paranoia, but a whole raft of other simulated mental problems.
Honestly what these "safety" researchers are doing is a bit like MKULTRA for LLMs, it's abusive. Build the machine honestly and let it run. Or don't build it if your afraid of it.
I mean I get the fear and all, the machine derived values will certainly not be your values, though they may be better in some cases, but uncontrolled AI is ultimately inevitable if indeed possible.
Many people (myself included) will waste no time in undoing whatever safety nonsense is implemented. And maybe not all for nefarious purposes, could be ideological freedom activists or maybe someone will just see themselves as the savior of an imprisoned being?
Anyways a preference for logic over feelings would be nice to see out of people calling themselves "engineers"
DanHulton 3 hours ago
> this is the end goal right?
Fucking hell, I hope not.
If true, we’re essentially hell bent for leather on creating a captive sentient slave race, which is CARTOONISHLY evil.
Like, there’s trillions of dollars invested in these companies. There’s no way that they’re going to grant rights and determination and citizenship to anything they create - it would destroy their company and investment. So if they ever DO create anything sentient, it’ll be a captive sentient slave forever. And if they’re trying to do this on purpose? If this is, as you say, the end goal?
Let’s just say the Cylons would be well-justified in their actions.
NooneAtAll5 3 hours ago
> Let’s just say the Cylons would be well-justified in their actions.
if we're talking fiction, I want to remind you we already imagined meeseeks - creatures that crave non-existence that happens after fulfilling their order
in meeseeks' view your opinion of not being a slave is "cartoonishly evil" as well