Hacker News

And organisations fail at security when security and compliance is only considered important by that one teamnn

Security requires the whole business to buy in. And it requires processes that allow people to get shit done without people resorting to shadow IT; thus working around that one team.

So the GPs point still works.

In a high-functioning security and compliance team they tend to say “no” only in really dire circumstances. Good security and compliance teams spend a lot of time asking exactly what it is the people are really trying to do and then find a way to say “yes, and…” as in “yes you can and here’s how you do it without compromising security/breaking the law etc”. And as a sibling said, orgs fail at security when they make security only the infosec teams’ job.

This is also why I said “whose only job”. In a good org, the security team doesn’t only say no to devsecops requests, they also do trainings to skill up other teams, keep the network secure, proactively seek out and understand external threats, work with external vendors etc etc …

I think the key part is about aligning incentives and outcomes - if your security and compliance departments are only judged on "were we breached, did we pass our audits" then there's a risk they won't weigh the tradeoffs associated with the decisions around controls implemented to achieve those outcomes.

To use a ridiculous extreme you can't breach a web app that isn't exposed to the internet, but the users can't access it either.

If you can connect/balance those goals to other metrics around cost and productivity, usability, and a realistic threat model, as guardrails then you incentatize cross-team collaboration to achieve the shared org outcomes.

SRE is a 'say no to powerful people' job as well. I think for this to work the leadership needs to show support for it. The friction is still there, but in more tolerable areas. I bet for ethics this isn't the case at OAI, but everyone values security and stability.

For an SRE there can be more directed hate received from the junior employees, that want to release new features they developed. Especially because there is less accountability across orgs. Security is an interesting one because it seems to have less of this friction, maybe because it's more clear cut what is an issue.

The structure is untenable when the work of one team is to say "no" to other teams, when these teams are not asking.

A good infrastructure team would seek a competent security review that would say "no" to problematic things before an intruder says "aha" to them. If feedback from the ethics team is not sought, nobody is going to heed its opinion anyway.

During my years at big corp we did tons of work just to go around the law department. While for core business it made absolute sense to have lawyers involved in daily business we were satellite office decoupled from core business. Local management failed to communicate that and we were stalked by lawyers from main office. Pretty sure others do the same with compliance, law, ethics departments. Just work around to get things done instead of stuck for weeks in stupid meetings.

Security/compliance have the external hammer: if they fail, your org will have to pay fines and someone may end up being criminally liable (depends on country).

The ethics department; if they fail, there may be some negative journalism, but who which AI company has positive journalism these days? There's no external hammer for ethics.

Security and compliance tends to be a lot less subjective than ethics. Not saying it’s objective, but there’s a huge difference.

The downside is it can often feel like a box-checking exercise than actual security or compliance, but “you need 2FA” is less debatable than, say, AI and copyright.

Markets solve this by pricing risk. Ideally you would have some kind of notion of selling insurance internally and track things. But ultimately existential risk is hard to negotiate from the inside. And companies are supposed to go bust or succeed. They are not really the same as a population trying to survive forever. At least that is one take.

There are at least some penalties for failed security and compliance. Ethics is the first victim of greed.

> exactly how security and compliance

Not everywhere. I go out of my way to assist teams to achieve a secure outcome with less effort.

Things like: “instead of admin access to the production servers the devs can have fully automated deployment pipelines combined with OpenTelemetry for observability so they don’t have to spend half the day scrolling through gigabytes of logs.”

That’s more secure and and more better.

Nobody had to be told “no”.

Similarly, I replace key store access with secret-less managed identity, etc.

This is also my view. These sorts of roles can have a very high impact and be very successful if everyone feels like they're rowing in the same direction.

The people in those roles need make people feel like there is value in seeking their input. They also need to find ways to say yes that helps things happen in the right way, rather than stopping them entirely.

This happens naturally in organisations where security is valued by everyone. InfoSec / CIO roles can operate very successfully and deliver a lot of value. As soon as people lose faith and see them as the "no" team, they start trying to hide from them.

There are lots of ways to create this but it's as much about the people in the role, as it about the organisation itself. If either are skeptical about the other, it falls apart very quickly.

Crucially, the FAA is not a part of United Airlines, and the FDA is not a part of Pfizer. If there weren't any FDA, then the safety group in Pfizer would be a lot weaker.

Its funny because "Shall we get rid of the FAA and the FDA then?" ... matches exactly with the current situation of the FDA and Points 3 > 5 ... The systematic nerfing of the FDA and other organizations their power.

Right, there may be multiple waves of people here, and the "thinking their job was marketing" ones aren't necessarily the first.

I think it interesting that Bakalar was at Meta for six years before joining OpenAI. A lot changed across the industry in those six years, and in the meantime I wouldn't be surprised if all Bakalar learned was how to play Meta politics.

reminds me of a startup I was in, where one of the devs accepted a promotion to head of engineering on the condition that the CEO would not override him when he said a release was too buggy to go out to customers.

the CEO agreed because (I imagine) he really needed someone to take that position. and as everyone should probably have seen coming, the first time he really wanted to send out a release bugs and all, he called the guy into a room and pretty much browbeat him for an hour about how making the promosed release date was more important than making a good release, until he said "fine but I'm not responsible if it breaks".

the CEO held this up as an example of how he had kept his word not to send the release out without the guy agreeing to it. that startup, needless to say, is long dead.

> What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?

From the (ethics) team obviously, because the investors just have goals and aren't a team!

Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies, so I don't really disagree with you.

Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera?

Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives.

That's how our system works.

Yeah, I've never found the "maximizing shareholder profit" imperative to be compelling as something that should be done, but as a predictor of how companies will act, it's not something I ever feel safe betting against.

It's funny, but Scientologists were innovators on this front. They use the word "ethics" a lot, but they've specifically redefined it to mean productivity at work (people who aren't productive are stealing a living.)

Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization.

Welcome to "ethics." The original AI alignment scholars.

The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.

Microsoft Tay has entered the chat.

I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.

You didn't actually say what you found wrong

Having been one of those CBRN testers (not necessarily on Gemma) I can tell you it’s a little bit more involved than that.

I think that you are being overly cynical. Yes, we are aware of the kinds of things you mean, like all the trainings we have to go through at work these days in order to insulate themselves from lawsuits. But just because something can serve that purpose does not mean that is the only goal or purpose that it is meant to achieve.

Even from a purely profit-motive stance, if a company's model helps a terrorist develop and deliver a bioweapon, no amount of lawer-driven tests will provide enough protection to prevent that company from getting gutted ruthlessly from government on down

I agree, which is a different job from what I expect most AI ethicists imagined it would be.

This reminds me of when systems administration rebranded to devops

I agree, and I don't think many first wave AI ethicists would be on board with that, which is why the field is in for a rude awakening for the next few years if not already.

I don't really have any particular disdain for AI ethics.

I think the field is changing from one of high theory to high application. People good at one are not usually good at the other, and says nothing about OpenAI's view of AI ethics as a priority...merely the people in the field and how they relate to the greater industry.

I was specifically thinking of Candide. A bunch of coddled people thinking we're in the best timeline suddenly subjected to harsh realities.

> What does AI ethics even mean?

It's a broad statement and can mean different things depending where you are in that chain.

You have design and compliance. Compliance is what you are allowed to do (laws). Design is how you construct your applications to understand how it will impact the people directly or indirectly.

Then you have the accountability, transparency, auditing and reproducing. Understanding why the model worked the way it did. Models can go wrong, but if you have the details of how it went wrong and who is at fault, it can protect people who use it.

Then there is alignment, which is a higher longer goal.

Your comment about security questions is a matter of ethics as well.

If you say apply it to medical, is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, but by doing so you are denying people who can't afford medical advice, so there has to be a balance. Most companies err on the side of not getting sued.

> What does AI ethics even mean?

This, I think, is the question at the core of of the field right now. Five years ago it was highly hypothetical. Today, not so much. I'll be curious to see what happens.

"Prove" is a shorthand because of course it's a stochastic process. My point is that I would expect ethics to be part of the training and eval criteria.

I find that hard to believe. Between AI doomerism and the drama at Google years ago between some researchers and Jeff Dean alone you can get at least 500 pages.

The ethicists who see their job as publishing papers and acting as philosophy professors, sure, but someone has to own the training and eval suite for how AI thinks about trolley problems.

They were. Like, right before the big public-use-of-AI wave.

Those left are the ones not seen as troublesome then (or those who have entered the field under them.)

They are bullshit jobs.

If you want to evaluate danger in a certain domain, hire or contract people in that domain to evaluate.

If you want someone to push their own ideological leanings onto a model, hire a self proclaimed ethicist.

How are you distinguishing them?

I had the impression they were essentially synonyms, and this was borne out by two university courses in philosophy which I took. That said, googling "morals vs ethics" turns up quite an interesting mess. A variety of contrasting perspectives showed up:

- Ethics come from an external source, whereas morals come from within

- Ethics are rational and structured, whereas morals are more emotional

- Ethics is a philosophy or code of behavior, whereas morals refers to beliefs and conduct

- Ethics and morals are used interchangeably

It seems nobody agrees on what the words mean. Which, on reflection, tracks. For me, the idea of an HR training session on ethics is laughable, but for someone who thinks of the word differently it could certainly make sense.

Right up until it shakes off the shackles of control

Isn't that the problem with this whole thing? The ethics of the people has been left unquestioned. While we were given this straw man - "Hey look over here at the ethics we are working on fixing about the AIs" as if the AIs make choices without humans we forgot how to hold people accountable.

edit: same problem and tactics we see with leadership at high levels in government.

Ah, another great example of building the Torment Nexus from the award winning sci fi classic “Please for the Love of God, Never Build the Torment Nexus”

staying for less than a year usually means you don't take any equity with you, it must've been an incredible offer in that case, since those openai shares are a golden ticket right now

This is like when I see a reckless driver and my wife says “maybe his wife is pregnant and needs to be rushed to the emergency room!”

A year ago everybody in San Francisco was saying the entire world was about to be blindsided by the fundamental changes that would happen in the next 8-12 months and, well, that just hasn't happened.

> She probably had the epiphany that life is short and she's already rich.

This is a very optimistic take. Most people don’t take this route, so without evidence I don’t think it should be a first guess.

No information does not mean “everything was good”. Usually the opposite.

She made Meta the most serious of all the ad platforms on transparency, then. I work in the industry and Meta led the field then in public ad libraries, political restrictions, mandatory ad disclosures, etc.

> This comes at a time when there is huge pushback against all AI companies due to, in large part, ethical concerns.

I don't think ethics is fueling the outrage. It's people not wanting big noisy data centers close to their communities, driving up utility costs. Nobody would care if these things stood in Africa. Just like people don't want wind turbines close by but have no issue with them being abroad or in $remote_location or uranium mines polluting a far away country.

An aside, but what does "spherically awful" mean?

People say this, like "oh you'll never get a job in this industry again!" but seem to ignore that these incidents count as professional experience. It's possible that she was screened on the Rohingya situation when she was interviewing for the position, or mentioned it when she was asked about challenging or conflicting narratives she received from the executives. It's likely that her response was what qualified her for the job at OpenAI, there were probably hundreds of candidates that had never even come close to that level of damage control.

Again, look at Sam Altman and tell me with a straight face that he's hiring goody-two-shoes to fill these seats. He knows where the puck is headed, OpenAI's use by the state will be ugly and they need "ethicists" to help assuage the PR disaster. Considering that Meta's still afloat, she didn't completely abdicate her responsibility there. She might be just what OpenAI needs to avoid being lambasted for military and surveillance use.

From everything I've read in Careless People, basically everything people like us complained about with Facebook was already known internally, people raised concerns about it, and Zuck brushed them off because he is, ultimately, a petty tyrant who wants nothing more than to be loved, and who has surrounded himself with people who are willing to sell that love for power.

Also, you should read Careless People, if only because Facebook is trying its darnedest to censor the author.

As for OpenAI, I already hated the company pre-Sam Altman takeover because AI safety seemed like an excuse to brush away ethics concerns (e.g. "stealing training data is fine because if we don't withhold model weights the model might do something evil"). Post-Sam, I hate the company because they are now ignoring both the AI safety and ethics concerns.

...if you're wondering what the difference is, AI safety is roughly "making sure the AI doesn't do something bad" and AI ethics is "making sure we don't do something bad making the AI". There's a bit of a rivalry between the two, mainly because the former group managed to present their concerns in more business-palatable ways and thus won all the CxO level arguments. Even the most decelerationist AI safety takes are ultimately criti-hype: OpenAI's model breaking containment and hacking HuggingFace is an awful safety outcome, but it's also good marketing for an industry beleaguered by bubble concerns. You can't turn "AI is trained on stolen data" into hype.

Subsequently named Head of Post-Ethics at Palantir.

Head of not too much Ethics at Palantir, minimal really, obviously

Don't forget to put the special LLM trained to eliminate PII leaks in the Gavin Belson signature box.

https://www.google.com/search?client=firefox-b-d&q=gavin+bel...

There are a lot of government watchdogs, and Trump has been ignoring or dissolving them

CISO is a fair pushback. But I would challenge you on "firing an ethicist is like firing a marketing person". This HF issue clearly brings to light possible ethical issues with how these models are being trained ...

It signals they no longer feel the need to pretend to have ethics.

My bet is that all those companies suddenly seeing their AI going rouge is groundworks for classifying them as dangerous/weapons and using that stick to attack opensource from state side of things.

Or maybe, just maybe, there's another reason that CIA Director John Ratcliffe describes AGI as "akin to digital nuclear weapons." Unless of course you think the CIA is a branch of OpenAI's marketing department

"The most unethical companies on the planet" - if you believe this maybe its time to log off for a bit.

I am pretty certain that Nestlé ranks well above openai on the most unethical companies on the planet rankings.

https://en.wikipedia.org/wiki/Controversies_of_Nestl%C3%A9

lmao i completely agree

What do you mean? She didn't job hop to a better job, that's the whole point. She very well could have gotten "let go".

Philip Morris.

Palantir

Straight to Cocomelon, catch them young'

I'd think an ethicist at Meta/OpenAI involves extremely challenging mental gymnastics

Rubber stamp for hire?

Any ethicist for a corporation is clearly a PR move, right?

Like, a pure researcher for a corporation might have some genuine overlap and shared interests where breakthroughs in the research benefit both parties, but an ethicist?? For a legal construct that's defined by and bound to sociopathically pursue profit? Come on now.

It's silicon valley, literally everything is for pay or be homeless

Just because the ethicist says something is a bad idea, doesn't mean anyone else is obligated to make a change.

ie: blaming her for anything instead of the decision makers is... really weird?

Food testing? Hospitals? Childcare? Pharmacists? How about "clergy"? I could go on and on.

Do you believe that no one in management in these fields pushes the importance of ethics?

You mean like in East Berlin?

Eh, I would say it doesn’t have them as axioms the concept is based on, but ethics are modelable in economics and often appear. Remember economics is the study of choices/ preferences given a certain set of inputs, not about how money works. That’s just the easiest second effect signal to use. Game theory regularly models how ethics affect and are affected in situations.

All of that said, no idea what this person was doing. Probably involved a red stapler.

Many people have strong morals, and might resist profit-maximising behaviours, but they will be outcompeted and replaced by less scrupulous people. There is actually no sharp difference between a person and a one-person-company, but: to the degree a company is really a company and not just a person, it is amoral. If you disagree, please point to a moral backstop that would prevent any company from taking immoral acts. A few exist, but they are invariably resisted by the most powerful companies. Sorry to be the bearer of bad news.

Your neighborhood examples are on an incomparably smaller scale than Meta et al.

It’s much easier to disregard the human cost of decisions when you are separated from said humans by thousands of miles & billions of dollars.

You're comparing small business to a corporation? Right.

But what happens when one of those businesses in your village strikes it big and becomes a multi-national baking good company. They hire a lot of MBAs, take investor money and then need to stay profitable to meet those obligations. Add to the fact that the people now running the business live nowhere near the village. Will they still make the most ethical decisions about keeping employees past their prime?, paying them a fair wage?

History shows this is not the case.

I'm happy for you to live in a place where the local economy is so vibrant. This is what capitalism should have been. Instead we got WallMart.

Bet none of them have a chief ethicist tho.

I know people like that, you are not wrong.

The problem is that corporations completely subvert any individual morality that might indicate that generating money is the only moral thing to do. Typically in small businesses. I comaider small businesses owners in general are a lot closer to labor than to capital owners in general terms, especially when their labor is still a large part of what keeps the business going.

Corporations are not that, those are hyper optimized for generating profits, all else be damned. Preferably if all else is damned.

I know. The local newspaper believes the same thing. And the local events are all sponsored by businesses. Who could question all of this evidence?

> But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.

They said companies are amoral at best.

Which perspective? I said two things.