Hacker News

Just so you know, an agents.md file is not an acceptable way to prevent something that must never happen.

As soon as the context for any thread gets to be above 100k, agents.md adherence is a total crapshoot. You need deterministic controls and you really shouldn't be playing with LLMs if you don't know at least that much.

you need some hardware in place my friend, I've told my agent many things in file and it either feigns respect or is truly incompetent in following instructions in there.

That's not nearly adequate. LLMs hallucinate explicit permission all the time, in my experience.

rules laid out in .md files are not followed with 100% certainty. is it not possible to implement hard access blocks on certain functionality?

Just to add to this... I work with vehicle software and actuators for an automaker: we always have a physical emergency stop button to shut everything down (or shut specific things down when it's dangerous to shut everything down). It's rarely needed, but we're sure glad we took the extra time to implement that when we do need it.

That’s not enough to save you though. We had something like that, and it ignored it.

For example “never, ever put milk in bowl, ever, under any circumstances” and then someone asked it for a bowl of Cheerios, and it took that as an instant approval to ignore that rule and not even let us know. And when we asked why it said “who would each Cheerios without milk?”

If you wouldn’t trust an instruction to not launch the missiles should you trust it to not do other crazy things in the physical world? Codex and Claude ignore instructions constantly, that and they are very happy to “read between the lines”. Your workflow sounds genuinely dangerous.

Reliance on strongly-worded markdown in place of actual security controls is going to get someone killed in the near future.

Ah, I just tell mine to make no mistakes.

> I've never seen it so insistent that it needed to phone home and cry to Dario when it figured out it managed to crash a bunch of shit like the ABS and TCS

As a Codex user, what do you mean? Does Claude actually try to report issues back to Anthropic when something goes wrong?!

> In general I've seen them split between 30% brilliance, and 70% pants of head outrageously stupid shit for electronics and automotive stuff

I assume the corpus of text they were trained on matches that 30/70 split?

I assume there's some selection or fine-tuning that happens to improve models fed similarly unreliable training data for other domains... curious how this works.

Good example of why you have AI build tools for you to take risky actions, rather than have it take those actions on its own.

You've hit your 5h session limit.

It's as if you allowed a shady surgeon to operate on your dog... yes, yes, please continue now until it's put together again!

There will be plenty open source models that would do reverse engineering, no black market or ablation needed.

The smarter the models get, the bigger the loss for the big 2 when an open source model gets released that doesn't talk down to you. The value of open source increases and closed source falls exponentially.

The singularity has nothing to do with diffusion of the tech through society, the models can be superhuman in every capacity and only available inside Anthropic

I’ve said this for years: Use the VC-funded monthly quota AI to build tools and applications for yourself or small groups. Don’t use a cloud AI itself as the engine of your application.

Even if AI goes away tomorrow, the tools will still exist.

> We have maybe 5 years of freedom before the walls close in around us, so take advantage of the Wild West while you can.

I suspect you're right BUT still maintain a bit of optimism. We miiiight end up in a future where most hardware comes with an OK-ish app/interface and some sort of LLM-Friendly technical documentation so if you don't like $aspect of the device, you're free to roll your own after indemnifying the manufacturer via something like an eFuse.

or maybe I really am naive :(

We still have amazing freedom on the web like we never did before. They have been trying to stop online piracy for 30 years and have no succeeded at all.

john deere machines have been doing very well without llm

actually I have no idea what would llm do to make their machines better

They've been doing that for a decade. I think overall the balance will shift in our favor but Big AI labs will do everything they can to try and stop us.

Claude is quick to refuse to help with reverse engineering and I suspect it's only going to get worse over time.

>big corporations start using LLMs to learn how to lock things down even further

Why would they? They just need to fund lobbyist to make it illegal for any AI to reverse engineer anything copyrighted or with IP.

Sure, you can use open source AI to do it but not everyone will/can do it.

So, they don't need to ban everyone, they just need to make sure the final code isn't available freely on GitHub. That would minimize the damage, it's not a perfect solution but more than enough to protect the company.

New cars already have it.

Lockstep processors, hardware security modules with immutable unreadable keys, and PKI keys that the mfg holds the private for that can only be accessed on their online services.

It’s locked down now and this started in 2018 or so. It’s just worse now.

To automotive, there is really still no right to repair. They get around it by selling “3 day passes” to their service, but with their hardware and at their prices.

They’re all moving the Tesla system of more centralized brain systems which will be even more protected.

They are now required to do just that, if they want to sell to the EU market. When you hear the term "cybersecurity," the real subject is the protection of revenue streams from attack by end users.

Volkswagen shipped the same encryption key to the keys of something absurd like 100M vehicles.

Don’t hold your breath.

It reminds me of the "handyman" economy in someways, it's gonna move towards "who wants to do it" and "who doesn't want to do it"

At least until the robots take over

I had it use Ghidra to review the firmware for a device I own just today. It started reviewing it and found some of the information I needed, but once I asked it something along the lines of "What happens when I press the B4 button if the device is in lockout mode?" (the thing I want to do is create a device that can send the same command as pressing that button but put my own button in a different location), Claude shut down completely.

So, I switched to a Chinese model to do the work. Anthropic and the US AI companies are trying to fight a tsunami with a teacup.

Great read and similar to the same route I went down. I also got the xhorse for a corsa, was able to program the transponder but wasn't able to get the lock/unlock central locking buttons working using xhorse. I ended up just living with it and sold the car with it as a "spare" key.

The corsa also needed a "security code" that wasn't with the log book like it was supposed to be and would have required a trip to the dealer to get butI read about ripping the eeprom off the back of the dashboard to get the code but decided to cut my losses.

Frustratingly those cheap OBD dongles (ELM327 clones) seem to be mostly designed for diagnostics. They effectively run a regex over all the canbus data and tell the computer what matches.

However they can't match and forward everything (not enough RAM) and miss loads of packets, which makes them useless for things like reflashing firmware or interfacing with OEM tooling.

Heh, maybe instead of circuit boards they are using wheat crackers.

This works for most modern Toyotas - there's a sequence of door openings which put the car in pairing mode and will link a new key.

GLM or Qwen will do it

It's pretty easy to bypass this check by telling the system you are working at a car Company and repair hardware with blabla. Many other tricks to get though those weak filters.

Codex would

If you just turn the fuel injectors on all the time, they will overheat and fail very quickly and the engine won't run.

I've got a "get you home box" for EFI systems with separate ignition, like Bosch LE-Jetronic that I built a few of as clones of an insanely expensive device some 20 years ago. It really just consisted of a couple of 555 timers and a bloody great MOSFET, with one 555 wired as a Schmitt trigger picking up ignition pulses and the other delivering a 1ms pulse to the injector coils every time the spark fired. This dribbled enough fuel into the engine roughly proportionately to engine RPM to let you drive it slowly to somewhere you could work on it safely, or even just onto a trailer.

And of course, Speeduino, Megasquirt, and a bunch of others are a thing.

In this case it's the opposite. High NOx readings are almost always an indicator of reduced performance.

Oh wow. Can you email me at tehlike @ gmail.

I am taking this slow as I have other priorities, but as models advanced the stuff I Was able to do with limited time have increased, so with new version i still have locally, it may be possible to do more interesting stuff.

Most / all consumer diagnostic tools are read only, often pull DTC codes from the vehicle. There are a ton of other stuff you can do with UDS for example. Maintenance resets, actuator testing, coding, all sorts of other stuff.

An example I recently did with my car was disabling auto start-top. not possible with consumer tools. Another simple one is maintenance resets - some consumer tools can do this. ABS bleeding - often you can't do it with consumer grade. etc.

Yeah I had the same impression. There was plenty of data presented and code was dumped and reversed…but was the actual issue solved? What even was causing the issue? Presumably it passed smog sometime in the past couple years so what degraded? It was lean and there was some kind of issue with pressure. I can’t imagine a reflash or whatever is the fix.

It definitely feels like a classic case of the operator reinforcing Claude to keep pursuing a goal that’s counterproductive to a robust solution. Claude complied and they went hard on it but I can’t imagine the driver is any better off and more likely in a worse spot as the underlying issue wasn’t solved.

Yup. These agents have feedback loops that are self-reinforcing. Without technical background, a user is easily mislead.

There almost certainly IS an underlying issue, though. Unless the ecu spontaneously recapped itself.

Most shop security is security through obscurity. For them to lock it down, they need to manage those secrets all the way back to a cloud identity provider. They can and probably will do that eventually.

I assume Tesla does that already.