LLM Agents Play Pokemon Showdown (olivergrabner.com)
13 points by grabnero an hour ago
jfreds 2 minutes ago
Did you try out different reasoning levels? Wondering if messaging is basically a proxy for additional reasoning
minimaxir 10 minutes ago
As a tangent, there has recently been drama with AI and Pokemon Showdown because someone trained a bot (using standard reinforcement learning + a rewrite of the Pokemon battle engine to Rust for max self-play parallelism) that is apparently good enough to get to the top of the leaderboard.
grabnero 6 minutes ago
Yeah I saw that too which was partly the inspo for this experiment. I feel like its kind of getting to a place like in chess where AI is getting better and while theres alot of drama right now, I feel like it'll be something like stockfish in the future
pgn 42 minutes ago
what was the selection for the teams? random per round? how does it play against people in ladder?
grabnero 11 minutes ago
Yeah! So I did teams from a fixed, random seeded pool and then would rerun it with the switched teams. It also doesn't play against ladder yet but I played against it and it's not horrible. It plays better than an amateur but I was able to beat it a good amount
doctorpangloss 32 minutes ago
Pokémon Showdown isn't a game so much as it is competitive Wiki reading.
grabnero 16 minutes ago
Not going to lie I agree lol but still interesting to see
deadbabe an hour ago
Jev will give better results on this kind of thing.
nico 39 minutes ago
For sure are many better models for this use case[0]
At the same time, it’s interesting seeing how generally capable language models/LLMs/agents/harnesses are getting
We’ll probably see a lot more experiments and benchmarks like these in the near future
0: simple classifiers can play quite a few game, including Doom https://playground.jeffyclassify.com/#doom
grabnero 17 minutes ago
Yeah Jev can play it but Jev cant communicate with its teammate in the 2v2 + it cant reason and is just a decision model