Hacker News

This is long and rambly - let's email (see profile) if you want to work through it together.

You're effectively trying to understand how motor inputs change the end hand position, and in particular, you want to know where the table top is so you can position the hand close to it to pick up/ put down.

This means you have some tuning to "learn" before you can apply a control policy / algorithm - and you should be careful how you phrase this so claude/ai can pick up the right vocabulary and bias towards good solutions.

Adding multiple views helps as follows:

0. Measure from multiple views the table top - Keep cameras stead and rigid, and ask claude to use opencv to do multi-view registration so the plane of the table is known precisely. Keep the cameras steady throughout this process - if they wiggle, you can do multi-view registration each measurement...

Paint the "finger tips" bright orange. Not kidding. Use a very flat chess board under the arm for your "workspace". Also not kidding.

1. Move arm to known position, the multiple cameras will measure the april tags movement AND THE FINGER TIP LOCATIONS. The chess board provides a very nice texture. Or a big flat texture of any kind helps here. Since we know the cameras and table positions, we're getting closer to knowing how the arm movements move the hand w.r.t. the table.

2. If your arm has encoders, then manually touch the table at several points, and claude will record the april tags + joint angles + finger locations.

2b. If your arm does not have encoders, then manually touch the table at several points, and record the april tags + finger locations only, but SPECIFY that the arm is now touching the table. --> Claude can use more opencv code to actually locate the touch point on the table. The measurement is noisy, but you will use many movements to figure it out.

Repeat many times, 10-20, multiple touch points. Then, ask claude to do an error analysis and suggest more touch points. Tell it to use system identification techniques / camera/arm calibration techniques. tell it to research these techniques and report errors.

When you're done, you have to do repeatability experiments - the key is this is now automatic. Claude picks joint angles for the arm, the fingers move, the april tags move, the multiple camers calibrate and record positions, and claude repeats. The manual steps are just the bootstrap - you should be automated now.

What you want is claude to output and ORDF of the whole system - bang you can now control the arm using off the shelf software, which claude is happy to set up for you.

When it comes time to pick things up, the multiple views will locate the object precisely and a control network or algorithm can plug in to control it via the ORDF.

The only problem with this approach for learning is that LLMs are meant to be something that you give goals, and it solves the problem. There's no space for learning there. The challenges become stating the goal precisely, and working around the stupidity and idiosyncrasies of the LLM.

Setting tiny goals, getting to them yourself, then setting a slightly larger goal, that's much more intense.

What I mean is that I'm not sure what "success" means in this context. There are already programs that control robot arms. I would think that success would mean that you managed to write a program that would control a robot arm.

You're an experienced programmer although maybe not a physics or mechanical person. Executing this would mean that you learned the mechanics (most people trying this don't have your programming experience, and have to try to do both things at once!)

Learning the mechanics would mean that you would have a good instinct to critique the AI when success in future projects would mean manifesting something you'd never seen before (and being happy you had AI to help you punch above your weight.) This would also give you a good foothold to understand more complex movements and coordination.

At least that's my perspective. The deliverable isn't on the table, it's in your brain. Asking the LLM to do it is like asking the LLM to copy a famous painting. You copy a famous painting so you can learn the movements of the person who painted it, not for the painting itself.