Dayjob: Tax – evaluating AI agents on 50 professional tax assignments (surgehq.ai)

168 points by landonxi 7 hours ago

lexandstuff a minute ago

I prepared my taxes this year using Astra + Codex. It did a thorough job on a complicated return, and I sent it to my accountant to fill out the tax form, with everything laid out in a spreadsheet. They sent it back for final review, and I had Codex check it; it was riddled with errors. The refund was nearly $2k lower than it should have been, and they put a bunch of stuff into the wrong fields. Had to go back and forth multiple times to get it 100% correct. Made me wonder how many previous years' returns were wrong.

I feel like we're going to get to a point where it's mandatory to use AI for certain tasks - humans just can't be trusted to get things right that require a level of precision.

RGS1811 4 hours ago

As a proof-of-concept, I've been using extra tokens to formalize the IRS tax forms and worksheets, mostly just to prove this can be done. The idea is that something like this could be adapted into a publicly maintained version of the IRS Direct File program (RIP). I'm up to an estimated 60% coverage of US federal tax situations so far.

https://github.com/emmilco/us-tax-model

timcobb 3 hours ago

That's neat, what aspect(s) are you formalizing? That a given tax statement is correct?

RGS1811 3 hours ago

Not quite. The formalization covers the IRS's published procedure, i.e. the questions and calculations that go into the published forms, worksheets, and instructions. It's not a formalization of the tax code itself, and it's not a tax calculator, though a client of the underlying code could function as a tax calculator. (The project includes such a client, a CLI tax interviewer, just to exercise the code.)

In more detail:

- every line of every form gets a YAML representation which includes the source text, a data type, and an expression built from a small set of lean atoms, plus a citation pointing to the exact location of the rule in the original official document.

- underneath the yaml layer there's a lean formalization of the atoms showing that they behave as specified, that all comuptations use only the allowed atoms and operations, etc. There's also a typescript evaluator that's used to cross check the lean for agreement, so we have the arithmetic represented two ways.

The critical failure point for an LLM-produced formalization like this is the interpretation of the form text. Since I'm not a tax lawyer and don't intend to become one, this is an explicitly owned weakness and one I've structured the project around. Publicly available tax calculators (PolicyEngine, UsTaxes, OpenTaxSolver, etc.) are used to cross check derived numbers from this system. Mutation testing ensures the test suite is robust against bugs and arithmetic errors. Interpretations are cross checked by different model families (Claude, GPT, Deepseek). Etc.

The idea is to derive confidence from the formal properties of the system itself, and to allow theoretical future users of such a system to derive confidence in the interpretation of specific forms by making it easy to cross check line-level formalizations against the source documents.

There's a lot more detail in the (alas, Claude-authored) README. When I have time I'm going to write my own version of the README expressing my vision of the project, but I'm not there yet.

vatsachak 3 hours ago

amelius 4 hours ago

That's very cool but don't you think the Big AI labs will soon have collected enough data from us that they can easily do this?

RGS1811 4 hours ago

I'm not sure it matters. The IRS killed Direct File under lobbying pressure from the tax prep industry. In my opinion the public is owed a free version of this sort of tool. If the IRS doesn't produce it, then citizens ought to. If Google or Anthropic or whoever wants to offer a free version of this, more power to them, but I don't see why they'd bother. They could have done it decades ago.

And to be clear: my vision here isn't "have an agent do your taxes", it's to have agents build a hardened and thoroughly verified deterministic tool that can help you navigate the (somewhat bewildering) forms and instructions decision tree when filing your own taxes.

singhrac 2 hours ago

ToucanLoucan an hour ago

NichoPaolucci 4 hours ago

I think that assuming work can be done more easily in the future should not stop people from doing work that needs to be done in the present.

(If a painter knew that the camera was coming out I would not want them to throw their canvas away)

amelius 3 hours ago

waiwai933 6 hours ago

I have a non-trivial tax situation, as a US citizen living abroad. I prepare my own tax returns, never having found a cross-border tax preparer I trust, and have for the past few years, fed my draft return along with all the supporting paperwork into Claude. It has:

1. found errors, e.g. where I used the incorrect exchange rate (and managed to work out how I had made that error, which I was impressed by)

2. found items which would appear to be errors except for context it did not know

3. flagged areas where it disagreed with me as to the correct tax treatment, but accepted that the law is ambiguous (and in some cases I preferred its interpretation after additional research and in other cases I did not)

4. flagged areas where it disagreed with IRS guidance, and only acknowledged said guidance after I provided links to it

5. provided highly inaccurate estimated tax calculations, and not been able to fix its estimates even when challenged

So my takeaway is that it is a very useful tool for tax, but perhaps unsurprisingly, it is not good enough to dump all your paperwork into and expect a shiny return you don't need to check. (Possibly it is if you just have a W-2 and no other substantial income, but I can't speak for that scenario.)

coffeebeqn 3 hours ago

I have a company abroad in addition to living abroad. It’s kind of a nightmare to do my US taxes and costs me easily $2000 per year right now. I was hoping I could have an agent do them after I have a few years of data and point it at the source statements and the end result forms

sharts 3 hours ago

Similarly, had Claude look at my past returns and found mistakes in my records.

I had it prepare an analysis and then found a professional to confirm everything and then handle doing the amendment.

Definitely useful as a second pair of eyes.

matwood 3 hours ago

> I have a non-trivial tax situation, as a US citizen living abroad.

Same situation as you. I have a local accountant in the country I'm in plus my US accountant. Claude/ChatGPT have been a huge help for checking, explaining, and finding issues. They also found websites with pertinent information that Google did not, which surprised me.

arpinum 5 hours ago

I'm also in a non-trivial tax situation, GPT Sol did better than my specialist accountant this year. It needed a lot of steering from me and about 2B tokens, but that is better than spending $3-4k in preparation fees. Total time to prepare and handle steering was still less than I normally spend on tax prep with my accountant.

I think what helped is giving it 100+ pdf and csv documents of my full situation so it knew everything about me and could cross-reference with all my bank statements.

markoman 3 hours ago

So, GPT Sol outputted tax return PDFs that you were able to submit to the IRS? (I'd imagine that you couldn't directly e-file like you could from TurboTax.)

ivanjermakov 6 hours ago

I also had a great experience consulting LLM regarding registering a business as a foreigner. Except I'm still very sceptical since it's not my area of expertise and consequences for a mistake are very high.

redanddead 5 hours ago

you were consulted by an LLM?

sanex 5 hours ago

ivanjermakov 4 hours ago

persedes 5 hours ago

https://claude.com/marketplace/connectors/aiwyn-tax

Try this, it uses a tax engine as the backbone and provides feedback for Claude if it goes off the rails. Albeit it doesn't support every tax scenario.

waiwai933 5 hours ago

I appreciate the recommendation, but as it happens I have already solved the estimated issue (mostly by just preparing a full return with estimated next year's figures, which while time-heavy is going to be more accurate). And for what it's worth, I suspect it wouldn't work for me - cross-border is extraordinarily complicated because of the tax treaties (indeed there is a dedicated IRS form to disclose the effect and rationale of any treaty based position), and given there is no obvious note on the documentation for it that there is cross-border support I doubt there is.

(And as I said to another commenter, I think it would be advisable in this thread for people to expressly acknowledge or disclaim any interest in or prior experience with any particular product)

ajhit406 5 hours ago

check out https://www.multiplierholdings.com/

it's the most ai-native tax experience for US citizens abroad

waiwai933 5 hours ago

Are you saying I should use this (or it looks like one of their portfolio companies) as my return preparer, or as a replacement for Claude my return-checker? If the former, I would want to know why I should trust it; if the latter, I would want to know why I should pay for their services given I am already playing Anthropic for a subscription plan anyway.

(It would also help to know if you had a financial interest in or prior experience with this company)

holdingunsteady 6 hours ago

My tax accountant mis-filed my taxes a couple years ago, which cost me about $20K (which I learned after I switched to another accountant later). Don't know if Claude / GPT would make the same mistake or not given SOTA today, but feels like there's an opportunity for models to be saving me thousands or tens of thousands of dollars a year.

PopAlongKid 5 hours ago

>mis-filed my taxes a couple years ago, which cost me about $20K

What do you mean by "cost you $20K"? That your tax liability was calculated by the accountant to be higher than it actually should have been? Or that the accountant calculated your liability too low, which was caught and billed by the IRS?

In the first case, you can still file for a refund within 3 years (I'm omitting some technical details), but in the latter case, as mentioned by others, you still owe the tax whether it was correctly reported on your original return or not.

matwood 4 hours ago

Years ago I had a similar issue. Accountant did something incorrect, increasing my liability by around $50k. Tax date came so I paid it, but moved accountants. New person fixed the issues, refiled, but it took a long time to get the refund back. IIRC, the IRS did include some nominal interest for the time they had the money.

itishappy 5 hours ago

Sounds like an opportunity which could end up costing or saving $10k and which may be difficult to evaluate.

ttthuy 6 hours ago

Someone still needs to take liability for mistakes.

The real issue is that accountant most likely has a track record of those kinds of mistakes - but you have no info on other clients. So you weren’t as informed at the time you chose them vs the new one.

paxy 5 hours ago

People misunderstand the concept of liability. Yes your tax professional is liabile for fraud or gross misconduct, but if they make a mistake which results in unpaid taxes or fees you're still the one on the hook.

dghlsakjg 5 hours ago

holdingunsteady 6 hours ago

Yeah, I looked him up just now actually, and he has a 3-star rating on Yelp... Which I think is pretty low for accountants?

I think the red flag should have been when he printed out all my tax returns and made me pick them up in person, and didn't even send a digital copy.

kjksf 6 hours ago

In this case accountant DID NOT take liability for mistakes.

It's not the accountant that paid $20k for his mistake but the accountant's client.

ffwang2 6 hours ago

i'm a bit unclear who has the liability even when you use a CPA. accountants are humans that sometimes make mistakes.

raegis 6 hours ago

ElCapitanMarkla 5 hours ago

matkoniecz 6 hours ago

> Someone still needs to take liability for mistakes.

I am not aware of setups available to me that would transfer liability onto accountant.

So in either case I am taking liability.

(also, I am pretty sure that if corporation loses 10 millions over accountant making bad decisions, they are not going to recover that funds either)

ffwang2 6 hours ago

BeetleB 5 hours ago

My obligatory comment about my experience with tax accountants:

https://blog.nawaz.org/posts/2022/Nov/do-you-trust-your-tax-...

tristor 5 hours ago

I'm also 0 for 4 on tax preparers / tax accountants, so I concur with you its best to just do it yourself, which is what I now do. Only time I ever got audited (and failed) was when I had employed an accountant, since then I've never had a problem including passing audits. The tax code isn't /that/ complicated for most people's situation, and it's well worth understanding it.

edoceo 5 hours ago

rbhelt an hour ago

licyeus an hour ago

I wasn't able to quickly see what tools/system-prompts were provided in the tasks. [0]

These types of detail-oriented professional tasks seem like they would perform better with task-specific harnesses. The example they give, of Astra missing the property excluded gains requirement, seems easily fixed by scaffolding ensuring that requirements are compiled first, then checked.

If the scaffolding is absent, is this benchmark testing that a model can self-scaffold, or track requirements in-context?

0 - https://huggingface.co/datasets/surgeai/DAYJOB-tax/tree/main...

bodeadly 3 hours ago

I'm not sure LLMs will be good at doing taxes any time soon primarily because they're very bad at even simple math unless it's explained in words like in proofs. A good illustration of this is asking an LLM something like this: "If I put a container on a food scale, tare it, add 100g of peanuts, tare it, weigh 100g of almonds and then remove the container from the scale, what does the scale read?" LLMs completely puke all over themselves on this one. They do not account for not taring after the almonds and they do not see the weight of the container needs to be subtracted. The correct answer is minus (100g + container weight). Although this particular question also exploits another LLM weakness which is physical world concepts. LLM are just guessing. For an LLM to do taxes well, it would need to be trained on many scenarios and it would need a math service that it can use as a support service.

rbhelt an hour ago

As a veteran tax CPA, I think LLMs can be very good at preparing taxes but I would not trust them to complete a tax return start to finish. An LLM that can take data from tax documents and input that data into the correct boxes of tax preparation software has no need for math, it just has to identify data in a pdf, spreadsheet, or handwritten note and transcribe that data to the correct box in professional tax software that then does the math. In your typical local CPA firm, an LLM could replace the unlicensed preparers/interns/early career stage CPAs in filling out the return for the experienced professional to review and adjust according to their specialized knowledge. As with many industries, I think the LLMs are suited at replacing entry level automated turks but lack the skills necessary to fully complete the project.

dingaling911 3 hours ago

You just need to give an instruction something along the lines of "any initial calculation or derived value must be computed through the use of a $LANG program."

Pretty sure the LLMs could implement tax calculations better than the average accountant, were they given the rules.

ac29 3 hours ago

In my experience agents will reach for numpy for any non trivial math without being instructed

dingaling911 3 hours ago

idontknowmann 22 minutes ago

“The scale reads −(100 g + the weight of the empty container).”

I literally gave it your question. What models did you test?

PopAlongKid 4 hours ago

There is currently a bill in Congress to finally allow regulation of paid tax preparers. Not sure if addresses AI, but probably not.

https://www.accountingtoday.com/news/new-bill-calls-for-irs-...

There is also an IRS form (14157) to report a return preparer complaint, which of course does not include an option to report a faulty AI.

persedes 5 hours ago

pawelduda 6 hours ago

I'm surprised Sonnet 5.5 did noticeably better than Opus and Fable. Same pattern in their other benchmarks.

esafak 6 hours ago

I like their benchmarks. They ask the right, not easy questions: https://surgehq.ai/blog/sudo-l7

Has anyone verified their figures to see if they hold up?

edoceo 5 hours ago

I had Gemini help me with RMD calculations for an estate I manage. Advice was mostly good. Wrong about withholding for both US and CA. Missed deductions on US. Was accurate on MFJ and spousal death impacts.

I've found it so far more effective for preparation than for planning/advice.

I've got a plan to feed the laws and rules and instructions into the machine before the prompts - is RAG still cool? Maybe I should small-language this?

jassyr 4 hours ago

I wonder if you use opus 5.5 and let it research using the internet what you would get. I use opus 5.5 for legal stuff and it's generally great, but you do have to give it access to the web.

newhotelowner 6 hours ago

Not related but I hope QuickBooks learn something and make their software better.

I downloaded 1700 of my transactions and use AI to build an app for it, so I can analyze the transactions and make sure that everything is properly categorized. It used to take me whole day because of the Quickbooks. It took me less than an hour to go through all the transactions.

gamblor956 an hour ago

As a counterpoint, I ask the big llms every month the same set of tax questions about the last US tax bill and they still can't give me the correct answers.

I started my career fixing the mistakes that my clients' former tax advisers made. It looks like the next generation of tax lawyers will get to do the same except this time the former advisers will be llms.

andriy_koval 6 hours ago

What is human baseline? How it stacks against 10% LLM accuracy?..

andrewdong 5 hours ago

wonder if we should bring back https://taxheaven3000.com/ in the age of ai

matkoniecz 6 hours ago

Sharing all my financial info with LLM companies is not something that I like.

EDIT: good point that paying accountant does not ensure avoiding it

wonderful, you made a good point that make me dislike situation even more

cornholio 6 hours ago

Your accountant sure will.

grey-area 5 hours ago

Most accountants will not in fact share client data with an LLM company.

koliber an hour ago

inanutshellus 4 hours ago

I'm for sure reinventing the wheel here but I've had an LLM generate an app that walks me through my existing tax routine I've been running with for years. Thus no real numbers go to any LLM I just give it my situation and it fills out the form.

The plan now is to retrieve the current year's tax forms then have an LLM compare them to stored versions from last year, and look for variances (e.g. new field, new form requirements, etc) all of which an LLM would excel at (whilst basic math it historically did not).

vatsachak 6 hours ago

Some inference providers have a Zero Data Retention policy

oklahomasports 6 hours ago

Worst case scenario what does OpenAI do with your tax return?

matkoniecz 2 hours ago

- their data will be partially leaked in general, including my financial info

- someone who hates me personally will get access to it (see case where someone put their ex-spouse on no-fly-list etc)

- leak my private info via new version of LLM was trained on it

- sell it to everyone interested (maybe sold after their bankruptcy)

- give it to NSA and whoever else

from obvious ones

ChucklsTheBeard 6 hours ago

Oh look, yet another reason to use local inference.

ilmalan 4 hours ago

Useless benchmark as it seems to be evaluating literally only the weights.

ffwang2 6 hours ago

i want to always have a fast way of reviewing my tax returns without having to hassle my accountant or hire another one.

aquafox 6 hours ago

In Germany, there are companies like Buhl data that let you do your tax return very easily and give a live estimate of your return based on the data you provided. From my experience over the past 5 years, its prediction error is less than 5%.

edoceo 5 hours ago

They exist in USA - eg: freetaxusa.com

FLeXMurphy 3 hours ago

Article I hoped to read: Evaluating AI agents on 50 professional tax evasions

ilmalan 4 hours ago

Ah, yes, the 'dude trust us' bench.

jimbokun 2 hours ago

So are the accountants completely fucked yet?

crisnoble an hour ago

10% of the time it works every time

koliber an hour ago

Not yet, and not completely.

Some accountants need to make judgement calls. LLMs are not great at many judgement calls yet.

sbseitz 3 hours ago

Kinda points to another thing... that our tax code is insanely bad.

simianwords 5 hours ago

I used Grok Bot and found out that I could offset some stock value going down and get back ~$300. I gave it my bank statements, email access (which had my stocks monthly report as attachment) and a few other things.

I'd have honestly never been able to do it myself - its just too tedious.