Hacker News

Agreed. However:

That's a factor of 3 to 5 improvement reported on here. If you keep stacking these kinds of improvements, you end up at month to minutes pretty quickly.

I'm wondering if the bottom is going to fall out of frontier models when people start to realize this. Sure, as an experienced programmer you can tell the difference between a compact free model and the ones that are 100x bigger and cost billions to train, but 90% of busy work is dead simple: turn a PDF into json fields, or do the inverse and fill out a form. As the tools for this conversion become more widespread you can also imagine an economic shift toward efficient information brokers that make frontier models less necessary.

It's also not really clear if everyone is going to want a frontier model when the real implications sink in. Maybe we'll get sick of incomprehensible code optimizations and people wile tire of reading AI prose that feels ever-more-human. There might be a few use cases, but who is going to pay for this when providers start charging enough to be profitable.

This is really where Gemini still excels. Honestly from my experience the others aren't as good.

Are they? I've had excellent success. The confusing part of this is that there are two types of PDF. The first is a "normal" digital PDF. The second is a scanned PDF. The first can essentially be read like a document. LLMs have no issues with this. It's the second kind of PDF where the constraint becomes the vision capability, and this is very impressive with Astra. I've had no issues with either. I imagine there could be issues with unusually dense and/or misaligned text on scanned PDFs, but I have not tested this.

The bottom line, though, is that PDF OCR is usually regarded as a solved problem. LLMs won't usually do the recognition itself. It will farm it out to established tools which are very good.

Which models are you using.

Yes, remember that these are effectively random PDFs in various different designs and formats, some of them not editable or even OCR'd.

It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality).

Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour.

Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.

You don’t need a trained attorney to schematize data. The LLMs are used to make the data easier to understand and manipulate.

Reviewing something takes less time than producing it.

>Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.

I used LLMs to replace a lawyer this year to great effect. I successfully advised myself as to the right strategy and drafted an immigration petition in the correct language. Granted it wasn't 100% LLMs (I also supplemented it with "traditional" online research and wrote the letter myself using the LLM version as rough guidance), but still, I certainly would've gone to a real lawyer in 2024 for this stuff but didn't.

Professional advisory work can be divided into two types:

1. Once you know the rules of the game, the next move is immediately obvious.

2. Knowing the rules is not enough to know the next move -- that requires judgement and experience.

LLMs in the hands of someone smart and savvy can easily replace the first type of work.

> Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.

In my experience, it is incredibly hard, practically impossible, to find a lawyer who will even talk to you. I think this is a valid use case to get at least some understanding what the next steps might be and if it even makes sense to seek legal representation.

Might be true today, but I still feel pareto principle applies. These points also sound like gatekeeping a bit and I hope future AI versions prove that law is also essentially something that can be made as programmable as possible for most common answers if the underlying principles and constitution are organised such.

If there is no precedence for X happening when Y , Z, P, Q, R are special combinations , the judge decides and sets precedence. But if another judge interprets it differently with similar set of circumstances it is not considered incorrect I presume especially if in different jurisdiction unless overridden by a higher court. This is my layman understanding but I think this design is wrong and essentially where human subjectivity controls the system.

Subjectivity is not totally avoidable in law but the goal of the legal system should be reduce it as much as possible along with ambiguity.

> There is a reason why interns are made to do grunt work. The "chain of command" is built so they learn on basic stuff, learn to make decisions, etc. Without these basic decisions under their belt, an intern can't take bigger decisions later on. They just won't know....

That might be true, but that doesn't mean you benefit from training up juniors.

I do think non legal background folks can use LLMs today to sense check legal ideas, like for instance, "what are my legal rights in this situation?"

These situations usually are not ones that an individual can justify the time or money to contact an actual lawyer, but then if they do decide to contact one they will come in with better questions and more of a sense of what they are expecting.

This is similar to medical. Should you use LLM to diagnose yourself, treat yourself with prescription drugs you buy from shady gray market online sellers? No. But you can use it very well to know when it's time to go to the doctor and what to ask.

You can replace "laywer" with "software engineer" in your post and it holds equally true. I think this also applies to other factions.

>I see 3 kinds of potential users.

3 kinds of users that don't scare the shit out of you.

My buddy works with lots of folks that have gone all in on Harvey. There are other kids of users.

There are 3x as many lawyers per capita in the US as we had in the 1970s. Credit agreements that used to be 50 pages are now 500. This increase in volume has not lead to any increase in actual value. The legal profession is a metastatic cancer on our society. Un-fuck this situation and maybe I'll think about taking your advice of not using a LLM.

I'm thinking back to some of my interactions with corporate lawyers and you know what? My bar (<- pun) for trusting Astra more than a lawyer is pretty low.

The proof would be that the attorney who did the work before, and who still reviews all ingested data, says it is.

You should free your dog then because a creature that intelligent should not be anyone's pet.

> You wont want or need associate attorneys

In commercial law, the business model is to rent the associates out to clients at rates higher than you pay (d/t your supervision). No client will pay $350/hour for your AI.

And for direct billing, it's not your interest to do things faster for the client (at risk of making a mistake).

So the real question is whether these pay-by-the-hour models will die as a result of competition from fixed-price transactions made viable via automation. I think not because the entire difficulty of legal work is that it's a complex bundle of issues until you sort them out; once sorted, there's really little to do other than execute (read: paralegals).

> No. It'll be like software. Entry level employment will be affected.

Whether everyone at the labs knows it or not, this it what they are doing to entire industries, starting with software: Embrace, extend, and extinguish

https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...

What a time to be alive!!?

It'll be like software. The demand for both software and legal services is effectively infinite. If it becomes cheaper to file lawsuits then we'll get more lawsuits, not the same number of lawsuits at a lower price.

There will still be jobs for competent associate attorneys to prompt LLMs and review output. But frankly a lot of attorneys — especially many of those who attended "Third Tier Toilet" (TTT) law schools — were always a waste of money and should have never entered the profession in the first place.

There are hundreds if not thousands of divorce lawyers. How is it a “cartel?”

Divorce is expensive because divorces are acrimonious and people in that situation would rather spend the assets fucking each other over than letting the other person get them.

AI will likely replace divorce lawyers (or, at least make existing divorce lawyers much more efficient) - for non-contested divorces that are amicable. Roughly 95% of what a divorce lawyer does in that context is entirely automatable.

Divorce is extremely cheap if you both agree on everything. Its literally one of the options on the divorce filing. You just pay the court fees. The problem is both parties rarely agree on everything and that's when it gets expensive.

There’s an argument to be made if divorce was easier than more people might get married when they are less serious. My view though is make it harder to get married and easier to get divorced.

"Marriage has no friction" is only something that could be written by someone who was never married.

> Made impossibly expensive

huh? Many/most are no-fault jurisdictions with strict rules on dividing community property. Divorce is easy and cheap, unless it's hard to find assets.

It's not just the divorce lawyers. We have 3x as many lawyers per capita in the US as in the 1970s. The system is designed by lawyers for lawyers. They will create more bureaucracy and more procedure to consume any surplus.

> the cartel of divorce industry

So like a group of lawyers among the "divorce industry" who set prices for divorce and stuff? or wtf do you mean?

Yeah, this matches my friends - they use LLM's for tons of things, using Eve and such. But nobody is replacing the lawyers themselves in cases of any value.

I also have friends in the world of high-volume personal injury (IE the morgan and morgan's of the world) but they also aren't being replaced by LLMs so far.

The word “never” is a dangerous word. In my lifetime I have seen it many times where something was supposed to “never” happen only to see it happen a few years later.

I'll buy that - but, how often does one need an honest-to-goodness expert? 99% of the time most people's requirements can be satisfied with general-competence, and, while the AI systems we have today are mostly only generally competent in porting code from one language to another - they are getting consistently better. I expect we'll start to see general competence in many more areas in the next few years.

LLMs are replacing human experts

Yeah geeks just stop trying! Seriously? Yeah, we’re in a bubble and narratives are ahead of reality but if you really don’t think AI is and will continue to eat knowledge work, just keep making your buggy whips.

Many people stepping in their domains of expertise with surface level knowledge

i agree with this, maybe.

I’ve found that good lawyers are about getting things done. You could nitpick the terms of a contract, but would you rather buy the house or have a perfect contract? Once I realized that, my view of lawyers changed (for the better).

YAGNI isn't just for software.

> I want to take a step back and observe that you're saying expensive specialists will continue to be required if two parties want to make an agreement.

A lot of contract-lawyering involves getting ahead of when things go south and the two previously agreeing parties find themselves in disagreement.

The problem is those two people are in conflict not collaboration.

If the expensive person is still better at getting what _you_ want from a contract then it’s always worth doing that.

There’s a reason lawyers are so expensive to begin with - they do very complicated stuff with very expensive consequences if done badly.

Millions if not billions of agreements are made every day without the involvement of lawyers. Lawyers are only needed for particularly complex situations and even there they are generally only "needed" in the sense that parties feel they would add value.

In my experience, those expensive specialists help me protect against things me and the counterparty cannot even imagine when making an agreement. Do you mean that this would be fixed by AI or by making simpler agreements?

How?

Contract law has evolved because transfering the ownership of things is fraught with difficulty. Ownership as a concept itself is notoriously hard to define.

Consider housing - when you buy a house you own that land. Except if its an apartment, you just own the internal walls. Or when the government aquires your land through eminent domain. Or mineral rights - which don't belong to you, you just own the top 5 feet of topsoil. Or if you neglect a property and some squats in it. Or if you were given a fraudulent payment, and they take back the property.

Just getting people to agree is notoriously difficult when people have different often conflicting desires.

You should know - for coding they make terrible mistakes as well.

But programmers have this concept of a "code review" where another person looks at the code to look for problems.

We use this same technique with our LLMs. Most mistakes are caught by having a second LLM look at it. Doesn't even need to be a different model - just make sure it has a different context.

> LLMs are the first genuinely useful legal tech since the Internet

That is an incredible statement that could not be further from the truth. Large scale adoption of email, searchable document databases like Westlaw, LexisNexis, PACER, etc.. , OCR Software, electronic signatures, and tons more have had a much more defineably positive impact on the legal profession since the internet came about.

I have found that it’s useful generally speaking to get the intent of contracts and red lines, but actual drafting I agree is where I lose all confidence. My guess is that the significance of the difference between using a word like “and“ or “or“ can be so meaningful that that level of nuance can often be lost. But I know nothing I’m not in the space, I just pay too much money for lawyers.

Law is certainly a less verifiable domain

I would rather argue they SPECIFICALLY will read everything. However, they'll likely often just be like "I'd phrase this differently but that works, too".

I’m thinking asking the same lines, but also looking into research / lab chemist so I can hang out with my best friend. If im going to change careers so drastically, I might as well enjoy the company.

> Human professionals put their reputation and finances at risk when performing their work. This risk functions as a guarantee.

Do you know how difficult and costly it is to sue an attorney for malpractice?

Do they?

Not being snarky here but I've encountered some really crappy lawyers and they seem to be doing great ($$$).

It really sucks one cannot publish something like "this guy charged me $5k, wrote my name wrong and forgot important paperwork the day we showed up to court" as that could make you liable for damages. There should be a safe harbor for that.

Your comment is a great example of the phenomenon where people think AI is an expert in areas that are not because they have no way to determine if the seemingly good looking result is genius, gibberish, or somewhere in between.

> The result covered a lot of details that were highly relevant but were absent from the original attempt

Another one suckered by the plausibility engines.

Did you validate the output with a lawyer?

Their work will get cheaper so it will become more accessible for people, increasing demand over all would be my guess to some degree at least.

The salaries are a result of the insane hours they have to do. Biglaw needs to pay those salaries to keep getting young associates in the door because burnout is a real problem in the industry. If salaries were lower and hours/stresses were lower, you would probably still find people wanting to be lawyers. A different kind of person, maybe.

(Not that I think AI will necessarily have that effect. We just don't know yet)

It’s a professional organization that only allows so many people to pass the bar per year so I don’t see that changing anytime soon… if anything if they need fewer lawyers they will just pass fewer people to keep their wages high

As costs for legal work drop we might expect demand for that work to increase. For example, it may be the case that legal help becomes accessible to entire segments of the population that had no access previously.

I don't want to imagine a world where juries were responsible for reading briefs and making decisions.

Sure as long as they stop making things up.

The ability to clearly and effectively communicate, and the ability to know and recall the facts of a case is always going to be important. It'd probably be a more fair system if judges and jurors couldn't see the attorneys or even the people they're representing. Now that many cases are handled over video calls it'd be pretty easy to make that change.

A lot of communication is conveyed through more than the words themselves though and even though it's often misinterpreted people put a lot of value in that information. I suspect that even if we took away the ability to see the people involved judges and jurors would still find bias in the tone/quality of people's voices. Even if everything was reduced to text they'd find bias based on word choices, phrasing, spelling, and grammar.

Yep, just yesterday, Nike removed one leach middlemen called retailers and sell directly through their app. Turns out working out very well for them too.

The law is much more interpretation and argument based than logic routing based than you seem to believe.

Nope. Paralegals and assistants do the type of thing GP hired a lawyer for (and lawyers do, as well).

Lawyers can practice, can represent you in court, etc. that's not going away soon.

If you think paralegals + AI can do the job of good lawyers you don't understand the profession at all.

Switching from ChatGPT Enterprise to Legora at my firm was a godsend, it's so much better for legal work, even with the frequent changes to the underlying models.

>Max Junestrand has consistently said Legora treats the model layer as swappable, selecting across frontier providers rather than building the product around one.

Vendor-neutrality for LLMs is such a weak thesis all around, whether for providers or consumers. It weakens the product by being promiscuous and gains no material benefit at all.

LLMs are magic byte(byte) functions, it doesn't make sense to say "we have different providers for magic".

the thing is a lot of the legal work which will go through this is drafting 100 and 1 variations of draft versions of standard contracts not containing any trade secrets where the contract can be drafted with "replacement/place holder names"

the kind of work mostly done by juniors not yet through their final exam and other "non" lawyers etc.

so it's a slippery slope of "lets just use it for <this> things where it doesn't matter" and then out of laziness and convenience it creeps into all the other places (at least for drafts).

Lawyers are already llm’ing their work and getting slapped around in court for making shit up.

There’s another case making headlines every week.

I get the feeling a lot of them won’t care about this stuff.

Yeah, it should be freely available, you have to be able to know the rules you're supposed to obey in order to obey them well. I've been making a free API for US law search, you can point whatever model you want at it: https://law.agentlookups.ai/

Very much a work in progress, only federal and state so far, no municipal codes yet, and no case law yet. Big hole, I know. Also working on making the search ranking work better.

> By using the legal search index, Astra for Law can search U.S. case law, statutes, regulations, court rules, and administrative decisions across a corpus of more than 230 million URLs, with sources added daily. Our work with Free Law Project, the nonprofit behind CourtListener, brings its case-law collection covering more than 99.9% of published U.S. precedential case law (opens in a new window) into this research experience.

Found that to be very interesting

Let's swap it out for the AI cartel!

>are pretty good at surfacing some stuff without having to pay lawyers

They are also really good at making stuff up as evidenced by the many, many, many examples you read in the news about actual lawyers using AI to write briefs that are full of errors and hallucinations.

In civil courts, you'd likely get more sympathy from a judge if you represented yourself and admitted your lack of understanding, rather than try to appear as someone you're not because you wrote some prompts and copied the output.

Current court systems around the world are just not built to handle the flooding of cases from the citizens.

The stupidest analogy is open source projects having a hard time accepting LLM generated PRs from the masses, because review process is the bottleneck.

No idea how to fix this, to be honest. In coding world, with some mental gymnastics, I can see code not being reviewed by people anymore. In courts, things generally have more consequences, and you can’t really roll back decisions that easily.

Relevant:

https://www.ftrain.com/nanolaw

JG Ballard could probably turn that into something darkly atmospheric, and even more dystopic than his usual depopulated worlds...

The legal system is slow and inefficient by design.

You presented the concern from my adjacent comment perfectly (“LLM performance: common law vs civil law” essentially). So is AI possibly just growing the “Reverence for Professional Experience” factor that plays such a big role for legal compensation here in the US?

I am not sure "eaten out" was the correct phrase to use here.

Call it a Prediction Market

justice doesn't have to be productive it has to be fair

construction productivity is actually down since the 60s!

>Every other profession has increased their productivity.

You dropped this /s

Yes. And you(r lawyer) can collect lawyer's fees and you can be made to pay the court fees, if you lose.

IME the American legal system is set-up to discourage litigation, though. A common tactic is to bury your opponent in the threat of heavy damages or jail-time to get them to settle for what you were originally after, which courts are perfectly happy to facilitate because it gets a potentially lengthy trial off their dockets. They'll punish (or be biased against) whichever party seems responsible for not accepting a "reasonable" settlement.

Algorithmic abuse of the system to extract payments already exists in the form of the debt collection industry.

I wonder how many parties would prefer an AI judge over delayed justice.

Judges don't care if people have to wait months. What does it cost them?

How are we fighting it online? You cannot

It'll be like radar detector detector detectors, which were a thing for a while. Each side (cops, speeder) buying detectors to detect the detectors.

can “penumbras and emanations” compete with hallucinations?

The blog post makes no mention. If the rate isn't 0% it should be mentioned for a field like this.

> it seems like for law specifically all relevant facts will be cited and checked easily by humans.

Don't be too sure about that. [0]

0: https://www.damiencharlotin.com/hallucinations/

> all relevant facts will be cited and checked easily by humans

I've talked to a lawyer about how they handle this. They do indeed double-check everything, since it'd be embarrassing (or worse) to send hallucinated statements to opposing council or to the court. They still find the assembly a huge time saver

But based on stories in the news on the subject, not everyone has this same level of diligence

Sure but just like generating 100x more code, someone has to review it. So you are wasting everyone in court's time (defendants, prosecutors, judges, staff) by making them parse through what is quite often a bunch of hallucinated slop. Time that could be much better spent on parties who prepared and reviews their own arguments.

The lawyers I know are very fixated on the problem of hallucinated case citations which is amusing to me as a onetime programmer, since case citations have a well-defined syntax and would be relatively easy to check programmatically.

It’s barely any better than a coin flip

54.0% isn't particularly high.

waiting for the actual Prodigy Corp. to show up and offer to buy your username out?

<my own consternation, stated>

a la Google.

As opposed to taking advice from the glucose wetware?

The business value will be when platforms can underwrite their LLMs' legal products.

That will be a decacorn product or more.

https://commonpaper.com/product/ai-agent/

In the US at least if a lawyer screws up a contract beyond what a competent lawyer would and causes financial harm they can be sued.

Poets.

Agreed. Six months ago, it was basically a gloried grammarly.

But lately, I’ve been taking hints from the “company brain” models, where it develops a running model of the case, and assesses each new piece as it comes in and updates the file.

I’ve also been using “Ralph Wiggum”-type models where you pass letter or contract drafts back and forth between agents with different goals (rules compliance, grammar, conciseness, ai slop detector, an opposing counsel critic, etc.). After a few rounds, it’s not perfect — but I start with a very good first draft in my hands.

What are you using for semantic searching?

> It doesn't really take a position.

Interesting! In other unrelated domains, models seem more willing to take a position. It may be nuanced, but they do tend to take a stance. I think it's good that it leaves the interpretation to humans, but I wonder if this is also some sort of a guardrail to minimize liability...

worse, it was trained on reddit law comments

Haha clever

Drop the s and p. Just law.

It’s cleaner.

You know exactly what they mean. But if somehow you miraculously don't there are hundreds of stories like this:

https://www.reuters.com/legal/litigation/lawyer-state-farm-f...

I love the slop word, to me it means effortless, average.

People confuse slop with "bad", but slop isn't bad per se, it only becomes bad when real effort was required.

Yeah you're probably right, the purpose would be to lock in whole industry verticals.

There are other possibilities:

1) Lawyers are not as naive as software engineers and will fight being replaces by new laws.

2) If they are replaced, OpenAI will take a cut commensurate with the amount in dispute (OAI, please credit me for the idea in the IPO brochure).

Now there will be even less friction to horribly abuse the law. To democratize the law we would need to move in the opposite direction - to always keep it simple and aligned with our intuitions. The more intricate and complex legal arguments become, the more abstracted they are from their original purpose and spirit.

Hence the crypto analogy - it was also supposed to "democratize", but the opposite happaned - it only further empowered the most powerful. Imagine legal case so purposefully complex that only those with access to best models have chances to participate and win the dispute.

Maybe I’m misunderstanding, but isn’t that what legislators do?

crop-maxxing

no, but they are responsible for all work being done under them. What is the point of paying a partner $2000/hr if the work of their subordinates is wrong?

Most interactions with big tech firms involve 4-5 people so a basic phone call is $5k-10k. It shouldn't be unreasonable to expect after paying $80k for a financing round that they issue the right docs to the right people.

I used ChatGPT the other day to resink a CPU with termal paste, replace a PSU in my PC and snake my kitchen sink from the wall. I didn't exactly need it's help but wanted someone looking over my shoulder so to speak. It can help you repair all types of stuff, but as far as county / building codes and such, that probably isn't far off. It seems to understand things quite well.

By that point money ceases to have value, because the value of money comes from the motivation it gives people to work. If AI does everything, then money is useless (unless AI like money for some reason).

there are always be next frontier of problems which require creativity, unless AI become supersmart completely make humans redundant in all cognitive functions.

> Those professions will likely evolve, but the traditional forms (ie writing code by hand, writing law filings by hand etc)

there will still be writing initial and incremental prompts by hands, until and if LLMs surpass humans in all intellectual functions.

in that case, legal cases would just come down to who has more compute lol. Many times cases win on their merits, but we've also seen evidence where overwhelming legal pressure can influence cases.

You know most of that was wash trading, right?

This also assumes that the AI owners _want_ a global utopia, or that governments will enforce that outcome.

As it stands, it seems far more likely to result in a wonderful life for a few, and an absolute catastrophe for most.

Doubt it. When someone can go to Astra MD for 75% of what they used to go to the doctor for, then the remaining doctors only have 25% as many visits. When doctors only have 25% as many visits, they have to compete on price and they make less per visit.

Same argument for plumbers. Everyone always jokes about what a good time it is to be a plumber. But what happens when all the software engineers turn to plumbing? Suddenly it's not such a good time to be a plumber anymore.

"7.5 hours - reviewing output"

How could that be enforced/checked?