The need for actual lawyers will persist I think from my own experience. I attempted drafting a contract with some points myself using AI, but after several edits I wasn't sure if it was correct. Sending it to an actual lawyer ended up in so many corrections I couldn't imagine the first time. One big thing was the overly excessive protective clauses which didn't make sense for reality or conflicted with another.
Its just like code I suppose, if you can read and understand and validate, you can use it to scale and otherwise it could end up being a vibe effort.
LLMs are the first genuinely useful legal tech since the Internet. I'm pretty shocked, though, at the delta between how competent Claude is on code versus legal work. It's good for research and data organization, but terrible for drafting. I wonder if this is a structural problem with the lack of feedback loops. In law, there's no compiler to check for logical or continuity errors in your brief, and there's no unit tests to check for correctness or performance.
Even without that, I think it'll be extremely valuable to clients to allow them to answer simple questions without a lawyer, figure out the lay of the land so they can supervise their counsel, etc.
You should know - for coding they make terrible mistakes as well.
But programmers have this concept of a "code review" where another person looks at the code to look for problems.
We use this same technique with our LLMs. Most mistakes are caught by having a second LLM look at it. Doesn't even need to be a different model - just make sure it has a different context.
Programming has a long standing culture of accepting the code to be somewhat wrong, so we have various tests, linters, reviews and error handling. Also in programming there are many ways to do something and it's the end result that matters most.
Not so in other knowledge work. There's no test harness for a contract and error is non-recoverable. Likewise in finance. There are specific ways of doing things and these ways are many times set in regulations. LLMs can assist all day, sure. But replacing the human, in highly regulated, zero tolerance for error environment?
You could say the Salvatorian Clause in contracts is like exception handling:
a "catch (all)": even if some clauses in this contract are illegal,
the remaining contract stays in place.
Logically, this actually doesn't make sense strictly speaking because the sentence creates a paradox: doesn't it make clear whether it includes itself or not, and each reading ends up in trouble. There is a "tradition" in law around the world to accept the only benign reading of such clauses, which I always found funny given that in all other ways lawyers adopt the most adversarial mindset imaginable.
The highly regulated, zero tolerance for error environment is a huge problem because anyone doing these jobs is more like a small sample size LORA than a general model.
It doesn't matter how smart someone is, they need specialized training to be good at these jobs. Specialized training in the area the company specializes in.
There is a category error in all this that is hard to think about because of the normal discourse and ordinary language. We say people work in "finance" but no one works in just "finance". They work at a company that has a specialization within "finance", inside a hierarchy that has specialization on top of specialization.
What we really need is exactly what we don't have and aren't going to get. A type of LORA that generalizes the task specific intelligence needed from a very small sample size and that in practice makes so many less mistakes in a highly regulated, zero tolerance for error environment that it is irresponsible to not use the model.
I have worked in this type of environment for 3 years and I have made zero mistakes in 3 years. The people that make even a small number of mistakes get fired.
Any real automation in this area is going to be incredibly slow and piecemeal over a long period of time because even an amazing model would need a long time to prove itself against what the human standards for error rates are.
> Doesn't even need to be a different model - just make sure it has a different context.
I find that the number of issues discovered is noticeably higher if you do use different models though. I'm doing some very finicky things (formal semantics) and find that there's value in review panels as large as 5 different models from different families. It gets even more profitable if you set it up as a truly agentic panel where after writing their own separate reviews they get to see what others have said, and adjust their positions or defend them etc. Some models are not that good by themselves but can be surprisingly good at finding flaws in others' reasoning (Grok for example).
I'm trying to understand why I'm getting replies basically saying "they'll never be good lawyers" when all I said is that you can improve the output by getting them to check each other.
halamadrid · · focus · HN ↗
Its just like code I suppose, if you can read and understand and validate, you can use it to scale and otherwise it could end up being a vibe effort.
rayiner · · focus · HN ↗
Even without that, I think it'll be extremely valuable to clients to allow them to answer simple questions without a lawyer, figure out the lay of the land so they can supervise their counsel, etc.
qarl · · focus · HN ↗
But programmers have this concept of a "code review" where another person looks at the code to look for problems.
We use this same technique with our LLMs. Most mistakes are caught by having a second LLM look at it. Doesn't even need to be a different model - just make sure it has a different context.
flyinglizard · · focus · HN ↗
Not so in other knowledge work. There's no test harness for a contract and error is non-recoverable. Likewise in finance. There are specific ways of doing things and these ways are many times set in regulations. LLMs can assist all day, sure. But replacing the human, in highly regulated, zero tolerance for error environment?
qarl · · focus · HN ↗
flyinglizard · · focus · HN ↗
>> You should know - for coding they make terrible mistakes as well.
>> But programmers have this concept of a "code review" where another person looks at the code to look for problems.
jll29 · · focus · HN ↗
Logically, this actually doesn't make sense strictly speaking because the sentence creates a paradox: doesn't it make clear whether it includes itself or not, and each reading ends up in trouble. There is a "tradition" in law around the world to accept the only benign reading of such clauses, which I always found funny given that in all other ways lawyers adopt the most adversarial mindset imaginable.
telliosic · · focus · HN ↗
It doesn't matter how smart someone is, they need specialized training to be good at these jobs. Specialized training in the area the company specializes in.
There is a category error in all this that is hard to think about because of the normal discourse and ordinary language. We say people work in "finance" but no one works in just "finance". They work at a company that has a specialization within "finance", inside a hierarchy that has specialization on top of specialization.
What we really need is exactly what we don't have and aren't going to get. A type of LORA that generalizes the task specific intelligence needed from a very small sample size and that in practice makes so many less mistakes in a highly regulated, zero tolerance for error environment that it is irresponsible to not use the model.
I have worked in this type of environment for 3 years and I have made zero mistakes in 3 years. The people that make even a small number of mistakes get fired.
Any real automation in this area is going to be incredibly slow and piecemeal over a long period of time because even an amazing model would need a long time to prove itself against what the human standards for error rates are.
int_19h · · focus · HN ↗
I find that the number of issues discovered is noticeably higher if you do use different models though. I'm doing some very finicky things (formal semantics) and find that there's value in review panels as large as 5 different models from different families. It gets even more profitable if you set it up as a truly agentic panel where after writing their own separate reviews they get to see what others have said, and adjust their positions or defend them etc. Some models are not that good by themselves but can be surprisingly good at finding flaws in others' reasoning (Grok for example).
chrisjj · · focus · HN ↗
More to the point, programmers have this concept of a complier - which delivers 100% success in catching uncompilable errors.
Lawyers don't.
qarl · · focus · HN ↗
Defensive much?
chrisjj · · focus · HN ↗
As much as needed.
qarl · · focus · HN ↗