Gen AI is Good for Neither Urgent or Important Work
We have been spending a bit of time on a question that sounds simple, at least on paper — what kind of legal work can you actually get gen AI to do reliably?
Working through this we stumbled into the Eisenhower Matrix, accidentally. When we mapped legal work to the two axes of urgent and important, we found that the only tasks most of us as lawyers are comfortable assigning to gen AI are the ones that are neither important or urgent… and this is exactly the sort of work that should be deleted and ignored. The work which, historically, would and should not have been done is the work that is best suited to the machine.
And although this sounds like the most controversial opinion we have published, let’s hold the flaming pitchforks until we come to the end of this piece.
Start with “important”
If something is important for a lawyer, that means we have got to get it right. That’s the whole point of lawyers. So, is this a task we want to assign to a machine? The answer is “no”, unless we also have the means to check the work afterwards. Which leads to the inevitable question — if the quality assurance has to be there to check the work, then where does it benefit us?
In our opinion, gen AI only fits if we have designed our workflows and processes around its capabilities, and that means undertaking a transformation exercise before anything “important” can be done by a machine.
As organizations, law firms are some of the best QA systems ever created. We have written about this before — juniors, then seniors, layers of review before anything reaches a client. Gen AI is fundamentally different, and while many of us have spent the last three years remapping processes onto gen AI capabilities, most of us have not yet completed the transformation exercise.
What about “urgent”
For urgent work, we would argue gen AI is even less suitable.
If work is urgent, the work has to get out the door. Can we suddenly say that as long as it done, getting it half right half the time is “good enough”? For most of us, probably not.
Then, what can gen AI do?
“Urgent and important” and “not urgent but important” work can go to a machine, provided the right QA exists, because the human expert verifies and signs off on the work.
“Urgent but not important” and “not urgent and not important” can go to a machine, if whomever owns the risk decides they can tolerate an error. That means, sometimes, an in-house legal team can decide they can accept a level of risk. e.g. reviewing 500 NDAs, because the alternative was reading none of them. Unfortunately, that means external counsel is almost never entitled to make that call, unless the client has explicitly signed off on some degree of mistake tolerance.
So all four boxes collapse into two questions — have we built the processes around the machine, or are we willing to tolerate the risk?
Good for nothing?
Gen AI is perfectly suited for work that is neither important or urgent. The machine can produce an answer very quickly, and no one is harmed by the occasional hallucinated answer.
So, it is almost like saying generative AI is a really good replacement for work that would never have been done in the first place.
Except, two things are worth noting:
first, litigation already ran this experiment. Technology assisted review has done probabilistic review for over a decade, with disclosed methods, agreed recall rates and judicial approval.
second, we measured gen AI against whole tasks, rather than components of tasks. So, we are not assessing whether components within tasks can be easily verified or tolerate higher risks.
Alternate way to measure
So, let’s retire the use of “urgent” and “important”, and instead, we can measure:
Cost of error — what happens when legal work is wrong, and how long before someone finds out.
Cost of transformation — what it takes to catch errors in legal work before it is sent
If a piece of work has a high cost of error, and a low cost of transformation, this should be the first focus of transformation — this is important work where gen AI can potentially make the largest difference.
Except, what’s the real cost?
Take a transactional lawyer doing diligence. Historically, we draw a materiality threshold and say: “we are not going to look at this, we are going to take the risk”. With gen AI, now we can lower the materiality threshold and let a machine do the review — knowing the thing producing it is probabilistic and will not get it all right all of the time. So, we are throwing token economics at a problem we previously just took a risk tolerance on. We have gone from turning a blind eye to turning a probabilistic eye.
Is that better? Some would argue yes.
Worth it today, at least… but we are assuming cheap tokens.
The rule of thumb
For now, we believe in a rule-of-thumb:
if you haven’t completed your process mapping and transformation work, then put the machine on a tranche of work that was never going to be done, and stop trying to make it “good enough” for the important and urgent work; and
if you have decomposed matters into steps small enough, then the machine can tackle and automate a lot more components of the work — which is the benefit of gen AI that we have all been promised.
