Moneyball for law: picking the best fantasy league legal team
Improving the likelihood of success
Our guest for this blog post, Mark Verhagen, is a PhD candidate at Nuffield College at Oxford (https://www.sociology.ox.ac.uk/people/mark-verhagen). In addition to his PhD, Mark also operates a machine learning consultancy focusing on optimizing people and talent.
We met in late 2020 on an Oxford University Law Facility Zoom presentations, and struck up a conversation about combining the use of computer vision and natural language processing to analyze legal documents. We decided we had to find a project to tackle together. While we waited for the right project, we decided to toy with the questions – how do we optimize the likely performance of legal teams, and therefore how do we improve the likelihood of success of legal work?
Much of legal technology seems to be focused on solutions that seem to “robotize the lawyer”, but Mark and I want to use this blog post to explore what could happen if we apply technology and data science to the layers above the expert services, and “robotized” those layers?
In other words, what if we optimized:
team selection
information preparation
What if we “robotize” everything up to the point when the lawyer goes into a room with a client?
Picking the fantasy league legal team
There is no single “best legal team” for all matters. Each matter will require different expertise, personalities and tendencies.
Mark notes during our conversations that there is an emerging practice in the management consulting world where data is used to improve hiring and assembling a team in order to ensure “case success” (assuming we know what “case success” means).
Unlike management consulting, when lawyers are picking their team for transactions or litigation, partners at law firms typically pick their team based on:
who they know in their firm
who they have worked with in the past
who they know might have relevant industry experience
In other words, the process is based almost wholly on intuition or “gut feelings” of senior lawyers. While it is possible that some firms have adopted a data-driven approach to pick the best team for each piece of work, the majority of lawyers seem to select their team using subjective metrics, or, at least, based on imperfect information.
… I wonder, can we explore the question of “what if the whole management team of your law firm is replaced by robots?”
This is an optimization problem – how do we pick the best team for any piece of legal work?
If the management team is replaced by robots, what decisions would be made a robot management team? How would we design a “training problem” to ensure the robot management team makes the best decisions?
What stops us from immediate experimentation?
If we had data about lawyers, such as their skills, their strengths and weaknesses, and their experiences, then perhaps we can immediately conduct experiments… but unlike sports, vital statistics and information about lawyers are not widely available. Even where data is privately available, the data tend to be either self-reported or trapped in the minds of those people who are within two degrees of separation from a lawyer.
How do we optimize everything up to the point the lawyer picks up the work?
Since we do not have data, Mark and I discussed how we can theoretically address this optimization problem, assuming that we could access relevant data, such as:
education and training history for each lawyer
documents opened by each lawyer
hours billed by each lawyer
narration of work done by each lawyer
anonymized annual performance review scores for each lawyer
This is how we would pull together our heist crew
Watch Mark and I discuss how we might solve this optimization problem in our unscripted conversation here:
Transcript of conversation
(Note: transcript is automatically generated, and may contain misspellings and typographical errors)
00:00
welcome to another not so casual
00:02
conversation this time
00:03
um we're talking to mark verhagen from
00:06
oxford university
00:07
he is an expert in machine learning and
00:09
we're going to talk about this idea of
00:11
money ball for law mark we've been
00:14
talking for the last couple of months
00:15
and i'd love your synopsis of this
00:17
problem we're trying to solve
00:19
i would have to give a synopsis of the
00:20
problem we're trying to solve i think
00:22
it's
00:22
easiest to make it as concrete as
00:24
possible so
00:26
many people have been talking about
00:27
robotizing all kinds of
00:30
manual tasks usually relatively low cost
00:33
tests which are done at scale
00:36
but there are other places
00:39
where we can use artificial intelligence
00:40
or analytics to actually improve our
00:42
decision making or or
00:44
efficiency and so so let's let's picture
00:46
the following
00:47
scenario where instead of optimizing
00:50
a lawyer which is basically the high
00:52
frequency worker
00:53
but let's automate that which happens
00:56
above the lawyers or the management
00:57
layer the part
00:59
so let's let's imagine the following
01:00
situation which is
01:02
we have a case a concrete case which we
01:04
have to
01:05
echo with our firm and we need a team of
01:07
five people
01:08
and we have four already and we need to
01:10
pick the last one
01:12
so how do we do that how do we pick this
01:14
fifth person
01:16
what would be the optimal way to do this
01:18
what is the actual way we're doing it
01:19
right now and how can we move from the
01:21
actual way
01:22
if it's up ideal to the optimal way
01:25
so i think that's the that's the
01:26
question which is most feels most
01:28
intuitive to me
01:29
to try to answer so how do you do this
01:33
and in particular let's say we have
01:37
a sort of sphinx who you can ask a
01:40
question
01:41
right so let's say we have a 100 lawyers
01:45
and you can ask the sphinx a question to
01:47
which the swings gives an answer
01:49
and so what would it be if you would ask
01:51
these things
01:53
what will be the most precise question
01:55
we can ask
01:56
based on which we can pick up the phone
01:58
call that person or
02:00
whatever way we want to do it pick the
02:02
fifth person
02:03
and and what are what is the current way
02:06
we're doing it and how can we
02:08
incrementally improve this process
02:12
so i think that's the that's the that's
02:14
the question in a nutshell
02:15
i suppose and that's that's where all
02:18
the complexity starts of course
02:21
and for me i mean how are you how do you
02:23
think people do this right now
02:25
in law firms right now these sorts of
02:28
decisions about
02:29
what you or who you put on a team is
02:32
very
02:32
personality driven it's very experience
02:35
driven
02:36
a senior person in the team would say i
02:38
have worked with person a before
02:40
therefore i like person a and he or she
02:43
is going to join my team
02:46
rarely is there any statistical or
02:49
deeper data analysis that's done in this
02:52
selection
02:53
now more and more these days law firms
02:55
are thinking about capacity
02:57
this purely quantitative factor of who
03:00
has time to help out and therefore who
03:02
can we bring on
03:04
but it doesn't bring into account well
03:06
what's that person
03:07
good at what's that person's experience
03:10
what is
03:11
the relationship that person might have
03:13
had to the client on that matter
03:15
and all of these other variables which i
03:17
think play a greater role than simply
03:19
how many hours do you have free this
03:21
week
03:22
i mean if you care about uh the
03:25
performance
03:26
right if you only care about billing
03:27
hours then possibly not
03:29
well that's a strong assumption yeah
03:31
yeah
03:32
i mean it's one of the many reasons why
03:34
applications of ai in practice that they
03:36
feel because you don't take into account
03:38
the actual capacity constraints which
03:40
are there
03:40
so it's great that they are already
03:42
accounted for so there are only a
03:43
limited number of hours in here so i
03:46
think that's a very
03:47
important component but unfortunately
03:50
the way you describe it there's very
03:52
little optimization going on
03:54
within the self-constraint so law firms
03:57
suffer from the problem of they don't
03:59
really know where to
04:00
start and and you know you've you've got
04:03
this experience from the management
04:04
consulting space having
04:06
essentially not maybe not solved this
04:08
problem but um
04:09
having worked on solving this problem in
04:12
management consulting
04:13
so maybe i'll ask you how how is it done
04:15
in management consulting
04:17
the situation the status quo that you
04:18
described for law
04:20
is very applicable to management
04:22
consulting so capacity
04:25
and personal preferences and biases so
04:28
in that sense they're very similar
04:32
so let me describe
04:35
one of the added value or advantages of
04:38
working in management consulting from
04:39
the perspective of selecting teams and
04:41
composing teams
04:42
which is the fact that they are quite
04:44
rigid
04:45
in the way they collect information
04:47
about their individual consultants
04:50
how they perform along a variety of axes
04:53
basically
04:54
so you'll have a performance report
04:56
after every project it'll be filled up
04:58
by using the project manager
05:00
um you can relate this over time so
05:02
every three months you get a new one
05:03
and you obviously have a lot of data on
05:05
promotions which are very formalized etc
05:08
so in a way it's in the management
05:10
consulting it it
05:11
it almost fundamentally
05:14
boils down to a question of bias in
05:17
evaluation and i think in the legal
05:20
industry
05:21
what we suffer from is that there's a
05:23
lack of data
05:24
now i i know a very very few firms and
05:27
in fact i can't think of any
05:29
that does a post-mortem after every deal
05:32
at best you might have a quarterly
05:35
review of an employee's performance
05:37
generally speaking it's once a year and
05:40
so
05:42
those are the data points that people
05:44
have in terms
05:45
of assessment of the performance of
05:47
individuals
05:48
and that's a big span of time and
05:50
they're very infrequent little points
05:53
supposing that is true for all law firms
05:56
that there's only
05:57
annual reviews of lawyers what other
06:00
data points can we look at
06:02
to get that quality qualitative data on
06:04
performance
06:06
yeah now this is the this is the big the
06:09
big
06:09
big question um what to collect
06:12
and how to collect it in a consistent
06:15
and accurate way
06:16
let's assume there is a lawyer there is
06:19
a record
06:20
somewhere that um ranks lawyers out of
06:22
five stars
06:23
um and and that is in the database
06:26
somewhere so
06:30
i guess i would i would ask there would
06:32
be the naive naive way of doing this so
06:34
which
06:35
which person has the best rating uh but
06:37
i think
06:38
well an obvious qualification of this
06:41
approach would be well
06:43
a lot of things around this case are
06:44
already set in stone for example
06:46
the industry or the the general topic of
06:49
the case
06:50
so a more precise question would be
06:53
sphinx
06:55
from all individuals who have done maybe
06:57
at least 10 cases within this industry
07:00
who has the best performance who had the
07:02
most win rate it doesn't have to be the
07:04
same person as before right so you ask a
07:06
more specific question
07:08
which would lead should lead at least to
07:10
better performance because you would
07:11
have the person who's best in this
07:12
specific thing
07:13
right the question is then obviously
07:16
what are these cases which
07:17
levels do you specify because you can't
07:19
ask
07:20
two specific questions you just we won't
07:22
even even if we were perfect in
07:24
collecting everything you just won't
07:26
have
07:26
the volume the way you described it it
07:28
almost sounds like you're saying hey
07:30
pick the features that are most
07:31
meaningful first
07:33
um and and one of the things that i
07:36
think we're going to have trouble with
07:37
inside the legal industry is that those
07:39
features are not necessarily measured
07:42
and so what would work as proxies
07:45
for for those features and i can tell
07:48
you the data that is gathered by law
07:50
firms and it's gathered
07:51
with great stringency it's time um
07:55
everything is billable time and so you
07:58
have a lot of record
07:59
on what lawyers have done how they spent
08:01
their time and
08:03
generally they'll give a narrative
08:05
description of what they spent their
08:07
time doing
08:09
on the flip side of that is you can see
08:12
what documents
08:13
and what words were written during those
08:15
time periods or what words were read
08:17
during that time period and so without
08:21
this external qualitative
08:24
assessment by more senior partners you
08:27
do have
08:28
time record and documentary record
08:33
so then what would you do with those two
08:35
parcels of data
08:37
and what can we possibly what can we
08:39
possibly extrapolate
08:41
now i think i think this is a crucial
08:43
part this content content-based
08:45
predisposition so i think actually on
08:48
learning this it's
08:49
very interesting because management
08:51
consultants don't necessarily have this
08:54
finesse in the way that people actually
08:56
do their work so how many hours do they
08:58
spend on this part in that part and
08:59
what's the preference between it
09:01
so this is i think it's fascinating for
09:03
me new world that would open
09:05
so i've worked much more with for
09:07
example the traits and the personality
09:10
traits of individuals and how does a
09:11
team
09:12
of very dominant people together feel
09:16
even though every individual one seems
09:18
to be a high performer
09:20
but i think this is this is truly um
09:23
yeah i mean this is something which i
09:24
think the law scene might be
09:27
fairly unique in in a way that you can
09:28
actually go so precisely into the way a
09:31
person
09:31
does his or her job like yeah that's
09:34
fascinating yeah
09:36
yeah and i think it's not utilized at
09:38
the moment um
09:39
at least it's not utilized by anyone
09:41
i've spoken to
09:43
um so so i guess maybe i'll
09:46
turn the question around and go back to
09:48
this features id that we spoke about a
09:50
few minutes ago
09:51
which is if we were to have
09:54
the sphinx and the sphinx has all the
09:57
power in the universe
09:59
what do you think are the five up to
10:01
five
10:02
most meaningful features that you would
10:05
look for
10:05
in assembling a team in professional
10:08
services
10:11
you need a baseline familiarity with
10:14
what you're supposed to do
10:15
but i think most people would maybe
10:19
you need that one and after that in this
10:22
case
10:22
it's the other four team members hands
10:26
down i would say
10:26
it's the interaction which will be
10:28
generated
10:30
unless it's a type of business where you
10:32
can really
10:33
decompose the tasks at hand and everyone
10:35
works in a sort of
10:36
independent way but generally speaking
10:39
at least in the management consulting
10:40
sphere
10:41
i would say that the composition is most
10:44
leading so the energy that's created by
10:46
the interaction between individuals
10:49
which is not something which stands on
10:51
its own it's not the case that you can
10:53
always assemble the
10:54
five best people it's in the context of
10:57
the project which is going to be done so
10:58
the relationship to the client
11:00
etc um so it's just
11:03
it's more complicated than maybe maybe
11:06
this is first i would say that
11:08
the people plus the high level
11:11
context of the case so what is the
11:13
industry what is the type of work has to
11:15
be done
11:16
high pressure low pressure etc i think
11:19
that that will be the
11:21
key ones i would look at first from
11:24
[Music]
11:25
two aspects one the way that legal work
11:28
is compartmentalized and broken out
11:32
you do find in a lot of cases there are
11:35
very clear dividing lines
11:37
so i'll give you an example from a
11:39
transaction where
11:41
if you're doing a merger and acquisition
11:44
you would have a corporate lawyer or
11:46
corporate team leading this transaction
11:49
but then the specific issues that arise
11:51
like employees and tax
11:53
they go to specialists and they do
11:55
siloed work
11:57
and it doesn't really matter about
11:58
personalities in that case
12:01
but what does matter is your core
12:05
emanating your corporate associates and
12:07
partners
12:08
they have to work well together and they
12:11
have to work well
12:12
with the client which
12:15
is a really difficult to measure
12:19
thing and i think this is why
12:22
at the moment the current practices
12:24
partners will go i like
12:25
person a i like to work with them
12:27
therefore we're going to have them on
12:28
the team
12:30
even if they're not the most skilled
12:31
person now i can definitely see that and
12:34
i can see
12:34
it's also i mean we always deride the
12:37
strategy in a way
12:38
that just seems like the the most sort
12:41
of stupid thing to do but
12:42
i i actually personally think that it's
12:44
a pretty good heuristic
12:46
in a limited information setting so that
12:49
that's a lot of complexity
12:52
to take into account definitely it's a
12:54
web yeah yeah
12:56
indeed but i mean if if you if you're
12:58
able to define the axes
13:00
and make a matrix and you can actually
13:02
fit every client or every aspect of the
13:04
case into it
13:07
you at least go some way into
13:10
determining the context of the case so
13:11
with context i mean the things you
13:13
cannot change
13:14
it's the people you have to work with
13:16
the client that's the
13:18
case in content-wise duration even
13:22
whatever so i think these things have to
13:24
be set in stone
13:26
because it's so if the variation within
13:29
between cases is large and this is the
13:30
most important one to actually find
13:32
some way to get a grasp on it somewhere
13:34
to categorize it
13:36
so i think intuitively speaking that
13:38
will be one of the ways to
13:40
to make make progress to find
13:42
classification and it can be
13:44
i mean we don't have to be perfect from
13:47
the start we just have to be able to
13:48
start
13:50
looking at so let's say a table of the
13:53
following form so
13:55
every row is a type of case and we have
13:58
maybe five
13:59
which is way too few but let's say
14:02
we start somewhere right exactly exactly
14:05
yep and then we have four types of
14:07
so five like a rows and four types of
14:11
let's say four types of lawyers in this
14:14
case we don't we neglect the interaction
14:17
between the team
14:17
etc and we just want to see performance
14:20
you want to see how
14:22
type a lawyer does for every five of the
14:25
different
14:26
types of cases and we just want to look
14:28
at this this data and say like okay so
14:30
we see that the type a lawyers are best
14:32
in
14:34
case type d and case type e for b
14:37
they're all round doesn't really matter
14:39
they just perform well everywhere
14:41
c is maybe only good in a and b etc and
14:43
that's just a starting point something
14:45
to to look at
14:46
and the moment you look at it you know
14:48
it's wrong you just know that we have to
14:50
be more specific right since it's just
14:53
i mean it's just too high level and
14:55
that's the moment when we decide to
14:57
either
14:58
split up one of these columns to be more
15:00
specific
15:01
or fill up one of the rows be more
15:03
specific we just have to keep on doing
15:05
this iterative process until we're happy
15:06
with the way this matrix
15:08
and it can get very large obviously we
15:10
have to feel comfortable with
15:12
the granularity of the roads and the
15:14
colors
15:16
so i think we have to yeah we have to
15:18
have a high level categorization of
15:20
cases a high level generation of lawyers
15:23
then we even discard the whole
15:25
the whole team complexity i think that
15:26
that would be the starting point
15:28
um and and i mean i'm not sure because
15:31
i think it's it's not straightforward it
15:33
wasn't said for the management
15:35
consulting but
15:36
so every cell in this table is the
15:38
performance of this type of lawyer for
15:40
this type of case
15:41
but do we really are we able to generate
15:44
performance is it
15:45
is it straightforward i i think so i
15:48
mean you can have
15:49
client feedback that tells you dude are
15:52
they happy
15:53
um and sometimes you even have
15:57
actual measurements like if you go to
15:59
court did you win or did you lose
16:02
and so i think i think it is measurable
16:05
and i really like the idea that you
16:08
suggested which is
16:10
start somewhere create a matrix start
16:13
start putting it down and then you'll
16:15
know where the gaps are and then you'll
16:17
know
16:18
what to break apart be more nuanced
16:21
um i do have a an interesting question
16:24
which is
16:26
measurable improvement what what sort of
16:29
uplift can we expect looking at other
16:32
industries
16:33
if law firms were to start taking this
16:36
money ball approach to picking teams
16:39
so so yeah so the question is how much
16:41
can you actually
16:43
so how well does this matrix discern
16:46
differences in performance
16:47
but let's say i mean if it's reasonable
16:50
that a
16:51
type of lawyer has a performance of 30
16:53
percent where
16:55
30 of the cases this person does have a
16:58
reasonably
16:59
successful outcome and this person has
17:03
70 percent
17:04
in some other type of case and that 40
17:07
has this delta just this gap between the
17:08
two becomes larger that's where the
17:10
possible gain is
17:11
because right now this type lawyer a is
17:13
just randomly assigned to a and b
17:16
so basically if it's a 50 50 split this
17:19
person will have 50
17:20
success but if you would just assign
17:22
this person to
17:23
type d which this person has percent
17:25
success well
17:26
that 20 percent more case of success
17:28
that that's your gain there's a possible
17:30
efficiency
17:31
question is is it as straightforward as
17:33
that and are the gaps as large as this
17:35
right and what's the cost in producing
17:37
the data in order to have this
17:39
efficiency gain
17:40
exactly so it's complicated it's very
17:42
difficult to know
17:43
how much to be gained from this
17:47
um they're also completely different
17:49
performance metrics or outcome metrics
17:51
to think to account so let's say
17:52
you as a law firm assume that you always
17:55
get a reasonable
17:56
outcome let's just test it how much time
17:59
did it cost
18:00
person a or person b and we want to
18:02
minimize time
18:03
maybe be a reverse incentive because you
18:05
want to build more so so
18:06
this is definitely a question here well
18:09
it's changing the landscape is
18:11
morphing towards profit rather than
18:13
revenue right
18:14
i see i see so because in an efficiency
18:16
perspective it will be completely
18:18
reasonable to so let's say in a fixed
18:20
fee contract
18:21
maybe a management consulting setup
18:23
where you just you bill
18:24
a number of hours pre-determined and
18:27
then you just have to make
18:28
student work basically then it will be a
18:30
strong incentive to make sure you
18:32
assemble the team which does it in the
18:33
least amount of time
18:35
in the most efficient way so there you
18:37
would have a sort of
18:38
the cells wouldn't be successful or not
18:40
the cells would be number of hours
18:42
necessary for a success
18:44
there's something else and that would be
18:45
a different perspective how you look at
18:47
the possible gains to be had so if there
18:50
is a strong intuition amongst
18:52
law firm partners or stakeholder
18:54
shareholders in a way right now
18:56
that some people are just good at
18:58
something and i'm bad at
19:00
other things and it costs a lot more
19:02
time but that delta has your potential
19:04
efficiency gain
19:05
but i mean i agree it's extremely
19:07
difficult to know
19:08
a priori what these these differences
19:11
are because if it's just a percentage
19:12
point or something
19:14
it's really marginal the possible gain
19:16
you get into optimizing allocation of
19:18
people to cases
19:20
um so it really is a it's an intuition
19:22
question
19:23
at the heart but if it's as big as 20
19:26
from just allocating one person around
19:28
then that's an enormous gain
19:31
it almost i mean you can break it down
19:33
like any business problem
19:34
it almost feels like you have to
19:36
determine the objective first
19:38
then you just have to start taking
19:39
actions you have to you have to
19:41
experiment and go right
19:42
we have this amount of data we have this
19:44
available let's see what we can do and
19:46
then
19:47
iterate from that point onwards
19:51
i mean and you can do it in all kinds of
19:54
parts in the process if you look at
19:56
roboticizing or automating or at least
19:59
giving decision support to the
20:02
management level
20:03
it's even fair to say for example in
20:06
picking the cases you take up
20:08
as a as a as a law firm do
20:11
i want to do this or not and basically
20:13
using this type of approach what is my
20:15
availability
20:16
right now what do i know of these types
20:18
of lawyers doing this type of case
20:20
and actually having a sort of decision
20:22
support in terms of okay we only have a
20:24
30
20:24
probability of actually doing this
20:26
successfully right will be a different
20:28
part right it's the same data the same
20:30
so i think law firms have to be very
20:32
pragmatic here and i think
20:34
about where do we think this lack of
20:36
decision support
20:38
is the biggest where can we where can we
20:40
improve the most
20:41
and um pick those very different fruits
20:44
exactly
20:45
exactly because obviously you have to
20:46
think about how difficult it is but
20:47
which proxies matter
20:49
for this specific optimization problem
20:51
can we get them
20:52
yes there's a there's a really active
20:56
field in
20:57
legal tech at the moment and that's case
21:00
law analytics
21:01
looking at judges looking at the
21:02
circumstances and then measuring
21:04
what's the likelihood that this argument
21:06
would succeed in front of this judge
21:09
and i think this conversation is kind of
21:12
creating a much more broad
21:13
spectrum data analytics problem of
21:17
who do we pick as our team do we even
21:20
take on a case given this is a team that
21:22
we have
21:24
um and that's fascinating i think
21:28
your insights on that's just phenomenal
21:30
and the power levels to the
21:31
management industry is extremely
21:34
informative
21:35
the differences are also very very
21:37
interesting i must say
21:39
in the end of the day i mean it's just
21:40
this matrix thinking about
21:43
we we know this is this is external the
21:46
case
21:46
before us and this is what we choose and
21:49
finding a way to fill this first matrix
21:51
even if it's scuffed in all kinds of
21:54
ways just start looking at it
21:56
start trying to dig i think it's it's
21:58
it's very widely applicable
22:00
uh yeah marketing and magic resulting
22:03
it's just basic
22:04
data analytics of course so it's um
22:07
in a way it's the parallels are so
22:09
there's so many
22:10
but they're always very interesting
22:11
nuances where i think a lot tech or
22:13
legal
22:15
work is just such a fascinating
22:17
sub-specific
22:18
domain here yeah i would i would love
22:21
for a law firm to listen to this pick it
22:23
up and say hey let's experiment
22:25
because i think knowing what the
22:27
potential
22:28
impact would be from this type of work
22:31
would be
22:33
possibly possibly game changing
22:37
and you know uh if a firm says let's uh
22:41
let's experiment for six months
22:43
i think that would be personal you know
22:46
i mean this type of
22:47
problem is so rich i would definitely
22:49
love a good time
22:51
oh well look we will do what we can um
22:54
and mark i've had so much fun chatting i
22:57
think
22:57
what you've the insights you've given
22:59
the kind of pointers as to the actual
23:00
steps people can take
23:02
is hopefully gonna spark someone
23:05
inside of a law firm to take these steps
23:08
and if they do we'd love to hear from
23:11
them to see what the results are
23:13
of course and thank you again for your
23:16
time
