An Uncommon Task: Participatory Design in Legal AI

03/08/2022
by   Fernando Delgado, et al.
0

Despite growing calls for participation in AI design, there are to date few empirical studies of what these processes look like and how they can be structured for meaningful engagement with domain experts. In this paper, we examine a notable yet understudied AI design process in the legal domain that took place over a decade ago, the impact of which still informs legal automation efforts today. Specifically, we examine the design and evaluation activities that took place from 2006 to 2011 within the TeXT Retrieval Conference's (TREC) Legal Track, a computational research venue hosted by the National Institute of Standards and Technologies. The Legal Track of TREC is notable in the history of AI research and practice because it relied on a range of participatory approaches to facilitate the design and evaluation of new computational techniques–in this case, for automating attorney document review for civil litigation matters. Drawing on archival research and interviews with coordinators of the Legal Track of TREC, our analysis reveals how an interactive simulation methodology allowed computer scientists and lawyers to become co-designers and helped bridge the chasm between computational research and real-world, high-stakes litigation practice. In analyzing this case from the recent past, our aim is to empirically ground contemporary critiques of AI development and evaluation and the calls for greater participation as a means to address them.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/31/2020

Robustness and Overcoming Brittleness of AI-Enabled Legal Micro-Directives: The Role of Autonomous Levels of AI Legal Reasoning

Recent research by legal scholars suggests that the law might inevitably...
research
08/26/2020

An Impact Model of AI on the Principles of Justice: Encompassing the Autonomous Levels of AI Legal Reasoning

Efforts furthering the advancement of Artificial Intelligence (AI) will ...
research
09/10/2020

On the Fairness of 'Fake' Data in Legal AI

The economics of smaller budgets and larger case numbers necessitates th...
research
11/01/2021

Stakeholder Participation in AI: Beyond "Add Diverse Stakeholders and Stir"

There is a growing consensus in HCI and AI research that the design of A...
research
01/24/2023

Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards

Artificial Intelligence (AI) is taking on increasingly autonomous roles,...
research
11/01/2020

Lawmaps: Enabling Legal AI development through Visualisation of the Implicit Structure of Legislation and Lawyerly Process

Modelling that exploits visual elements and information visualisation ar...

Please sign up or login with your details

Forgot password? Click here to reset