Unproven Claims of LLMs in eDiscovery Spark Concern

Unproven Claims of LLMs in eDiscovery Spark Concern

In an eDiscovery Today article, Doug Austin discusses a new paper by law professors Maura R. Grossman, Gordon V. Cormack, and Jason R. Baron, which critically examines the use of large language models (LLMs) in eDiscovery. The authors draw an analogy to Hans Christian Andersen’s tale of the emperor’s new clothes, suggesting that many claims about LLMs in eDiscovery may be overly optimistic or unproven. 

In the past year, many legal professionals and vendors have said that LLMs could soon replace traditional keyword searches or technology-assisted review (TAR) methods. However, the paper’s authors note that no empirical evidence supports the idea that LLMs are as effective as TAR in identifying electronically stored information (ESI).

The paper calls for thorough empirical studies and statistically sound validation protocols similar to those used in landmark cases like Da Silva Moore v. Publicis Groupe & MSL Group. Without such validation, the authors caution against the widespread adoption of LLMs for legal research and discovery, noting that the variability in responses from LLMs depends heavily on the skill of the “prompt engineer.”

Austin notes the concerning trend of lawyers embracing LLMs without proper validation, possibly because of their familiarity with generative AI tools like ChatGPT, and warns against treating LLMs as a “quick fix” for complex legal tasks without rigorous testing, echoing the caution raised in the paper.