Defending AI in Legal Practice: Alicia Hawley’s eDiscovery Insights

Alicia Hawley, Of Counsel at K&L Gates, has over 20 years of experience in law. In 2020, she was an attorney on Livingston v. City of Chicago—a landmark case approving the use of technology-assisted review (TAR) in conjunction with search terms and without any involvement in the training and validation by the requesting party.

I recently sat down with Alicia to ask for her thoughts on that case and how this moment’s use of AI in legal practice echoes those early years of implementing TAR.

Her insights are telling. Compared to TAR, generative AI introduces novel efficiencies and user friendliness that have helped its popularity spread like wildfire. Though Alicia notes judicial approval of generative AI will take expert advocacy, that moment feels inevitable.

As the City of Chicago’s attorney in the Livingston case, can you tell us about the discovery disputes and how AI played a role in those disputes?

Alica Hawley: Discovery was contentious. We spent almost two years negotiating the ESI protocol and search terms and, when ready to start reviewing documents, we informed plaintiffs that we planned to use TAR 2.0—also known as active learning—to do so. They didn’t like that plan. Their objections were twofold:

• They didn’t think we should be able to use TAR and search terms together.
• They thought that, if we did use TAR, they should be involved in sampling, testing, training, and creating a jointly negotiated validation protocol.

On the second point, there was a disconnect in the parties’ positions. Plaintiffs’ arguments were grounded in TAR 1.0, which requires seed sets and is much more iterative, but the city intended to use TAR 2.0, which offers prioritized review and more rapid machine learning. So I explained to the court that, instead of having to train the model very intentionally with selected documents, the 2.0 model can also learn iteratively from each coding decision made by reviewers. I also explained that we were using TAR to review documents post-culling, not to cull documents from a larger set.

Plaintiffs also expected us to send them documents so they could decide on responsiveness as part of the validation, which of course we did not agree to.

The plaintiffs argued that attorney reviewers could improperly train the AI with incorrect determinations. What were your thoughts on that?

Alica Hawley: I hear this a lot, actually—“garbage in, garbage out.” That’s true. But the idea that a whole contingency of reviewers would make so many incorrect decisions that they’d steer the model in the wrong direction? There’s not really any science behind that. It isn’t an unusual argument; they wanted to be involved in the training because if the tool isn’t trained correctly, it’s not going to be correct. But that’s not where the law is regarding responsiveness reviews.

We hear the same argument today when it comes to legal AI tools like Relativity aiR for Review. As an attorney, you have an obligation to give directions to the model to effectively locate responsive documents—opposing counsel doesn’t need to be involved in that process.

What strategy did you use to help the court understand the technology?

Alica Hawley: I never want to assume what anybody knows about technology; I don’t assume they are uninformed, but technical understanding often goes beyond that. I wanted the judge to have a concrete understanding of the mechanics of active learning. During the hearing, I provided a detailed tutorial on how Relativity’s active learning works, citing Relativity resources. Then, I explained why we thought it was defensible to use it in conjunction with the search terms the parties had spent two years negotiating.

In the order, the court agreed with your position and cited Sedona Principle 6. Could you explain that?

Alica Hawley: Sedona Principle 6 states that responding parties are best situated to determine the methodology for meeting their production obligations. The judge cited it in response to this idea that plaintiffs should be involved in the training, noting that the responding party knows the best way to find responsive documents in their own universe.

I think he even made a comment along the lines of, “Until something goes wrong, you don’t get to tell them how to do this.”

Generative AI—including aiR for Review—now enables humans to use AI to predict responsiveness, validating its results in much the same way as TAR. What are your thoughts on where we are today versus five years ago?

Alica Hawley: One of the sticking points with TAR was a lack of understanding: people were resistant, thinking it was making coding decisions without sufficient human guidance.

Now, we have tools making coding decisions on documents—exactly what people were freaked out about before. But, of course, it’s supervised and validated by humans and follows human direction. Most generative AI tools for discovery provide highlighted citations to assist in verification. aiR for Review also provides rationales, which explain the “why” behind each prediction.

Lawyers can struggle to accept the idea that a machine can augment their decisions and work. But I think sound technology will always end up being adopted by the masses.

If the court had this same case today, but with generative AI, what do you think would be important in that decision?

Alica Hawley: We’re going to see that case, and the outcome will likely depend on how the tools are described, positioned, and defended and how the advocating party is using them. That was what was great about Livingston: it happened with this team of sophisticated eDiscovery professionals using TAR defensibly. I hope those first cases involving generative AI will be similarly situated; it will be critical to advancing its use and having it acknowledged and supported by the judiciary.