From Hype to Reality: Solving the Challenge of Legal AI Hallucinations

Legal AI hallucinations are a bigger problem than you might realize. Here’s a concerning statistic: Inaccuracies and falsehoods occurred up to 33% of the time when Stanford University tested legal artificial intelligence tools over 200 legal queries.

These AI hallucinations occur when AI generates plausible-sounding but factually or legally incorrect information. For example, an AI might confidently cite a non-existent case, misinterpret a statute’s application, or falsely assert that a cited source supports the AI’s answer.

While AI can speed up time-intensive legal tasks, there is a “jagged frontier” that separates tasks at which AI is more successful from those at which it is less. Which tasks are AI best-suited for, and how can we harness AI’s potential while ensuring accuracy for legal work?

Legal research may be beyond that frontier, for now. The Stanford researchers note that legal research requires AI “to synthesize facts, holdings, and rules from different pieces of text while keeping the appropriate legal context in mind.”

There are, however, a set of legal tasks that feel to many legal teams like the low-hanging fruit: contract review, document summarization, answering common questions from the business, brainstorming, and first drafts of emails and some legal documents.

AI contract review software is a useful case study to understand why AI can work today, but also what vendors must do to ensure AI is accurate and “professional grade” for lawyers.

Why AI for Contract Review Works

Contract review stands out as a great use case for AI, offering distinct advantages over less structured tasks:

1. Grounded in defined standards: AI can be effectively trained on specific contract types and clauses, reducing the risk of straying into unfamiliar and constantly-evolving territory (such as legal research).
2. Easier to verify: Unlike broad legal research which requires synthesizing across many documents, AI in contract review focuses on analyzing an individual document. This makes it easier to pinpoint source language and enables lawyers to efficiently inspect and validate AI’s work.
3. Focused objectives: Contract review often involves identifying standard clauses and deviations, tasks that align well with AI’s pattern recognition capabilities.

The Advantages of Professional Grade Legal AI

While contract review is amenable to AI, the best AI for this task – meaning AI that is accurate, consistent, and trustworthy – must be purpose-built and guided by legal expertise at every stage:

• AI trained and tested by attorneys: Ensures results are accurate, reliable, and designed for lawyers.
• Defined issue-spotting and playbooks: Ensures AI spots major and minor issues like an attorney, with thorough, pre-built contract review playbooks.
• Integration of legal knowledge: Reduces hallucinations by connecting AI to trustworthy content for offering suggestions and drafting language.
• Human-in-the-loop processes: Ensures end-users are involved in critical-decision making and final approvals.
• Transparency and explainability: Provides explanations for recommendations, allowing legal professionals to inspect and verify the reasoning.

Here’s how these elements come together to enhance the contract review process:

• First-pass review: The AI quickly analyzes contracts, identifying key issues, potential risks, and deviations from standard language.
• Customized playbooks: Legal teams can create and refine AI playbooks that align with their organization’s risk tolerances and preferred contract language.
• Intelligent redlining: AI suggests appropriate redlines, based on the playbook, speeding up the redlining process.
• Clear explanations: For flagged issues or suggested changes, AI provides context and reasoning, empowering attorneys to make informed decisions quickly.

The legal industry has always been cautious about adopting new technologies, and for good reason. By focusing on well-suited tasks like contract review, and knowing what to look for in AI products, lawyers can help themselves without compromising on quality or ethics.

The closer you get to these AI tools the clearer it becomes that legal AI isn’t replacing lawyers. It’s creating tools that amplify legal expertise, allowing attorneys to work more efficiently and focus on high-value, strategic aspects of their roles.

While some legal tasks remain beyond the “jagged frontier” of what legal AI can do today, there are big wins available right now in other areas, including contract review. In-house counsel are helping themselves and their teams by identifying this low-hanging fruit, preventing legal AI hallucinations, and choosing AI that is lawyer-led, consistent, and trustworthy.