Legal Tech Benchmark Claims Test Bar Advertising Rules

A new front has opened in legal marketing. Performance benchmarks, evaluation tools originally built to measure AI systems, are supplanting traditional attorney rating services. Recently, a law firm used a self-created benchmark to argue that its attorneys outperform artificial intelligence.

The question, writes My Shingle’s Carolyn Elefant, is how advertising ethics rules apply when a law firm is both the test designer and the subject being praised.

Benchmarks are structured datasets used to measure how well an AI model handles defined tasks. Legal work is difficult to score this way because much of it involves subjective judgment where reasonable practitioners can disagree.

Despite that difficulty, heavy investment in legal AI has pushed providers to create competing benchmarks that include detailed practice-area tests and standards of reliability.

The article examines Crosby.AI, an AI-native law firm that built its own benchmark, with help from an AI research partner, to test contract redlining performance.

Crosby’s attorneys wrote the evaluation criteria and scored the results against those same criteria. The AI judges concluded that current AI models fall short of Crosby’s human lawyers.

The article questions whether, under bar ethics rules, this constitutes ordinary tech marketing, or a comparative advertising claim. Crosby acts as both rule-maker and winner in its own contest, an arrangement that existing state bar guidance treats skeptically.

Firms exploring AI-driven marketing should review their claims against disclosure obligations and existing rules governing comparative advertising, particularly when self-authored methodology underlies the results. Counsel advising legal tech ventures should scrutinize whether claimed objectivity actually meets the verifiable, consistently applied standard regulators expect.

Lawyers structuring AI-related client communications should weigh transparency about testing methodology as a meaningful risk-mitigation tool. Firms operating across multiple states should also account for jurisdictional variation in how aggressively comparative claims are policed.