Generative AI Successfully Passes Professional Conduct Exam for Lawyers

Generative AI has once again demonstrated its capabilities by successfully passing a simulation of the Multistate Professional Responsibility Examination (MPRE), a critical test assessing the knowledge of professional conduct rules for prospective lawyers in most U.S. jurisdictions, according to a LawNext article by Bob Ambrogi.

LegalOn Technologies conducted the test to explore the potential of AI in supporting lawyers and evaluating its efficacy in ethical decision-making. The test evaluated four AI models: ChatGPT-3.5, ChatGPT-4, Anthropic’s Claude 2, and Google’s PaLM 2 Bison. Among them, GPT-4 emerged as the top performer, answering 74% of questions correctly, which was approximately 6% higher than the average human test-taker. Claude 2 achieved an accuracy rate of 67%, while GPT-3.5 and PaLM 2 demonstrated accuracies of 49% and 42%, respectively.

Both GPT-4 and Claude 2 surpassed the passing threshold (estimated to range between 56-64%) for the MPRE in all states where it is mandatory. The simulated exam questions, designed by law professor Dru Stevenson, covered various topics with a focus on professional responsibility. It is noteworthy that the AI models were evaluated using a “zero-shot” approach, indicating no prior training specifically related to legal ethics.

The research highlighted GPT-4’s proficiency, emphasizing its varying performance across different subject areas. While excelling in questions related to conflicts of interest and client relationships, GPT-4 showed less proficiency in topics such as the safekeeping of funds.

The study emphasizes that while AI can pass the legal ethics exam, the ultimate responsibility for ethical decisions will remain with legal professionals. However, it underscores the potential for technology to assist the legal community in consistently meeting high ethical standards, marking a significant turning point for both legal technology and the practice of law.