In a recent conversation with Raghu Ramanathan, the newly appointed president of Thomson Reuters’ Legal Professionals segment, he expressed a strong belief in the need for open benchmarking on the performance of generative AI (Gen AI) in legal applications. Ramanathan, who took on his new role in February, highlighted the importance of transparent, collaboratively funded research to establish benchmarks and emphasized that more extensive comparative studies would benefit the entire legal industry.
Ramanathan’s comments follow the publication of a study by researchers at Stanford University that scrutinized AI tools used in legal research. The study revealed that Thomson Reuters’ AI Assisted Research for Westlaw Precision produced hallucinated results nearly a third of the time, a rate double that of its competitor LexisNexis’ Lexis+ AI. The findings highlighted the urgent need for rigorous, transparent benchmarking and public evaluations of these AI tools.
Despite the study’s surprising results, which contradicted the company’s internal tests and customer feedback, Ramanathan affirmed the necessity for more industry-wide benchmarks. He stated, “Personally, I think the more well-funded, well-conducted research there is—even multiple ones so we can compare and contrast—it’s better for the industry, it’s better for the customers.” For more details, you can visit the original interview here.
Ramanathan’s background includes nearly two decades at SAP, where he oversaw the growth of the platform division from a modest $250 million to a substantial $2.2 billion business. His experience in enterprise software, AI, and machine learning positions him uniquely to guide Thomson Reuters through its transformation from a content-oriented company to a content-enabled technology entity. He anticipates that robust customer success functions will be critical in helping legal professionals derive maximum value from AI solutions.
Interestingly, Ramanathan notes that smaller firms find the case for adopting AI assistants more straightforward compared to larger firms because of the direct bottom-line impact from reduced need for paralegals. Moreover, he emphasizes the critical role of the judiciary in shaping AI adoption within the legal profession, pointing to ongoing discussions with court systems as promising and proactive rather than merely defensive.
Ramanathan stands firm in his commitment to open benchmarking, advocating for industry consortia of respected universities, law firms, and legal associations to conduct these evaluations. This approach, he believes, will foster trust and ensure the continuous improvement of AI tools in law.