Vals AI has announced an open call for legal technology vendors to participate in its upcoming benchmarking studies focused on legal AI tools, particularly targeting U.S. legal research tools. This initiative follows the release of the Vals Legal AI Report (VLAIR) in February, which was a pioneering effort to benchmark legal AI tools against a lawyer control group using tasks common in Am Law 100 firms. However, that initial report excluded legal research products, which will now be a significant focus.
Tara Waters, the project lead, explained that the decision to separate legal research from the broader study arose from a desire to ensure comprehensive vendor involvement and a diverse array of assessment questions. “We are collecting 200 Q&A from the law firms. As before, we will have a human baseline,” Waters noted. The initiative comes after feedback from vendors unaware of their earlier study, with Vals AI now encouraging diverse participation from companies with generative AI legal research solutions.
In this upcoming legal research assessment, responses from both AI tools and human lawyers will be evaluated by human graders, comprising a mix of law librarians and independent legal reviewers. The evaluation criteria will include accuracy, authoritativeness, and appropriateness of the tool’s responses, with a weighted assessment methodology.
Legal technology companies wishing to have their products evaluated in either the current or future studies can express their interest by filling out this Vendor Interest Form. Additionally, law firms, solo practitioners, and in-house legal teams are invited to join the study’s benchmarking consortium and can do so by completing the Law Firm Interest Form.
Vals AI’s study has been developed collaboratively with Legaltech Hub and a consortium of renowned law firms, including Reed Smith, Fisher Phillips, McDermott Will & Emery, and Ogletree Deakins. Further details can be found in the original article from LawNext.