A new benchmarking study of legal AI tools, which included August, Brackets, GC AI and SimpleDocs, has found that ‘several AI tools met or exceeded the baseline’ of expected performance for contract drafting as compared to human lawyers.
The study ‘Benchmarking Humans & AI in Contract Drafting’ by Legalbenchmarking.ai used a comparison group of experienced lawyers. Results (see below) were evaluated by three main dimensions: usefulness, reliability, and the overall support provided by the AI platform, e.g. how well does the solution support the drafting of a contract.
And here are the key findings:
‘AI tools matched and, in some cases, outperformed lawyers in producing reliable first drafts. Humans were reliable in 56.7% of tasks, but several AI tools met or exceeded this baseline.
The top AI tool marginally outperformed the top human. The top human lawyer produced a reliable first draft 70% of the time, whereas the top AI tool produced a reliable first draft 73.3% of the time.
Legal AI solutions surfaced material risks that lawyers missed entirely. In drafting scenarios with high enforceability or compliance risks, legal AI tools were far more likely to exercise legal judgment, raising explicit risk warnings in 83% of the outputs compared to 55% for general-purpose AI tools. Humans, by contrast, raised none.
Specialized legal AI tools did not meaningfully outperform general-purpose AI tools in both output reliability and usefulness. General-purpose AI solutions had a slight edge in output reliability (58.3% vs. 57.6%), while legal AI solutions scored marginally higher on output usefulness.
Platform Workflow Support is the key differentiator for specialized tools, not output performance. Two-thirds of the solutions we tested (66.7%) integrate into Microsoft Word, the primary work environment for lawyers, and most also provide context handling functionalities that most general-purpose AI tools lack.
Top performers in Band 1 for each dimension assessed include:
Output Reliability: Gemini 2.5 Pro, GPT-5, August, Brackets, GC AI and SimpleDocs
Output Usefulness: August, GC AI & Gemini 2.5 Pro
Platform Workflow Support: Brackets, SimpleDocs & GC AI.’
Note: some legal tech companies took part, but withdrew and/or didn’t wish to be named.
The overall conclusion from the team was that humans and AI tools both have strengths and weaknesses. AI tools do very well at ‘boilerplate’ contract drafting, whereas humans excel at the interpretation of legal instructions, and hence can handle more complex and subtle legal needs.
That said, the study authors concluded that AI tools can remove legal ‘drudge work’ and that inevitably ‘threatens some junior-level tasks’.
Moreover, on average across the study, humans alone scored 56.7% on reliability, and AI alone scored 57%….but, a combination of humans using the AI tools to assist their work scored 61.5% for reliability.
For Artificial Lawyer these results really sum up where we are: lawyers should not assume that AI cannot do their more standard work. But, for the best results combining human skills, e.g. subtle understanding of legal concepts, along with the productive strengths of AI, is – at least here – the optimal outcome for contracting.
You can see the full report here.
The team behind the study includes:
Anna Guo, Founder
Marc Astbury, CPO at Jenni AI
Arthur Souza Rodrigues, Securities and Technology Attorney
Sakshi Udeshi, AI Trust & Safety Expert, PhD in ML
Mohamed Al Mamari, In-house Counsel.
—
Legal Innovators Conferences in London and New York – November ’25
If you’d like to stay ahead of the legal AI curve then come along to Legal Innovators New York, Nov 19 + 20 and also, Legal Innovators UK – Nov 4 + 5 + 6, where the brightest minds will be sharing their insights on where we are now and where we are heading.
Legal Innovators UK arrives first, with: Law Firm Day on Nov 4th, then Inhouse Day, on the 5th, and then our new Litigation Day on the 6th.
Both events, as always, are organised by the Cosmonauts team!
Please get in contact with them if you’d like to take part.
Discover more from Artificial Lawyer
Subscribe to get the latest posts sent to your email.