By Pedram Abrari, CTO, Pramata.
This is the first article in a three-part series exploring the major technical challenges that prevent LLMs and AI alone from being effective for accurate contract intelligence, and how to solve those challenges.
Nearly three years after ChatGPT thrust the concept of generative AI into public consciousness, the number of use cases – both serious and silly – have increased by the hour. While it’s fun to see what your pet would look like as a human or create your own action figure, the real promise of large language models (LLMs) and generative AI exists in solving large, previously unsolvable business problems.
For attorneys who deal with hundreds, thousands, or even more contracts in their work, generative AI’s appeal is obvious. Imagine being able to instantly analyze thousands of contracts, extract key terms, and identify risks with the same ease as asking ChatGPT a simple question.
But here’s the reality check that legal teams and their technical counterparts are quickly discovering: Contract AI is not as simple as providing contracts to your favorite LLM and hitting ‘go.’
While popular LLMs seem remarkably capable of producing accurate outputs for many tasks, they consistently fall short when applied to contracts. The reason isn’t a lack of AI sophistication—it’s that contracts present a unique set of engineering challenges that extend far beyond typical natural language processing tasks. In this three-part series, I’ll explain why contracts are uniquely challenging for AI to deal with and how the right technological foundation solves these problems.
Not just plain text: Why contracts are different
Contracts aren’t merely text documents. They are highly structured, legally binding instruments with complex internal relationships, specialized terminology, and intricate interdependencies. These distinct characteristics create specific computational constraints that AI must address through specialized technical and architectural approaches.
When legal teams load contracts into general-purpose LLMs and ask questions about their contents, the results may appear effective – initially. ChatGPT, for example, is surprisingly good at analyzing one contract in isolation. But, when used for larger contracts and sets of related contracts, teams quickly encounter the limitations. We have identified nine major technical challenges that must be addressed in order to achieve true value from Contract AI and we’ll begin with the first three, along with how Pramata has addressed them to achieve unprecedented levels of accuracy in Contract AI.
Contract AI challenge 1: Security risks of traditional RAG systems
Before addressing operational challenges, organizations must confront a fundamental security vulnerability that can stop Contract AI adoption before it starts.
Traditional Retrieval-Augmented Generation (RAG) systems, used by most Contract AI vendors, follow this workflow:
1. Decrypt your contract documents
2. Convert text to numerical vectors (embeddings)
3. Store vectors in a database for semantic search
4. Retrieve similar content based on user queries
5. Generate AI responses using retrieved context
The critical security vulnerability exists in step two because converting text to numerical vectors requires your sensitive contract data to be processed and stored in unencrypted form. This creates an unacceptable security exposure for contracts containing pricing information, confidential arrangements, and strategic partnerships.
Pramata’s solution: RAG-E technology
Pramata’s RAG-E technology maintains encryption throughout the pipeline. Instead of semantic search through decrypted data, RAG-E uses structured Contract Intelligence and Pramata Prompt Language variables to enable precise, encrypted retrieval. This Context Engineering approach means your contracts stay encrypted, eliminating the vector database vulnerability while meeting enterprise security requirements.
Contract AI challenge 2: Garbage in-garbage out
The saying ‘garbage in, garbage out’ has never been more relevant than with Contract AI. If your contract storage is messy, cluttered, distributed across multiple systems, full of duplicates and non-contract files, and full of PDFs that can’t be digitally read, there’s no chance of achieving value from Contract AI.
Accuracy is key to working with Contract AI, yet even Open AI’s most advanced LLM, GPT-5, recognizes that artificial intelligence alone can’t get to the level of accuracy attorneys need if they want to rely on Contract AI for critical business decisions.
When you’re aiming for accuracy, especially for multi-contract analysis, these specific issues undermine even the best AI’s performance:
Duplicates, non-contract files and versioning confusion
Unreadable documents
Disorganized contract families
Fragmented storage/missing documents
These factors, and others, collectively form the basis of poor data quality—that is, ‘garbage in.’ Unfortunately, no AI can overcome the issue of ‘garbage in’ to produce quality outputs when it’s not given quality inputs.
Pramata’s solution: Patent-pending cleansing and organizing technology
Pramata addresses the ‘garbage in, garbage out’ challenge through several patent-pending innovations:
TrueDoc OCR: Converts even the most unreadable documents into digitally processable text
Files Cleanse: Removes duplicates and non-contract files from your document repository
Documents Organize: Creates clear document hierarchies based on order of precedence, tracks document history and changes, and establishes rules-based relationships between master agreements and amendments
Extract & Enrich: Pre-tags key contract clauses for efficient processing at scale
TrueCheck QA: Provides human-in-the-loop quality assurance for AI-generated results which is how Pramata users achieve 99%+ accuracy
A combination of sophisticated AI, multiple levels of verification, and human-in-the-loop expert QA is the only way to get to the 99%+ level of accuracy that lawyers need to trust what they get from AI.
Contract AI challenge 3: Context window & message token limit management
One of the first downsides AI users noticed as they started experimenting with Chat-GPT was the occurrence of AI ‘hallucinations.’ And while they are laughable in less serious use-cases, these types of ‘confident mistakes’ that generative AI is known for aren’t acceptable in the world of Contract AI.
The reasons why AI hallucinates have a lot to do with context window and message token limits. These are terms that describe the AI’s capacity to handle more and more information and continue to create accurate results. Essentially, this is the AI’s working memory.
As you interact with a large language model, you begin to exhaust the AI’s context window and hit its token limits: Unfortunately, you rarely know that you’re approaching these limits until it’s too late. This means the accuracy of the AI’s outputs degrades until it’s outright hallucinating, and it does that without warning the user that answers are becoming increasingly unreliable.
It should go without saying that this is a huge problem for attorneys and others trying to use AI with legal documents!
Pramata’s solution: Context Engineering
Effective context management requires sophisticated data structuring and retrieval systems that can intelligently identify and deliver the most relevant contract information for each specific query. Pramata’s Context Engineering approach addresses token limitations through these patent-pending technologies:
Extract & Enrich: Pre-tags key contract clauses so the system doesn’t need to rediscover clause types repeatedly when analyzing thousands of contracts
Contract Families: Maintains contract relationship precedence, enabling AI to determine effective terms across multiple related contracts
Pramata Prompt Language: Uses expert-designed prompts with specialized variables like {{master_agreement}} and {{price_escalator}} to target precise, relevant parts of contracts instead of requiring the AI to process every word
Through these innovations, Pramata allows an LLM to preserve its context window and avoid hitting token limits, which results in Contract AI outputs you can trust.
These three technical challenges are just the beginning
We’ve barely scratched the surface of the technical challenges that come with Contract AI, and why loading hundreds of contracts into an LLM alone won’t produce the results attorneys and legal teams require.
Make sure to come back for parts two and three of this series, or download Pramata’s entire whitepaper ‘9 Major Technical Challenges that Come With Using Generative AI for Contracts’ now.
—
[ This is a sponsored thought leadership article by Pramata for Artificial Lawyer. ]
Discover more from Artificial Lawyer
Subscribe to get the latest posts sent to your email.