Will CLMS vendors survive in the age of AI everywhere?
The Challenge
The abilities of Large Language Models like ChatGPT have surpassed the analytical capabilities of any available Contract Lifecycle Management System (CLMS). Ubiquitous and low-cost access to this technology may eliminate the need for bespoke, enterprise-wide CLMS solutions (at least in their current form). Developments such as the embedding of OpenAI’s GPT-4 LLM into MS Office and the broader enterprise landscape will instantly put CLMS-type functionality into everyone’s hands. Why pay for anything else?
CLMS vendors must rapidly and radically up their game to survive. Already, more agile vendors (such as ContractPodAI with their Leah product) are scrambling to stay relevant by using their products to front the underlying, generic technology. Slower vendors could vanish almost overnight.
A little example
In just a few hours, I’ve created a basic contract analysis tool that can take a bundle of contract documents, compare and comment on the contracts against a set of ‘playbooks’, extract key contact data points, and answer ad-hoc questions about the contract. It understands concepts like framework agreements and orders/statements of work, and can even process amendments to provide prevailing terms.
I actually used GPT-4 to write most of the code and even utilized it to create and update the playbooks based on existing contracts and a few instructions. The analyser itself uses GPT3.5 (the latest available API), so it will only get better, with no extra work required on my part.
The game changers
So, essentially, what CLMS vendors have been promising (but not always delivering) for years is now achievable with an OpenAI account and a few lines of code. The problem with old-school CLMS AI was that ‘training’ any bespoke AI was done on relatively small data sets (a few thousand contracts at most), and specific extraction was often actually done mechanically (aka pattern matching) on a customer-by-customer basis.
The generic LLMs like GPT-4 are trained with, literally, millions of times more data (billions of documents). As they process this data, they ‘learn’ language patterns — spelling, grammar, sentence construction, document structure, even code, and then, on top of all this, specific content (including contract content and law).
The LLM architecture is designed to ‘complete’ the text of an input prompt — which may include the ‘chat’ so far or extracts from a document. It’s just like asking a person to look at some text and complete it. So, giving the LLM the contract text and then something like, “Given the above contract text and only the above contract, provide the effective date”, will result in the LLM returning the effective date.
No ‘old-school’ CLM AI without an LLM can compete with this level of language understanding.
But contracts are big, and not one document
A problem with an LLM is its context window. This refers to the number of tokens (a token is about 0.6 words) it can look back over to come up with its completion. This usually isn’t an issue when asking a question already in its training data — ‘Why is the sky blue?’ — it doesn’t need to consider any specific data, just what it knows about physics already. However, it is a problem when considering a contract (or agreement) that may consist of many documents, some of which could be hundreds of pages that the LLM hasn’t seen before.
But the technology is already ahead of this problem:
- Semantic encoding and search: A byproduct of LLM technology is the ability to embed the ‘meaning’ of a word, sentence, or any chunk of text into a set of numbers (an ‘embedding vector’ that ‘embeds meaning’). Embeddings from chunks of text can be compared (with a bit of math called cosine similarity), and a number representing their similarity can be produced. Something like +1 is identical, while -1 is not similar at all (not even opposites). So, you can chunk up contract text and create an embedding for each. When you want to answer a question like ‘Given all the amendments to the agreement, what are the prevailing terms for payment?’, you create an embedding for the question and compare this question embedding with all the embeddings for the contract chunks. The chunks that contain concepts like amendment or payment will have a better similarity score than the others. You can just take these chunks and ask the LLM to give the answer based on them. Thus, the LLM doesn’t need to ‘see’ the whole contract to answer the question. For those interested, this is effectively what Microsoft is doing to integrate GPT-4 into Office and enterprise data (its ‘knowledge graph’).
- LLM Summary: Another approach is to simply feed the contract into the LLM a chunk at a time and ask the LLM to list any information in this text relevant to the question like this: “From the above text, extract anything relevant to the question below. If nothing is relevant, just say UNKNOWN: Question: Given all the amendments to the agreement, what are the prevailing terms for payment?” Once you have asked this for all the documents, you can collect the results for every chunk and ask the LLM to analyze the extracted text and find and summarize an answer. This technique is less efficient (you have to process all the contract through the LLM, which is slow) but provides better results as it ‘sees’ the whole of the contract in a very specific context.
- Increasing LLM token count: A version of GPT-4 will have a token count of about 32,000 — about 50 pages of text — rather than the current common limit of 2,000. This makes the summary technique completely viable; a 100-page contract can be processed in two visits to the LLM rather than sixteen.
Low Barriers to Entry
The above techniques are easy to build into apps and web pages with just a few lines of code. Indeed, no-code/low-code libraries (such as Langchain) and websites are already emerging, making all this possible with very little technical knowledge required.
The concept of LLM ‘plug-ins’ is rapidly rolling out, allowing the LLM to operate directly on local data (contract documents, contract data spreadsheets/databases, contract reports, calendars, etc.) Soon, giving an LLM the instruction ‘Extract any trigger dates from all the contracts in this folder <link>, add them to this spreadsheet <link>, and this calendar <link>’ will be possible with no custom CLM software required.
To be fair, the instructions to the LLM will have to be carefully crafted to get the best results — some pre/postamble to the main instruction is probably required. Phrases like: “Given the above contract…”, “if you don’t have enough information…”, “give the result in the format…”, “only give the answer with no explanation…”, “compare the intent of the clause…”, “give a brief answer…”, “give a detailed answer…”, “give exact references to the location in the document…”, etc.
However, once these are known, they can easily be selected and incorporated automatically.
Cost
With currently advertised GPT-4 pricing, a simple bespoke app using the most inefficient way of processing the contract may cost $2 per contract to analyze. However, using more efficient techniques (chunking, embedding, and only using the most powerful model — GPT-4 — for the final analysis), the cost drops to a few tens of cents per contract.
If an enterprise has already paid for GPT-4 enabled Office functionality, you could use this for virtually no additional cost (though Microsoft pricing mechanisms have not yet been announced).
Where does this leave CLMS vendors?
Well, the only way to go is up the value stack and to provide near perfect user interfaces. Users will not accept any level of poor experience when interacting with a CLMS when they know the basic value, and some of the more bespoke functionality, will be provided elsewhere now.
One could become super ambitious and try to create a virtual corporate lawyer that automatically handles all contract management (from soup to nuts) for you. However, this would likely be a mistake. First, this will repeat the past ‘over promise, under deliver’ mistakes of the CLM industry. The cost of creating a general-purpose corporate lawyer would be huge, and probably only be a front for the LLM vendors who are investing billions in such things anyway. The value-add for an enterprise, over direct access to the technology, would be small and potentially difficult for an AI-savvy enterprise to justify.
The concept of co-pilots, rather than replacing people and roles with pure automation, is the way to go in the short/medium term. The ability to offload tedious tasks to the co-pilot, while the human adds higher-level value and becomes more productive, is where the value lies. Microsoft is doing this with Office, and GitHub is doing this with code.
Specialist co-pilots will orchestrate the underlying generic tools and provide relevant information when you need it — usually before being asked. They’ll provide a slick, natural language interface to enact the user’s will and ensure everything is updated and synchronized.
The technical skills required to produce these co-pilots will be relatively low. The value will be derived from truly understanding the user needs for these co-pilots and evolving them to match.
In other words, good old-fashioned user-centric agile product development will trump technology and marketing-led promises to enterprise executives.
Integration into wider legal-tech (matter management, e-discovery, legal hold management, compliance management/reporting, etc.) to create a complete corporate legal solution may be a way to go. But instead of clunky, multi-product merger and integration — as legal suites have grown in the past — the underlying concepts of the co-pilot and generic tools can be used, practically from scratch.
Scale, implementation and, service
In the enterprise world, the ability to realize business change and deliver, then manage the supporting technical solutions at scale, is what organizations need. This, along with the ongoing support, improvement, and evolution of solutions (not just a twice-a-year upgrade), is what will differentiate bespoke business solutions from generic tools, not bespoke technology. It’ll be interesting to see where enterprises source these abilities from in the future. Will they need legal tech vendors at all, or just a set of SMEs?
