構建全公司知識圖以顯著加速代理人工作效率

We gave an agent a knowledge graph of the whole firm and asked it five firm-wide questions. It answered them just as accurately, with:

73%

less time to answer

92%

fewer input tokens read

66%

fewer model calls

Building the graph took almost no model tokens. General drafting rules read matter records as they are already written, augmented only when LLM mapping is required.

The graph runs from the firm’s matters and documents down to individual clauses, and out to its clients, the law it relies on, its know-how, its email, its deadlines and its time ledgers.

How do you actually build the graph?

A lot of a firm’s knowledge lives in people’s heads, and they rarely have time to feed it into a graph. Some of it can be captured implicitly, from correspondence and from the links between documents. How much of the rest is within reach depends on whether the data a firm already records encodes it. In a law firm matter, for example, we already know exactly who the counterparty is and which documents are theirs.

We think about it in four levels. First, is the knowledge implicit in how the data is already structured? Second, can rules extract it? Third, where can a language model add the connections the rules cannot, including semantic ones? Fourth, which of those connections are most valuable, how can they be made cheaper, and which general shape of graph works best? Every answer depends on the domain, and each one needs experiments.

Most of the graph already exists

A firm’s systems already hold much of the graph. The practice management system knows each matter’s client, the other side, the partner and the team. The company register gives each company’s parent, and the time ledgers say who did what and whether it was billed. Loading these into one place makes these joins explicit and parsable. A sparse search will not immediately show that Meridian Grid Services belongs to Halcyon Group, but a network search will. The documents supply the detail that underlies these markers.

Rules first, then a model for what they miss

Rules are cheap, fast and predictable, and they can say exactly why they found something, but they miss whatever they were not written for. A model can read anything, but it costs money on every sentence and sometimes makes things up.

So the rules go first. When a sentence shows a sign of a fact they could not read, such as a month with no date they can parse or a liability limit with no figure, only that sentence goes to a small model. The model is asked just for the graph’s fields, and its answer is saved, so a rebuild pays only for new text.

To test this without our own drafting helping it, we had a different model rewrite all 245 documents, and the five filed emails that state a date, in six other drafting styles. We then built the graph in four ways, scoring it each time.

Sending the 141 sentences that were flagged to the model, under 2% of the room’s sentences, cost under 5 cents and brought the judgment and cap questions to 100%. A model reading every sentence cost 27 times as much and was not better overall. It found every deadline, but it read service credits of 15% of the monthly charges as liability caps, and on our own drafting it missed four deadlines that the rules found.

The hybrid’s misses have plain causes. Two deadlines are in court orders in the scanned-PDF style, where a line break left “by 4pm” at the end of a short numbered line. The rules took that line for a heading, so the date below it had no context, and a bare date did not count as a gap.

What it cost to build

The graph can be built from two kinds of source. The first is plain JSON files: registers of clients, matters and fee earners, with every document as text. JSON is platform-agnostic, so any firm can produce it by exporting from its CRM, practice management system or document store, and our test firm arrived this way. The second is a matter-centric platform such as Lexifina, which captures the same facts as people work: matter records, uploaded documents, email filed to a matter and the finance ledger. Nothing has to be exported, but some of the reading happens when documents are uploaded. The flowchart has a tab for each.

Five questions, with and without the graph

To see whether the graph helps an agent, we asked the same model the same five questions both with and without the graph. The agent could list, search and read every file in the data room. With the KG it could also call seven graph tools: search, open a node, check for conflicts, trace what relies on something, find deadlines, compare clause facts and total time and billing.

See some real agent runs side by side

Each cell shows the average without the graph, then with it. At the model’s list prices the average run cost about 17 cents without the graph and about 2 cents with it. The model pass on the rules’ gaps, at 5 cents, pays for itself on the first question. Counting the whole upload reading against the graph as well, it pays for itself after about 26 questions.

Without the graph, the agent still got nearly every answer right. It searched sensibly, read the company register, followed the citations and totalled the ledgers, and it took about four times as long and read about thirteen times as many tokens to do so. The one accuracy gap was the billing question. The agent without the graph had to add up dozens of time ledgers by hand, and in one run it missed a client whose unbilled time was just over the threshold.

What this does not show

The redrafted room tests drafting style, not everything a real data room holds. In this experiment, a model rewrote the wording but kept every fact, and its documents are clean text rather than scans with OCR errors, tables or tracked changes. Adversarial messy text would also make the agent without the graph miss more, so the accuracy gap on real records may be larger than the one we measured, but we have not tested that.

Is it worth it?

A knowledge graph is not a one-off cost. Every new or changed document has to be read again, and some of its sentences go to a model, so the token cost lasts as long as the firm keeps working. There is a logistics cost in the software: an adapter for each system the graph reads, rules to maintain, and a store and cache to rebuild whenever the records change. Each rebuild costs computing time, and the graph and the saved model answers cost storage.

So the question we keep asking is whether it is worth it. A side-by-side comparison of retrieval figures, like the one above, means little until it is weighed against what each approach costs to run, in the short term and over years. Rather than treating the graph as a new paradigm, we find two plainer questions more useful. How can our agent make the most of the structured data the firm already has? And what structure is left that is cheap enough, and useful enough, to create or join with a language model?

Clause metadata, at about $3.10 for this room, is most of the model spend on a matter-centric platform. The graph does not read it. So the questions are whether it should, and whether clause metadata is worth computing for the graph at all. We ran Lexifina’s clause metadata step, unchanged, over both versions of the room. We mapped its fields onto the graph’s: its references became cited cases, legislation and know-how, and its effective dates and the dates in its trigger and expiry conditions became dates, with their kind judged from the clause type and the conditions. Then we built the graph from the metadata alone, and from the rules with the metadata added.

On its own, clause metadata is a poor source for the graph. It has no field for the figure in a liability cap, so it found none of the ten caps. It found 7 of the 13 deadlines in both versions of the room, because its dates are effective dates and conditions rather than deadlines, and it is not run on email. Added to the rules, it did about as well as the model pass on the gaps. On the redrafted room it recovered the tribunal deadline that the gap pass also found, and one of the two scanned court orders that the gap pass missed, but neither of the two caps the gap pass recovered. It cost $3.15 against 5 cents.

So we would not compute clause metadata for the graph. Where it is already computed for review and search, merging it in costs nothing more and caught one deadline the gap pass missed. But the better fix for that deadline is a wider flagging rule, so that a bare date with no context goes to the model too.

Knock-on applications

The graph is most useful when something changes. If the Court of Appeal departs from an earlier decision, the firm needs to know which of its live documents rely on that earlier decision. Reliance is rarely direct. A clause may cite the earlier case, or it may cite a later case that follows or applies it, or it may say that it follows the firm’s own notices precedent, and the precedent’s drafting note is what relies on the case. We mark these edges as dependencies and walk them backwards from the case: through the cases that follow it, through the know-how that cites it and the clauses that follow that know-how, to the clauses, their documents, their matters and the partners responsible.

For a judgment that departs from an earlier case, the walk starts from the earlier case as well, because that is the law the firm’s work was built on.

Lexifina has no citator, so it can see which cases a document cites but not which later case follows or departs from which. Without those links, the new-judgment question above finds 10 of its 14 matters.

Original claimOn LocalLLaMA