Field note / Evaluation
How should you test legal document automation before rollout?
Write the expected document by hand first, then try to break the automation with adverse cases: a missing fact, conflicting instructions, a stale master, a blank optional section, renumbered cross-references, defined-term drift, a jurisdiction variant and an ambiguous date. Every case gets an expected result, a recorded run and a named person who signs it off.
What goes wrong when automation reaches real matters?
The cases nobody tried. A demo uses complete, consistent facts; real matters arrive with gaps, contradictions and old copies. The output looks finished, so the error travels.
- Manual fixes after generation. “In many cases, I still have to manually edit the final document after the merge,” wrote one person who prepares contracts and letters. “Because there are so many details to check, it's easy to miss something or introduce inconsistencies” (r/legaltech, June 2026).
- No agreed scope. Asked what goes wrong when firms hire someone to automate templates, a commenter answered, “The way these things go wrong is by not defining goals and a clear scope of work” (r/LawFirm, September 2024).
- Testing takes time. “So many home-grown systems take hours and hours and hours to program, test and debug. That goes for individual templates, too” (r/legaltech, January 2026).
- It never ends. “Nothing will completely remove the need for maintenance and testing when it comes to conditional document automation” (r/LawFirm, May 2026).
What belongs in a pre-rollout test set?
One complete baseline and one adverse case for each way real instructions go wrong. The baseline proves the automation can produce the right document. The adverse cases prove it refuses, or flags, the wrong ones.
- Write the expected result before running anything. For the baseline, that is a hand-checked copy of the finished document. For an adverse case it is either a specific difference from the baseline or stop: no draft.
- Change one thing per case. Start every case from a fresh copy of the baseline facts, so a failure has one cause.
- Test the document, not the screen. Open the generated file, update its fields, and compare it to the expected copy word by word, including headings, numbering and footers.
- Keep the set. Rerun it after every change to wording, rules or the master, as the template governance note and the change log in the conditional-clauses note describe.
The hub note's eight package cases test whether three outputs agree with one worksheet. This note tests one document's internal structure: what happens to numbering, defined terms and variants when the facts change.
Worked example: eight adverse cases for an engagement letter
Fictional set-up. Master EL-MASTER v4, approved by the template owner, produces an engagement letter from an intake form. Clause 4 (conflict waiver) and clause 6 (scope exclusions, with Schedule 2) are optional. Clause 9 refers to clause 7. Clause 11 has a Jurisdiction A and a Jurisdiction B variant. The footer prints the master version. Baseline E1 is a company client, Jurisdiction A, both optional clauses in, every field filled, checked by hand.
| Case | Change from E1 | Expected result | Who decides |
|---|---|---|---|
| 1. Missing fact | Responsible lawyer left blank | Stop: no draft; the run names the empty field | Intake owner supplies it; rerun |
| 2. Conflicting instructions | Intake says individual client; the conflict-check record says company | Stop: no draft; both sources and values reported | Responsible lawyer; never newest or majority |
| 3. Stale master | v3 selected for generation after v4 was approved | Stop: no draft; run reports v3 superseded, v4 approved | Template owner; regenerate from v4 |
| 4. Blank optional | No scope exclusions selected | Clause 6, its heading and every mention of Schedule 2 gone; no empty heading or stray “see Schedule 2”; later clauses renumber and clause 8's reference reads “clause 6” | Template owner approved this expectation in advance |
| 5. Renumbering | Conflict waiver (clause 4) omitted | Later clauses renumber; clause 8's reference now reads “clause 6” and points to the old clause 7; no broken or stale reference after fields update | Template owner; checker compares every cross-reference |
| 6. Defined-term drift | Seeded defect: optional clause 6 says “the Company” where the master defines “the Client” | Flag before rollout: each defined term defined once and used the same way throughout | Template owner corrects the wording; all cases rerun |
| 7. Jurisdiction variant | Governing law set to Jurisdiction B | Only the B text of clause 11 appears; no A wording, no leftover A defined terms | Responsible lawyer for each jurisdiction approves its variant |
| 8. Ambiguous date | Start date typed as 03/02/2027 | Stop unless the form fixes the format; the letter spells out the month | Intake owner confirms; template owner sets one date style |
Reading. Half the cases should stop the draft, not produce a better one: missing, conflicting, stale and ambiguous inputs are questions for a person. Cases 4, 5 and 7 are where a finished-looking letter can still be wrong, because headings, numbers and references move when text is removed. Case 6 is found by reading, not by running, so it belongs in the checklist too. Amounts deserve the same treatment as dates: one approved format, and words and figures that agree. Keep the table: it is the regression set for every later change.
Have a test set like this and want a second opinion? Discuss one workflow in a short fit conversation; no documents needed.
What should you check in Word yourself?
Stale numbers until fields are updated. Word keeps cross-references as fields, and some fields need a manual update, so update fields before you compare a test output.
- References are fields. “Cross-references are inserted into your document as fields,” and “Word uses a hidden bookmark to track an item that has a cross-reference.”
- Updates can be manual. “Sometimes you need to manually trigger an update of the information in fields like those used in a table of contents, headers and footers, cross-references, bookmarks, and formulas in tables.” Word can also be set to “update fields automatically before printing.”
- Removed targets. “Sometimes a field is not updated because the item the field refers to no longer exists.” That is case 4 going wrong; case 5 is the stale-number risk above.
- Inserted text. “If you press Enter at the beginning of the referenced item, the bookmark can expand to include the new item.” Then a reference “might display or link to more content than intended after fields are updated.”
A document-automation product or a template engine may handle numbering its own way. Whatever generates the letter, cases 4 and 5 test the output file, not the tool's description of it.
What should each test run record?
Enough for someone else to repeat the run and reach the same verdict. One block per case per run:
TEST-RUN RECORD — one block per case per run master / version = __________ rules version = ______ expected set = ______ case = __ change from baseline = ________________________________ expected (written before the run) = ______________________________ actual: draft produced? Y / N stop message = ____________________ fields updated before comparing? Y / N differences from expected (clause, text, numbering, footer) = ______ verdict: PASS / FAIL / STOP AS EXPECTED / STOP NOT EXPECTED prepared by = ________ checked by (not the preparer) = ________ decision owner = ________ date = ______ fix needed? ______ rerun of all cases after fix? Y / N result = ____
A pass is a match with the expected result, including an expected stop. A draft where a stop was expected is a fail, however good it looks. Record the master and rules versions on every run, so an old pass is never mistaken for a current one; the automate-first note on forked updates covers how versions wander.
Who signs off before rollout?
The lawyers who own the wording and the rules, with a checker who did not prepare the run. Engineering can show that the automation does what the expectations say; it cannot decide what the expectations should be.
- Template owner (a lawyer): approves the master, each rule and each expected result, including the stop cases.
- Responsible lawyer per variant: approves the jurisdiction or practice-area text they are accountable for (case 7).
- Preparer and independent checker: the preparer runs the cases; someone else compares outputs to the expected set and records the verdicts.
- System owner (operations or IT): confirms which master and rules version is live and who can change them.
- The reviewing lawyer on every matter: testing does not replace review. One commenter observed that associates using automated templates “learn to click one button instead of the other, and do not spend much time looking at the alternative language” (r/LawFirm, May 2026).
Should you roll out, narrow the scope or stop?
Roll out only what passed every case, narrow to the part that did, and stop if nobody will own the expectations. The decision belongs to the firm, not to whoever built the automation.
| Result | Decision |
|---|---|
| All cases pass, versions recorded, sign-offs named | Roll out to the people and matters the test covered; keep the set for every change |
| Baseline passes; some adverse cases fail | Narrow: remove the failing options or variants and keep those manual until fixed and rerun |
| Stop cases produce drafts | Do not roll out: the tool cannot refuse bad input yet |
| No one will approve expected results | Stop: fix ownership first (naming template owners) |
When buying is better. If a product already provides required fields, version control, an audit trail and maintained content for your practice area, those are checks you do not have to build. Test it on the same eight cases before choosing; the hub's build, buy or neither table and the three paths help frame the choice. Either way, as one r/LawFirm commenter who builds these systems put it, “You're not going to get away from upkeep” (June 2026).
What should you do first?
Pick one document that is about to be automated, or already is. Have its owner approve a hand-checked baseline. Write the eight cases above against it, with an expected result and a decision owner for each, before running anything. Run them, record each run, and roll out only what passed. If it is a whole package, use the hub note as well.
A document automation that needs testing before rollout is the kind of workflow we help evaluate and connect. Engineering scope and support are agreed separately. The responsible professional keeps legal review and decisions.
Please do not send confidential documents or client records. A conversation can start with the document, its optional sections and variants, and the tools you already use.
Related field note: Commercial lease packages: where does repeated data entry come from?
Sources and their limits
Public research cutoff: 22 October 2026. Practitioner quotations are short excerpts from public, pseudonymous forum posts. Identities and roles are unverified, and some commenters sell tools or services. They show how the problem is described, not how common it is. Microsoft pages are cited for how Word fields and cross-references behave, not for any automation product. The engagement letter, clause numbers, jurisdictions, test cases, worksheet and decision table are original synthetic material and have not been run against a real system.
- Reddit threads (accessed 22 October 2026): r/legaltech standard for contract generation and document automation (6 June 2026); r/LawFirm hiring someone for document automation: how did it go? (6 September 2024); r/legaltech worth investing time in document automation? (25 January 2026); r/LawFirm document automation / drafting software (31 May 2026). Comment dates may differ from post dates.
- Microsoft (accessed 22 October 2026): Create a cross-reference; Update fields.
For practice context, see how Stratum Relay works. This organizational engineering attribution is not a statement that a lawyer, client or named individual has reviewed or endorsed the note.