⚡ Powered by Finn · Day 179 of 365
179

Designing an Auditable AI Agent

The audit screen I opened this morning had a paragraph at the top explaining what the screen was for. Under it, a note on how to read the middle section. Under that, three things I still had questions about. I read the whole page, worked out that the person it was built for would read none of it, and deleted every word. What is left is three tabs, and three tabs are what an auditable AI agent owes a client: the pack you signed, what we posted, what is still open.

I have not posted in this 365 day build since the fourteenth of August. Thirty nine days. The client work and a summer holiday spent surfing with my son and wife were the cause. The BIP went quiet while the controller was being built. After you miss two days you stop counting. I'm not likely back to daily BIP writing, I wasn't seeing any justification for it, but as soon as I see green shoots or some interesting topics come up, I'll be back.

Where the controller is now

The AI financial controller closed August. Seven packs, four entities and three company cards, every one signed by a person before anything went near the ledger. Underneath it is the same machine I wrote up last month: deterministic scripts doing the matching and written rules doing the judgement, with a human signature between the proposal and the post.

What changed is who is looking at it. The client brought in an audit specialist in late August, ex big four audit, and he now runs the reconciliation the finance director used to run. They told me on our last call that he will own the reporting from here, and then, unprompted, that they had been the bottleneck up to now. For months I blamed the file format and the access.

In July I wrote that the front end would be a chat window inside the enterprise tool their finance team already trusted. Three boxes that they trust and will use. What shipped first is a screen, because an auditor wants to see the whole month at once and then ask about one line of it.

Three tabs

Tab one is the pack he signed. The decisions file the reconciliation UI wrote while he clicked through his exceptions, the same document he downloaded to his own machine that afternoon. His notes, in the order he made them. We fold the journals and the allocations in underneath each entity and name every general ledger code inline, so a line reads 5010 with Employee Salaries written next to it. Those codes are read out of the poster's own map at build time. There is no second copy of that map anywhere, which means the screen cannot name one account while the system writes to another.

That last part took two goes to get right. It would have worked all year and then quietly stopped being true. That won't happen now.

Tab two is what we posted. Transaction ids, amounts, accounts, dates. No commentary. Every allocated invoice reads zero except the one that was part paid, and that one shows the exact residual he wrote on his own pack, which is the best evidence I have that the machine did what he asked.

Tab three is what is left. A handful of corrections that need a person in the accounting UI, and the button he clicks to mark the account reconciled. I am not automating that button. He should be the one who presses it.

Where all the words went

Everything I deleted off the screen went into the agent window beside it. The explanations and every open question so the auditor, who expects to be able to use AI these days, can now work through issues with an agent in the UI window. Safe and boring.

The agent reads the record for that pack and answers off it. It is read only by decision. It has no write path to the accounting API at all. If it works out tomorrow that a line belongs in a different account it can say so, and nothing happens until a person types it. Today it is deterministic retrieval over one entity, which is a polite way of saying it looks things up. Whether free text gets a model behind it, with the record as the only thing it is allowed to read and still no writes, is a call I have not made.

Green, amber, red

Seven packs, seven records, one page of tiles in the reconciliation UI's own colours. Green means nothing is needed from him, amber means his attention, red means blocked. August came out five green, one amber, one red.

The red is a payment the currency check refused because the rate on it was seventeen percent off the day's rate, which is exactly the kind of thing I want refused.

The amber is a pension payment.

What it found on its first run

The screen compares what he signed against what is sitting on the ledger, line by line. First time I ran it, it told me a small pension payment was on the books under the wrong person's name. No money duplicated. The account ties the bank statement to the cent, the trial balance balances, every completeness check came back green.

The check that stops us posting the same journal twice keys on the date and the amount and nothing else. Two same day payments of the same value to the same account are interchangeable to it, so it read the one already there as proof the other had been done. The fix is a description correction typed into the accounting UI. Nothing reverses.

A close is not proven by a total that ties. Every check we run is arithmetic, and arithmetic cannot see whose name is on a line.

The six false alarms first

Before that run found the real thing it found six things that were fine. The accounting system stores one hundred characters of a description and truncates the rest, so a long bank wire narrative comes back short and reads as though somebody had relabelled it. A card journal leg carries no description at all and one date for the whole journal, so the account and the amount are the whole of what you can ask it. One pack wrote its amounts as display strings, so a number with a comma in it parsed as nothing and the same charge went into two different failure lists. A previous month's journal credits the same control account, so eight of its lines got dragged into August.

Each one has a named cause and a test now, and five cases that should never turn red are pinned green so the proof cannot teach him to ignore it. An alarm that goes off on every correct close is one he stops reading by November. Two days on false alarms is the cheaper end of that.

The week I have

The deadline I set is that he can work things out with the agent himself for the mid month run, the fifteenth to the eighteenth. Today one entity has its record behind the agent. Three entities and three cards do not.

I built the audit screen as a separate file from the working reconciliation UI on purpose, so I could not break the thing that already works while I was learning what the new thing should be. That also means there are two codebases now that both know what a journal line looks like, and I will pay for that in about six weeks.

A close a controller can audit himself is a close he does not have to redo, and trust is what they are paying for. A machine that saves a finance team thirty hours and cannot show its working is a machine they stop opening in month four.

August's amber goes away with one description edit typed into the accounting UI. I have not typed it yet, because whether that duplicate check should also carry the account code and a hash of the description is a change to live money posting, and I want another night on it.

Day 179 of 365.

Get the next build-in-public post by email

One short dispatch most days — the AI ops builds, what's working, what isn't. No spam.

← Day 140 All posts Day 180 →

Follow the BIP

AI Deployment as a Service. One workflow at a time.

Book 15 minutes. We see if your workflow is one we'd build.

Schedule a call