The problem

A regional engineering and surveying firm gets plans, permits and job documents from builders through one shared inbox. Two people open every email, figure out which job it belongs to and save the attachments to the right folder on SharePoint.

The owner wanted that off their plate. He did not want an AI writing into the live drawings library until it had proven it files the way his team does.

What we built

Not a set of filing rules. A feedback loop where the humans are the training data.

We started from history. Claude read about 300 inbound emails, matched each to where the staff filed it and wrote the firm's filing rules in plain language. The job list moved from an Access database into Airtable, all 8,455 jobs, so the AI and the staff work from the same whiteboard.

Then we let it run beside the team and watch what they do.

How the loop works

  1. Every weekday morning it reads the prior day's inbound email from a copy of the shared inbox.
  2. For each item it decides: file, skip or unsure. If it files, it saves a copy into a test folder that mirrors the real library. The real library is never touched.
  3. It writes one line on that job's Airtable record: date, subject, where the AI put it, "Staff: Pending".
  4. Two business days later it looks at where the staff actually filed the same document.
  5. It grades itself on that same line: Match, Different folder, Renamed, AI only or Staff only.
  6. Every miss enriches the instructions. When staff file something differently, the reason becomes a new or corrected rule, so the next run decides with more context than the last.
  7. Each morning a short summary goes out with match rates by category and a list of unsure items for a person to review.

Why "unsure" matters

In its first two days it logged decisions on 44 jobs. It filed 14 and flagged 30 as unsure instead of guessing. Almost all of the unsure items came from one large builder whose drawings live in a library the AI could not see yet.

That is the loop doing its job. It found a permissions gap on day one and said so, instead of quietly filing things in the wrong place.

What changed

The firm has its filing logic written down for the first time, and it gets better every week from the staff's own decisions. Every AI call sits next to the human call on the job record, graded. Nobody has to trust a black box.

The live switch comes when the match rate holds and the unsure list is down to true edge cases. Until then the staff keep filing exactly as they always have.

What we would do again

Our take: your staff already know the answer. Measure what they do and feed it back. That beats any rulebook we could write up front.

Build the grading before the automation. Without a scorecard a filing bot is a guess. With one it is a trainee you can measure.

Let it say "unsure". Unsure items are the cheapest way to find missing access, missing rules and real edge cases.