The system
This is a content operations system built for an enterprise media organisation. It runs as a pipeline of AI agents: a research agent that gathers source material, a drafting agent that produces a structured first draft, a consistency agent that checks the draft against a style guide and a facts database, and a routing agent that sends the draft to the appropriate human reviewer.
The system has been in production for twenty-eight months. It processes between thirty and sixty content pieces per week.
What it replaced
Before this system, the content team’s workflow was: a human researcher gathered sources, handed them to a writer, the writer produced a draft, the draft went through two rounds of human review, and the approved version was formatted and published. Total time per piece: eight to twelve hours. Total human hours: five to seven.
The system did not eliminate human involvement. It changed what humans are doing. Researchers now spend their time on source quality and accuracy checking rather than gathering and formatting. Writers now spend their time on review and revision rather than first-draft production. The system handles the gathering and drafting; humans handle the evaluation and approval.
The actual metrics
After twenty-eight months in production:
Error rate: Below three percent of published outputs require post-publication correction. Pre-system, the rate was approximately four percent. This is a modest improvement, not a dramatic one: the human review layer that remained is the primary quality gate.
Time per piece: Average human time per piece has fallen from five to seven hours to fifty minutes to ninety minutes. The system produces first drafts in twelve to twenty minutes that previously took two to three hours to produce.
Throughput: Volume has increased by approximately sixty percent without adding headcount. The additional capacity was directed toward higher-complexity content that the system does not handle.
Maintenance overhead: The system requires approximately three hours of maintenance attention per week: monitoring the agent pipeline, reviewing edge case failures, updating the style guide database, and managing the occasional infrastructure incident. This is my ongoing time as the system’s maintainer.
What went wrong
Three significant incidents in twenty-eight months of production.
Month four: A data pipeline change upstream caused the research agent to start pulling stale source material. The output quality degraded without a visible error: the system produced confident-looking drafts citing outdated information. We detected it through a scheduled quality audit, not through automated monitoring. We added a data freshness check to the research agent. Lesson: automated monitoring needs to check the quality of agent inputs, not just whether agents ran.
Month eleven: A library update broke the consistency agent’s style guide parsing. The agent silently skipped the style check for two weeks rather than failing visibly. We added explicit validation that the consistency check returned a non-null result. Lesson: silent failure modes are more dangerous than noisy ones.
Month twenty: The drafting agent started producing outputs that were technically compliant with the style guide but were missing a structural element that the style guide did not explicitly require. The issue was discovered in human review and traced to a model update from the hosted provider. Lesson: hosted model updates require a regression test against known good outputs before deployment.
What I would do differently
I would build the monitoring infrastructure before the first agent goes live, not after the first incident. In this project, monitoring was added reactively after each failure. A pre-built monitoring stack would have caught the month-four and month-eleven failures earlier.
I would also negotiate a more detailed SLA with the hosted model provider about advance notice of model updates. The month-twenty incident was caused by an update we were not notified about in advance.