Operations · Prometheus · agent-led review

AI Operational Reviews Reduce Operational Toil

At AWS, the operational-review meeting was the easy part. The work happened beforehand. Teams had to inspect dashboards, explain alarms, compare the week with the week before, and account for every unexplained spike. A latency jump might be a cache clear, a deployment, a dependency failure, or something nobody had noticed yet.

That discipline works, but it takes people. I wanted the same habit for the Raspberry Pi running my home agent and a handful of supporting services. A home lab is not AWS, even if I occasionally manage to make it fail at AWS-shaped scale.

I set up Hermes to run a read-only operational review every week. It inventories the available metrics, queries seven complete days, compares the preceding week, reconstructs alert intervals, and writes a short report. Findings land in three buckets: healthy, watch, or action needed. The agent can query and report. It cannot restart a service or change the monitoring configuration.

The week the graphs got interesting

One review found four out-of-memory kills. Available memory had fallen to 2.12 percent, and load peaked at 79 on a four-core machine. CPU temperature reached 85.35°C. Prometheus and node-exporter also missed scrapes during the incident.

4
OOM kills
2.12%
Memory available
79.14
Peak load / 4 CPUs
85.35°C
Peak CPU temperature

The report did not invent a root cause from those simultaneous symptoms. It did what a useful review should do: it narrowed the investigation and made the unexplained behavior hard to ignore.

The process that needed replacing

Follow-up work pointed us toward notes-quartz, the old Node-based notes browser. Quartz normally used roughly 535–640 MiB of resident memory and briefly spiked near 1.2 GB in the captured window.

Prometheus chart showing notes-quartz resident memory near 535 to 640 MiB with a brief spike near 1.2 GB, followed by Amythest around 40 to 60 MiB after the cutover
The cutover around 03:00 UTC. Quartz falls away; Amythest settles around 40–60 MiB. The annotated Quartz sample is 534.9 MiB; internal metric labels were removed from the public image. Tap or click the chart to open it full size.

We replaced it with Amythest, a single Go binary that serves the same vault. The replacement settled around 40–60 MiB, about 91 percent below Quartz's sampled baseline and 96 percent below its visible peak.

The review contributed to the rewrite decision. The graph does not prove Quartz caused every OOM event, and the agent did not authorize or perform the replacement on its own.

The next weekly comparison showed zero OOM kills and average CPU busy down about 58 percent. It also found a separate Hermes memory episode later. That second finding mattered: fixing one process did not earn the rest of the system a permanent green check.

The part worth automating

The agent handles the work that makes recurring reviews expensive. It uses fixed time windows, keeps the comparison honest, checks alert duration instead of counting screenshots, and records where the evidence stops. If a restart looks deploy-related, it says so. If the metrics cannot identify a remote source or prove a leak, it leaves the gap open.

Larger engineering organizations can afford this preparation because customer impact and scale justify it. Smaller teams make a rational trade: every hour spent reviewing dashboards is an hour not spent shipping or supporting customers. The operational risk does not disappear. It just stays unexplained until a customer finds it.

An agent changes that cost calculation. I still decide which risks are acceptable and which findings deserve engineering work. The agent makes sure I reach that decision with the week accounted for, the evidence in front of me, and fewer hours lost paging through graphs.

The meeting was never valuable because someone could operate a dashboard. It was valuable because the team understood what the system had done and knew what to improve next. That is exactly the sort of boring, repeatable work I want an agent to take off my plate.