Most fault detection projects for building systems do not fail on a metrics dashboard. They fail in a control room six weeks after go-live, when an operator silences the alert feed because it cries wolf four times a day. After that, the model can be 99% accurate and it does not matter. Nobody is reading it.
This is the part that does not show up in the model card. The constraint on an HVAC fault detection system is not how many faults you can catch. It is how many false alarms an operator will tolerate before they tune you out for good. That number is low, and it is the number you should be optimizing for.
Why false-positive rate is the metric that matters
Recall feels like the goal. You want to catch every failing chiller, every fouled coil, every stuck damper. So teams push the threshold to catch more, recall climbs, and the false-positive rate climbs with it. Each false alarm is a maintenance crew dispatched to a healthy unit, a work order that turns out to be nothing, a small withdrawal from the operator's trust. Trust does not come back at the rate it leaves.
Precision in alerting is what keeps the system alive in production. An alert that is right most of the time gets acted on. An alert that is wrong a third of the time gets a filter rule in Outlook. We tune for a low false-positive rate even when it costs us some recall, because a slightly less sensitive model that operators trust catches more real faults in practice than a sensitive one they ignore.
The 93 / 7 operating point
On our HVAC building management work we landed on roughly 93% true positive rate at about 7% false positive rate. We did not pick that point to win a benchmark. We picked it by walking the ROC curve with the operations team and asking, at each candidate threshold, how many false dispatches per week this produces and whether the crew would still trust the feed at that volume. Seven percent was the point where the alerts stayed credible and the missed faults were the marginal, slow-developing ones that the maintenance schedule would catch anyway.
Detection is only the floor. On top of the classifier we run remaining-useful-life estimation, so a flagged component comes with a runway rather than a binary alarm, and load-forecasting energy optimization that adjusts setpoints inside comfort constraints instead of overriding them. The constraint matters: an optimizer that saves energy by letting a zone drift out of comfort range gets switched off as fast as a noisy alarm does.
How it is built
- Telemetry streams off the BMS over Kafka, which decouples ingestion from inference and lets us replay history when we retrain or re-tune a threshold.
- Models are scikit-learn, versioned and tracked in MLflow so every deployed threshold is tied to the exact run, dataset, and operating point it came from.
- Alert suppression sits between the model and the operator: debounce, hold-down windows, and grouping so one root fault does not fan out into twenty alerts. This logic is what protects the false-positive budget in the real world, separate from the model itself.
- Serving is FastAPI, which keeps inference a plain, observable HTTP boundary the rest of the platform can call.
The suppression layer is worth dwelling on. Your offline false-positive rate is measured per prediction. The operator experiences false alarms per shift. A correlated burst of true-but-redundant alerts reads as noise even when every one is technically correct, so the debounce and grouping logic does as much for trust as the threshold does.
What it returns
Run this way, the suite delivered 15 to 20% energy savings across the sites, and shifted maintenance from reactive to predictive: crews work off a ranked list of components with time-to-failure estimates instead of chasing breakdowns. Because the value is measurable at the meter, the commercial model can be too. We are comfortable structuring these engagements around outcomes, saved energy and avoided downtime, rather than billing for a dashboard nobody opens. The model being accurate was never the hard part. The model still being believed in month six is the whole job.
