Where Does Your Monitoring Live?

On October 4, 2021, Facebook’s engineers had a diagnosis problem before they had a repair problem.

A routine maintenance command, meant to check how much backbone capacity was available, took down every connection in the backbone. The audit tool that should have stopped it had a bug. Then the DNS servers did what they were designed to do. They could no longer reach the data centers, so they withdrew their own routes (BGP advertisements) and vanished from the internet, even though they were still running. Facebook’s own postmortem says the loss of DNS broke many of the internal tools engineers would normally use to investigate. Normal access and out-of-band access were both down. So engineers went to the data centers in person.

The monitoring lived inside the failure. Ask where it lives.

You are not running Facebook’s backbone. Your outage will be smaller and your recovery shorter. The mechanism is the same, though, and it works on a three-node cluster as well as it works on a global network.

What does native monitoring already give you?

More than most teams use. On a Nutanix cluster, Prism Element keeps analytics data for 90 days, and you can configure Prism Central to hold it longer. On top of that sits Intelligent Operations, described in the Nutanix Cloud Bible against Prism Central 7.5.1 and NCM 2.0. Its anomaly detection learns an expected band for each metric from history. It needs 3 days of data for daily patterns, 21 days for weekly ones and 90 days for monthly ones. Capacity runway tells you which resource runs out first, and when. What-if planning answers questions like whether a cluster can take 50 more VMs. Alert policies and playbooks can open tickets and call REST APIs. With NCM 2.0, all of it works across several Prism Central domains at once.

That is a serious list. My default is native, and I reach for it first.

Now read the list again. Every line is about the cluster. Every baseline, every runway chart, every alert policy: the cluster.

So what can an outside stack do that native cannot?

Two things. It can survive, and it can join.

Survive first. Prometheus’s documentation says it is built to be “the system you go to during an outage.” Each server stands alone, with no dependence on network storage or remote services. That is good design. But independence belongs to placement, not to product. A failure domain is the set of things that die together: one host, one rack, one site. A Prometheus VM running on the cluster it watches sits in that cluster’s failure domain. Same failure, same blind morning.

Native has a recovery story, and it is a good one. Prism Central Backup and Restore keeps Prism Central’s configuration and data copied to Prism Element clusters, and a restore rebuilds Prism Central on one of them. Nutanix’s own diagram shows Prism Central hosted on Cluster 1 while it manages three clusters. Lose Cluster 1 and you rebuild. Rebuilding is recovery. It is not visibility, and while it runs you are blind. Only a watcher in another failure domain can tell you the cluster is gone.

Now join. Google’s SRE book says monitoring has to answer two questions: “what’s broken, and why?” It names four golden signals: latency, traffic, errors and saturation. Prism sees CPU, memory, storage and I/O latency on your platform. It does not see the errors your users receive. It does not see the sessions crossing your firewall. It does not see the retries in your application tier. The answer to why an application is slow can sit in any of them. Datadog’s Nutanix integration is built on this idea. It requires the v4 APIs through Prism Central, and it pulls Prism Central activity across so it sits beside everything else you measure.

Isn’t that just distrust of the platform?

No. Nutanix ships the recovery path and documents it. The point is narrower. Any single monitoring plane shares fate with its host. That is true of Prism Central. It is equally true of the Prometheus box you stood up beside it.

What does the outside stack cost you?

Less than people fear and more than they budget.

Google says a team of 10 to 12 SREs typically has one person, sometimes two, whose main job is building and maintaining monitoring, and that is with substantial shared infrastructure behind them. Your estate is smaller. One person’s week is still not zero.

Collectors ride on APIs, and APIs age. Nutanix’s AOS 7.5 documentation says the legacy v0.8, v1, v2 and v3 APIs lose support starting with a release planned for the fourth quarter of calendar 2027. The 2024 v4 announcement had named the fourth quarter of 2026. The direction has not changed: build on v4. A collector written against legacy endpoints is code you will rewrite.

And you will not get learned baselines or what-if planning from Prometheus unless you build them. Google’s guidance is interesting here. It keeps paging rules simple and avoids systems that learn thresholds on their own, but it says capacity planning and traffic prediction can tolerate more complexity. That describes where Nutanix puts baselines and runway. Use them for planning and diagnosis. Keep the pager simple.

What is the benefit analysis, line by line?

Benefit Native today What an outside layer adds Verdict
Visibility when the cluster is gone Recovery by rebuilding Prism Central from backup A watcher in another failure domain Worth it
Join with signals Prism cannot see The Nutanix estate, plus non-Nutanix ESXi through NCM One view with network, firewall and application data Worth it if you already collect those signals
Behavioral baselines Built in, with history windows of 3, 21 and 90 days Nothing worth rebuilding Stay native
Capacity runway and what-if Built in Nothing worth rebuilding Stay native
Retention beyond Prism’s 90 days on Prism Element, more on Prism Central Retention you choose Only if a policy demands it
Routing alerts to tickets and chat Alert policies and playbooks with REST actions One Alertmanager path Convenience, not capability
Cost to own Maintained by Nutanix A collector, a store and a person to maintain them Count it honestly

So what do you run?

Layers. Native for everything about the cluster. A thin outside layer for two jobs.

The first job is a heartbeat. In SRE terms it is black-box monitoring: test what a user would see, from where a user would stand. Prometheus’s blackbox exporter does this. It probes endpoints from the outside over HTTP, HTTPS, DNS, TCP, ICMP and gRPC, and it asks one question: does the thing answer? It needs no Nutanix API, so it has no API to age. Put it in another site, on another cluster, or on a small physical box.

The second job is the join. If the monitoring platform you already run ships a Nutanix integration, use it and skip writing an exporter.

Placement > features. A smaller tool in the right place beats a richer one in the wrong place.

Ask where it lives.

Takeaways

  1. Ask where your monitoring lives before you ask what it measures.
  2. Start native for anything about the cluster: baselines, runway, what-if planning and audits.
  3. Add an outside layer for two jobs only: staying alive when the cluster does not, and joining Nutanix data with signals Prism cannot see.
  4. Host that layer in a different failure domain, or it adds nothing.
  5. Collect through the v4 APIs or a vendor integration so your collector survives upgrades.
  6. Budget a person, not just a server. Google assigns one or two per team of 10 to 12.

Published October 6, 2026