Notes from a technical session on Nutanix Enterprise AI, presented by the man the community calls Mr. vSAN.
The model is the easy part.
Pull one from Hugging Face, point a container at it, and you get answers in minutes. What takes months is everything around it: who can call it, how much they can spend, where the data goes, what happens when the GPUs are full, and whether your floor can hold the rack.
Samsung learned this the public way. In April 2023, engineers in its semiconductor division pasted internal source code into ChatGPT to check it for errors. The company had allowed that use. On May 2, Samsung told staff in a memo that generative AI tools were banned on company devices and its internal network, as reported at the time. The memo called the ban temporary, until headquarters could build a secure environment. A Samsung survey that April found 65% of respondents believed these tools were a security risk.
Nobody in that story was malicious. The engineers wanted help and the tool was good. What was missing was a sanctioned place to send the request. Ban the tool and the request does not vanish. It moves somewhere you cannot see. Visible traffic > blocked traffic.
So why not build the sanctioned place yourself? That is the job Nutanix Enterprise AI (NAI) is built for. A disclosure first: I work in presales, and my preference is Nutanix. I will tell you what I think is strong, and I will mark what is not.
Who was talking?
The session was presented by Simon Todd, AI specialist for EMEA at Nutanix. Before it started, I typed one question into the chat: what should we expect from this session? The agenda answered it. Infrastructure, architecture and hardware first. Live demo second.
That order makes more sense once you know the presenter. Simon goes by Mr. vSAN. That is not my joke. He says it himself on mrvsan.com, where he writes that most people in the industry call him that. By his own account he joined VMware in 2009 as a storage engineer in technical support, became the Global Technical Lead for vSAN, and moved to Intel in March 2020. A 2017 London VMUG write-up introduces him with the same name. Now he teaches rooms how to run LLMs on Nutanix, and he started with the hardware.
What does NAI actually do?
It sits between your developers and your models. That is the whole idea. NAI is a Kubernetes-native application that acts as an AI gateway. Developers get an endpoint and an API key. IT keeps the models, the GPUs, the access rules and the logs.
Isn’t that just a proxy? A proxy forwards. A gateway decides: who is calling, which model they may reach, how many tokens they may spend, which tools their agent may touch.
Keep three layers apart in your head.
+------------------------------------------------------------+
| APPLICATION RAG pipelines, chatbots, agents |
+------------------------------------------------------------+
| PLATFORM NAI gateway and inference endpoints (vLLM) |
| on NKP, EKS, AKS, GKE, K3s or Tanzu |
+------------------------------------------------------------+
| INFRASTRUCTURE Nutanix Cloud Infrastructure (AHV) |
| Nutanix Unified Storage, NVIDIA GPUs |
+------------------------------------------------------------+
The application layer is where your RAG pipelines, chatbots and agents live. The platform layer is NAI plus a Kubernetes runtime. Per the session, that runtime can be the Nutanix Kubernetes Platform (NKP) on bare metal, a managed service such as AWS EKS, Azure AKS or Google GKE, or a lighter runtime such as K3s or VMware Tanzu. The infrastructure layer is where the storage and the GPUs sit.
GPUs are where the first wrong assumption usually appears. They reach the cluster through the NVIDIA GPU Operator, with AHV passing whole devices through to the worker nodes. NVIDIA’s AI Enterprise support matrix lists that combination for Nutanix, and it states that partial-GPU passthrough is not supported. Size in whole GPUs.
What if the site has no internet?
Then NAI still works, which is the point for regulated buyers. Simon said NAI supports air-gapped and dark-site deployments: models can be pre-loaded, or fetched through explicit forward proxy rules. Before NAI 2.8, NVIDIA NIM downloads still needed a live connection to the NGC catalog. Version 2.8, released on August 26, 2026, lets customers with NVIDIA AI Enterprise licenses upload NIMs from a local file share or bucket instead.
Models arrive three ways: the Hugging Face Hub, NVIDIA NGC through NIM containers, or your own fine-tuned model. Fine-tuning for models under 8B parameters became generally available in the same release.
Is it fast enough?
Simon covered two levers. Speculative decoding cuts latency. Batch inference mode raises throughput. They pull against each other, and Nutanix’s own engineers say so.
In a July 2026 engineering post, Neelabh Sinha benchmarked n-gram speculative decoding on NAI 2.6. On Mixtral-8x7B across two GPUs, mean inter-token latency fell from 27.75 ms to 22.78 ms, a gain of 18%. Llama-4-Scout across four GPUs improved 14.5%. Gemma-2-2B gained 5.53%. Llama-3.2-3B gained 1.15%.
Why the spread? Small models guess poorly. Draft acceptance sat between 15% and 25% for them, against 22% to 40% for the larger models. And the technique is a compute-for-memory trade. It wins when the GPU has idle cycles. It loses when the GPU is busy. So you enable it on a chatbot or a RAG endpoint where a person is waiting, and you leave it off on the endpoint that chews through a nightly batch. NAI offers it only on GPU endpoints, and only the n-gram method for now. Nutanix labels these results as coming from a test environment. Treat them that way.
Idle endpoints are the other cost. NAI endpoints can be active, hibernated or resumed. Hibernating one releases its compute and GPU resources and keeps its configuration, according to Nutanix’s NAI 2.3 release post. In 2.8 you can even change parameters on a hibernated endpoint instead of rebuilding it. A GPU is a shared pool. An endpoint should hold one only while someone is using it.
Who gets in?
Simon created an endpoint live, and the API key appeared once. After that it was gone. If you did not move it into a vault in that moment, you create another. I could not find this behavior stated in NAI’s public documentation, so I am reporting what the demo showed. Administrators can revoke or rotate a key per user or per application without touching the Kubernetes deployment underneath.
Identity follows the same discipline. NAI integrates with Active Directory, OpenLDAP and SAML SSO. RBAC separates cluster administrators from regular users, who can manage only their own models and APIs. In 2.8, a custom role builder adds more than 30 permissions across users, licensing, models, endpoints, API keys and observability.
Does that matter if your agents are the ones calling the API? More every month. Gartner predicts that 40% of enterprises will pull back or shut down their autonomous agents by 2027, after production incidents expose gaps in governance. Nutanix cites that prediction in its NAI 2.8 launch post, then ships answers. MCP server management is now generally available in the Nutanix Agent Gateway. One key can bind to both the inference endpoints and the MCP connector. Header-based rate limiting with token budgets arrives as a tech preview. Tech preview means what it says. Keep it out of production.
Before you hand anyone a key, test the endpoint yourself. NAI Labs ships sample applications for that: Chat, Talk to My Data and an Agent app. Use them to confirm the endpoint behaves before the application team builds on it. Then the developer’s side is short. If their code already speaks the OpenAI format, they change a base URL and a key.
# NAI exposes an OpenAI-compatible API. The path below follows a public
# NAI 2.6 example. Confirm it against your version's documentation.
export NAI_API_KEY="<key stored in your vault>"
curl -s https://<nai-host>/enterpriseai/v1/chat/completions \
-H "Authorization: Bearer $NAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "<your-endpoint-name>",
"messages": [
{"role": "user", "content": "Explain what a KV cache is in two sentences."}
],
"stream": false
}'
Do you need a GPU at all?
Maybe not for every workload. Simon’s CPU slides made Intel’s case for small language models on Xeon: knowledge assistants with RAG, document intelligence, classification and data extraction, edge deployments. The enabler is Intel AMX, an accelerator built into every core of 4th, 5th and 6th generation Xeon processors. Intel’s slide claims up to 10x higher PyTorch training and inference performance with AMX in BF16, measured against the prior generation at FP32. It also notes that Nutanix AI supports AMX.
Read that number with care. It compares two Xeon generations. It does not compare a Xeon to a GPU, and the footnote cites 4th generation results with the usual “results may vary” warning. The honest framing is narrower and still useful. If your model is small and your SLA is modest, an existing CPU cluster may meet it, and the GPU budget stays free for the workloads that need one.
Where does the rack end?
Simon closed on the part nobody puts in the pitch deck: the building. In the session he described rack power climbing from a standard 14 kW toward 250 kW, delays in grid connections, and floors with finite weight ratings.
Check that against the hardware. SemiAnalysis puts a general-purpose CPU rack at up to 12 kW and an air-cooled H100 rack at about 40 kW, and it puts the GB200 NVL72 at roughly 120 kW, liquid-cooled. HPE specifies 132 kW nominal for its NVL72 rack and advises provisioning for about 192 kW at peak. One deployment guide puts the compute rack at 1,500 kg, a point load of 1,875 kg/m², against raised floors commonly rated at 1,000. I could not tie 250 kW to a shipping rack, so I treat it as Simon’s forward-looking figure, not a spec sheet.
Power budget > GPU budget. Grid connections arrive on the utility’s schedule, not yours. Call that a lecture if you like. Facilities will give you the same one after the purchase order.
One more time
The model is the easy part.
You need a gateway your developers will actually use. You need keys you can revoke. You need GPUs that hibernate when nobody is looking. You need a rack that exists. Samsung’s engineers had a good model and no sanctioned path. Build the path.
What to do on Monday
- Give developers a sanctioned endpoint before they find an unsanctioned one.
- Size GPUs as whole devices; partial-GPU passthrough is not supported.
- Enable speculative decoding only on latency-bound endpoints with spare GPU, and benchmark first.
- Hibernate idle endpoints so the GPU pool stays shared.
- Move the API key into a vault the moment it is created; you will not see it again.
- Ask facilities for kilowatts per rack, cooling type and floor rating before you sign the GPU order.
Published October 8, 2026