Module 5: DSF Deep Dive (How Storage Actually Works)

The Promise

By the end of this module you will:

1. Trace a VM write through DSF end to end, naming every component it touches. From the guest's write() call to acknowledged-on-stable-storage, you will know what happens, where it happens, and why. This is the single most-tested concept in NCP-MCI storage.
2. Recommend RF2 vs RF3 vs erasure coding for a given workload with the math behind the choice. Capacity overhead numbers, performance tradeoffs, failure-tolerance differences. No guessing.
3. Pass roughly 28% of NCP-MCI and 30% of NCM-MCI. This is the heaviest single-module weight in the curriculum because storage is the heart of the platform. NCM-MCI lab scenarios disproportionately involve storage troubleshooting.
4. Defend the architecture against a senior storage admin who has been engineering arrays for 20 years. Real concerns like tail latency, write amplification, rebuild times, capacity efficiency, and vendor-managed firmware all have real answers. None requires hand-waving.
5. Size a cluster correctly for given workload patterns. Capacity reservation, thin provisioning, OpLog sizing, Curator scan duration on large clusters. The math you need for actual customer designs.
6. Talk about Stargate, Cassandra, and Curator in their operational roles, not just as service names. When NCC reports "Stargate degraded on node 3" or "Curator scan stalled," you know what that means and what to look at first.

This is the technical core of the platform. The reader who internalizes this module has internalized Nutanix.

Foundation: What You Already Know

You have managed shared storage. SAN or NAS, FC or iSCSI or NFS. You know the vocabulary: LUN, datastore, volume, RAID group, parity, hot spare, cache, write-back, write-through, controller failover, multipathing.

You have also been burned by it once or twice. Maybe a controller failed at 2am and the failover took 90 seconds longer than the customer's RTO allowed. Maybe a RAID rebuild ran for 18 hours during which performance was unusable. Maybe the array's compression algorithm was great in marketing but disappointing in reality. Real stuff.

Hold that experience. You are about to look at storage in a fundamentally different shape.

DSF (Distributed Storage Fabric) is the storage layer that runs across all the CVMs in a Nutanix cluster. There is no array. There is no controller in the traditional sense. There is software on every node that takes the local disks of every node, replicates data across them, deduplicates and compresses, presents one logical pool, and self-heals when nodes fail. The terminology is partly familiar (replication, compression) and partly new (OpLog, Extent Store, Curator, EC-X). The architecture, once you see it, is internally consistent.

The Foundation question for this module: how does a distributed storage layer running on the same hardware as your VMs match (and in some ways exceed) the performance and resilience of a dedicated array?

The honest short answer: by trading one set of architectural costs (extra hardware, separate controllers, FC fabric) for another (CVM resource tax, network round-trips for some operations, software complexity in distributed coordination). Whether the trade is favorable depends on the workload. For most enterprise workloads in 2026, it is. We will get to which workloads are which.

Core Content

The Distributed Storage Fabric (DSF) Overview

DSF is the storage layer that runs across all CVMs in a Nutanix cluster. From a 30,000-foot view it does these things:

• Pools the local disks of every node into one logical capacity pool.
• Replicates writes across nodes for durability (RF2 or RF3).
• Caches reads in CVM RAM (the Content Cache) for performance.
• Compresses, deduplicates, and (optionally) erasure-codes data for efficiency.
• Tiers data between hot (NVMe / SSD) and cold (HDD, where present) based on access patterns.
• Self-heals when nodes or drives fail by re-replicating data to surviving copies.
• Snapshots and replicates for backup and DR (Module 7).

DSF is built around three key services from Module 2: Stargate (the data path), Cassandra (the metadata store), and Curator (the background scrubber). Pithos owns vDisk configuration. Zeus maintains cluster consensus. These five services together implement DSF.

The Storage Hierarchy

Before tracing any I/O, you need the vocabulary. DSF uses a specific hierarchy:

1. Storage Pool. The aggregate of all physical disks in the cluster. Each cluster has one Storage Pool by default that includes all node-local disks. You almost never manipulate this directly.
2. Storage Container. A logical "datastore equivalent" carved from the Storage Pool. Containers are where you set storage-policy attributes: RF (2 or 3), compression on/off, deduplication on/off, erasure coding on/off, reservations, advertised capacity. A cluster typically has multiple containers.
3. vDisk. The virtual disk attached to a VM. From the VM's perspective, a vDisk looks like a SCSI or virtio block device. Internally, a vDisk is a logical entity backed by extents.
4. Extent. A contiguous range of bytes within a vDisk, typically 1 MB. Extents are the unit of metadata in Cassandra. Each extent has a metadata record indicating where it is physically stored.
5. Extent Group. The physical allocation unit on disk that holds extents. An Extent Group is 1 MB on non-deduplicated containers and 4 MB on deduplicated containers. Multiple extents from one or more vDisks may share an Extent Group.

Diagram: The Storage Hierarchy

The Data Path: A VM Write, End to End

This is the single most important concept in the module. We will walk a VM write from write() syscall to acknowledged-durable.

1. The guest VM issues a write. A Linux or Windows VM running on a Nutanix node calls into its storage stack: write(fd, buf, len). The OS turns this into a SCSI or virtio block command directed at the VM's vDisk.
2. The hypervisor presents the I/O to the local CVM. On AHV, the VM's vDisk is exposed via a paravirtualized interface that routes to the local CVM. On ESXi-on-Nutanix, the CVM presents an NFS datastore that ESXi mounts; writes traverse to the CVM via NFS. Either way: the I/O lands at the local CVM's Stargate process.
3. Stargate accepts the write. Its first decision: is this a sequential streaming write (likely large and benefits from going straight to Extent Store) or a random write (benefits from the OpLog write buffer)? Most database, VDI, and general-purpose workloads are random; Stargate routes them to the OpLog.
4. The write goes to the OpLog (local copy). The OpLog is a persistent write buffer on the local node's hot tier (NVMe or SSD). It is essentially a write-ahead log. Writes are appended and acknowledged once durable. The OpLog is where DSF gets its low-latency write characteristic.
5. The write is replicated to remote OpLog (RF copies). Simultaneously with the local OpLog write, Stargate replicates the write to one or two other nodes' OpLogs (RF2 or RF3 respectively). The remote OpLog write must complete before the original VM write is acknowledged. This is the cost of strong consistency.
6. Acknowledgment to the VM. Once the local OpLog and the required number of remote OpLog replicas have all confirmed durable write, Stargate acknowledges to the hypervisor, which acknowledges to the guest VM. The VM's write() call returns success.
7. OpLog drains to Extent Store (asynchronous). The OpLog is a buffer, not the permanent home for the data. As the OpLog fills past thresholds (and during quiet periods), Stargate drains writes from the OpLog to the Extent Store, the persistent backing storage on the local node. The drain is sequential and efficient.
8. Compression and (optionally) deduplication apply during drain. Inline compression compresses data as it moves from OpLog to Extent Store. If dedup is enabled, fingerprints are computed and dedup pointers created.
9. Erasure Coding (if enabled) is applied later. EC-X is a post-process operation. Curator runs periodic scans and converts qualifying data from RF replication to erasure-coded form.
10. Reads. When the VM later reads the data:
    • First check: Content Cache. Stargate maintains an in-memory cache of recently-accessed extents in CVM RAM. Cache hits return immediately.
    • Second: local hot tier (Extent Store on NVMe/SSD). Local reads are fast.
    • Third: remote node (over the network) if the data is not local. Curator may eventually migrate the data to be local for repeat reads.

Diagram: The Data Path

The Cycle, Frame Two: DSF as a SAN, Distributed

Step back. In a traditional all-flash array, an incoming write hits the array's controller, lands in NVRAM (write cache, battery-backed), is acknowledged to the host, and is later destaged to flash. The controller's NVRAM is the latency-critical buffer. Replication for redundancy happens between dual controllers in the same chassis.

DSF replaces the array controller with the CVM, replaces NVRAM with the OpLog (persistent on the local node's hot tier), and replaces dual-controller redundancy with cross-node replication (RF2 or RF3). The write path looks similar in shape: incoming write, durable buffer, acknowledgment, asynchronous destage. The major architectural shift is that the durable buffer and its peer copy are on different physical nodes connected by the network.

This is why network quality matters more in DSF than in a traditional array. Within a SAN array, controller-to-NVRAM is a backplane operation. In DSF, OpLog-to-OpLog crosses the cluster network. A 25 or 100 GbE switching fabric with low jitter is not optional; it is the substrate of write performance.

The Cycle, Frame Three: DSF as Cassandra-Plus-Stargate

Storage architects who appreciate distributed-systems internals will gravitate to this frame. DSF is, in essence, two coordinated systems:

• Cassandra (the metadata store). Tracks every extent: which vDisks reference it, what extent group it lives in, where physically (which nodes, which disks). Every extent operation involves Cassandra metadata reads or writes. Cassandra is built on a fork of Apache Cassandra, optimized for the NoSQL key-value access patterns DSF requires. It runs in a ring across all CVMs.
• Stargate (the data path). Processes the actual I/O. Reads and writes flow through Stargate. Stargate consults Cassandra for "where does extent X live?" and operates on the data accordingly.

The architectural separation is deliberate: metadata is a small, hot, frequently-accessed problem (good fit for a distributed NoSQL store). Data is a large, less-frequently-randomly-accessed problem (good fit for distributed object storage with caching). This is the same pattern that powers Ceph, S3, and other distributed storage systems.

The Cycle, Frame Four: DSF as the Value Layer

For an operations leader or CIO, the durable DSF story is not the data path. It is what DSF enables:

• Snapshots that don't degrade performance. DSF native snapshots are metadata operations.
• Replication as a built-in primitive. Async, NearSync, and Metro replication (Module 7) all leverage DSF's primitives.
• Compression and deduplication without an array refresh.
• Erasure coding as a capacity optimization.
• Capacity portability. vDisks can move between containers and clusters.
• Self-healing. When a node or drive fails, DSF rebuilds without human intervention.
• API-first storage. Storage operations are JSON, not vendor-specific CLI.

The CIO-level pitch: "You are not buying storage. You are buying a software-defined storage layer that grows when your compute grows, self-heals when hardware fails, and has every storage feature your team currently buys separately built in."

Replication Factor (RF) 2 and 3

RF is the per-container setting that determines how many copies of every write are stored across the cluster.

• RF2. Two copies of every write. One local, one on a peer node. Survives any single node failure or single drive failure with no data loss.
• RF3. Three copies of every write. One local, two on peer nodes. Survives any two simultaneous node failures (or two drive failures across nodes) with no data loss.

Capacity reservation (the cluster reserves headroom for self-healing) further reduces effective usable. A typical planning number is 50% effective for RF2 and 33% for RF3, with another 10-15% set aside for reservation. Compression and EC recover meaningful capacity back.

Performance implications:

• RF2 has slightly lower write latency than RF3 (one fewer network round-trip per write).
• RF3 has more network traffic during normal writes and during rebuilds.
• For most workloads the latency difference is small (single-digit percent on modern 25/100 GbE networks).

Erasure Coding (EC-X): The Capacity Optimization

Erasure coding is an alternative to replication for redundancy. Instead of storing two or three full copies, EC stores data plus parity in a way that survives node failures with substantially less capacity overhead. DSF's implementation is called EC-X. It is configurable per Storage Container.

Take a stripe of N data blocks plus K parity blocks across N+K nodes. Any K nodes can fail and the data is still recoverable. The capacity overhead is K/(N+K).

Going from RF2 to EC 4+1 reduces capacity overhead from 100% (50% effective) to 25% (75% effective). On a 200 TB raw cluster, that is the difference between 100 TB usable and 150 TB usable: a 50% increase in effective capacity, with the same failure tolerance.

The catch: EC has performance tradeoffs.

1. Write amplification on small writes. Writing a small block requires reading the existing stripe, modifying it, and rewriting parity. Expensive for random write workloads.
2. Curator overhead during conversion. EC is applied as a post-process operation by Curator. New writes start as RF2 or RF3 and are converted to EC later when data has cooled. The conversion consumes background CPU and network.
3. Recovery is more expensive. Rebuilding from EC parity requires reading from multiple surviving nodes and recomputing. Single-replica rebuilds (from RF) are simpler.

Diagram: RF vs EC Capacity and Failure Domains

Compression

DSF supports compression at the Storage Container level. It happens during the OpLog drain to Extent Store (so writes are acknowledged before compression runs; the compression cost is in background drain throughput, not in write latency).

• Inline compression. Applied during drain. The default in modern AOS for most workloads.
• Post-process compression. Applied later by Curator on data that wasn't compressed during drain.

Compression algorithms. DSF uses LZ4 for inline compression on incoming writes (fast, modest ratios, suitable for the latency-critical path). For cold data that has aged onto the cold tier or that Curator processes post-hoc, DSF uses LZ4HC (high-compression LZ4), which trades CPU for tighter ratios. Inline compression is selective: it applies to sequential streams and large I/Os (greater than 64K) to avoid impacting random write performance.

Real-world ratios: mixed enterprise workloads typically achieve 1.5x to 2.5x compression on DSF. Marketing numbers of 4-6x reflect best-case scenarios. Plan capacity using 2x for general workloads, 1.2x for already-compressed data, and let actual performance guide adjustment.

Deduplication

DSF supports deduplication at two levels:

• Cache deduplication. In-CVM-RAM deduplication for hot data. Always on for compatible workloads.
• On-disk deduplication. Configured per Storage Container. Computes fingerprints during drain, deduplicates blocks across the container.

When dedup is valuable: VDI workloads with persistent profiles, server VMs with similar OSes, test/dev environments with cloned VMs.

When dedup is expensive without benefit: already-deduplicated sources, small clusters with low data uniqueness, workloads with high write churn.

Sizing implication: dedup is metadata-heavy. Enabling on-disk dedup increases the CVM's metadata footprint, which means larger CVMs (more RAM) on dedup-enabled clusters. Size accordingly.

Tiering and ILM

On hybrid platforms (NVMe/SSD + HDD), DSF tiers data automatically. Hot data lives on the NVMe/SSD tier; cold data drifts to the HDD tier. Curator drives this via ILM (Information Lifecycle Management) scans.

In 2026, all-flash (or all-NVMe) Nutanix deployments dominate, and tiering is less operationally relevant for most customers. On hybrid platforms (still shipping for cost-optimized capacity tiers like backup or large file workloads), tiering is significant.

ILM operations include: promoting hot data to the fastest tier, demoting cold data to capacity tiers, migrating data for locality after VM moves, rebalancing when nodes are added or removed, and converting data from RF to EC. ILM runs continuously in the background; it does not appear in the synchronous I/O path.

Curator: The Background Worker

Curator is the background scrubbing and rebalancing service. It runs on every CVM. One Curator is master; others are followers. Curator does:

1. Periodic scans (full and partial). Full scans every 6-24 hours; partial scans more frequently. Scans walk metadata to identify operations to perform.
2. Re-replication after failures. Restores configured RF or EC after a node or drive loss.
3. EC conversion. Converts qualifying data from RF to EC.
4. Compression and dedup post-process (in some configurations).
5. Capacity reclamation. Garbage collection of deleted vDisk space.
6. Tiering / ILM operations.
7. Rebalancing when nodes are added or capacity is uneven.

Curator is a workhorse. On a healthy cluster you barely notice it. When something is wrong (a stalled scan, a long-running re-replication, a backlog of pending operations), Curator status is one of the first things you check. The CLI command curator_cli and the NCC checks expose Curator state.

Failure Recovery: When a Node Goes Down

Walk through what happens when a node fails:

At the moment of failure:

• The cluster heartbeat (Cassandra and Zeus mechanisms) detects the node loss within seconds.
• Stargate on the failed node stops responding. Other CVMs route I/O around the failed node.
• VMs that were running on the failed node are restarted by Acropolis (AHV) or vSphere HA (ESXi-on-Nutanix). Module 3 covered this.
• Cassandra removes the failed node from the active ring. Operations continue with the surviving nodes.

Curator's response:

• Curator detects the missing replica copies (data that was on the failed node now has fewer than RF copies remaining).
• Curator orchestrates re-replication: reads surviving copies, writes to other nodes, restores configured RF.
• The duration depends on cluster size, data volume, and network bandwidth. For a 4-node cluster with 50 TB on the failed node, re-replication takes hours. For a 20-node cluster, much less.
• During re-replication, the cluster runs at reduced redundancy. A second failure during this window can result in data loss for RF2 clusters.

For RF3 and EC clusters: the cluster tolerates the failure plus one more (RF3) or one more on a subset of nodes (EC 4+2). Re-replication still occurs to restore configured redundancy.

Diagram: Node Failure Recovery (Curator Scan)

Capacity Planning: The Math Customers Need

When a customer asks "how much usable capacity will I get?", the math is more nuanced than just dividing raw by RF.

1. Raw cluster capacity. Sum of all drives across all nodes. Example: 8 nodes × 4 drives × 7.68 TB NVMe = 245.76 TB raw.
2. Apply RF or EC. RF2: divide by 2. RF3: divide by 3. EC 4+1: 80% of raw. EC 4+2: 67% of raw.
3. Apply capacity reservation. AOS reserves headroom for self-healing on node loss. Typically 10-15% on smaller clusters, less on larger ones.
4. Apply compression and dedup gains. General compression: 1.5-2.5x. Dedup: variable; conservative planning is 1.0x unless workload-specific data supports a higher number.

Worked example: 8-node cluster, 4 × 7.68 TB NVMe per node, RF2, compression on, no EC, no dedup, general workload.

Raw:                       245.76 TB
After RF2 (÷ 2):           122.88 TB
After 12% reservation:     ~108 TB
After 2x compression:      ~216 TB effective usable

The customer who hears "raw 245 TB, usable 216 TB" in a marketing slide is being told a real number, but only after compression. State all the assumptions when you walk through this.

Storage-Only Nodes

For workloads that need a lot of capacity but not much CPU (some backup repositories, some file workloads, some object backing), Nutanix offers storage-heavy nodes and storage-only nodes.

A storage-only node has minimal compute (just enough for the CVM and DSF responsibilities) but large storage capacity. They contribute to the cluster's storage pool without contributing meaningful compute. They allow asymmetric clusters where a few storage-only nodes provide capacity for storage-heavy workloads while compute-balanced nodes handle the active VMs.

When to use: backup repositories on Nutanix (especially with Mine), some Files / Objects workloads where the storage-to-compute ratio is high, clusters that have outgrown their compute-balanced node sizing on the storage axis.

Tradeoffs: erodes some of HCI's operational simplicity (asymmetric node configurations), still simpler than maintaining a separate dedicated storage tier. Storage-only nodes do not eliminate the storage-imbalanced economics issue from Module 1; they soften it.

Lab Exercise: Storage Container Manipulation and Failure Simulation

1. Inventory the existing storage hierarchy. SSH into a CVM:

   ncli storage-pool list
   ncli container list
   ncli vdisk list

2. Create a Storage Container with non-default policies. From Prism Element: Storage → Storage Container → Create. Name lab-rf2-compressed, RF=2, compression on, EC off, dedup cache-only, no reservation.
3. Create a second container lab-rf2-no-features with all advanced features off.
4. Provision a VM into each container. Power them on with a small Linux OS.
5. Generate I/O: dd if=/dev/urandom of=/tmp/test.dat bs=1M count=2048 for 2 GB random, then dd if=/dev/zero of=/tmp/zero.dat bs=1M count=2048 for 2 GB zeros (zeros compress to nearly nothing).
6. Compare in Prism. The compressed container should show meaningfully less physical space used for the same logical data, especially the zeros.
7. Examine the data path components.

   nodetool -h localhost ring     # Cassandra ring status
   curator_cli get_curator_state  # Curator status
   stargate_status                # Stargate health

8. Look at vDisk metadata.

   vdisk_config_printer
   vdisk_usage_printer -vdisk_id <id>

9. Failure simulation (lab cluster only). From Prism, place a node in maintenance mode, then shut down the CVM via SSH (sudo shutdown -h now). Watch alerts fire, Data Resiliency drop, VMs restart, Curator begin re-replication. Power back on, exit maintenance, watch redundancy restore.
10. Run NCC after recovery. ncc health_checks run_all.

Practice Questions

Twelve questions. Six knowledge MCQ, four scenario MCQ, two NCX-style design questions. Read each, answer in your head, then click to reveal.

What You Now Have

You can trace a VM write through DSF end to end: guest syscall, hypervisor passthrough, local CVM Stargate, local OpLog, remote OpLog (RF=2 or RF=3), acknowledgment, asynchronous Extent Store drain, eventual compression and dedup, eventual EC conversion. Every component named.

You know the storage hierarchy: Pool → Container → vDisk → Extent → Extent Group. You know the Container is the policy boundary.

You know RF2 and RF3 in detail: capacity overhead, failure tolerance, when to choose which, the math of the choice. You know Erasure Coding (EC-X): 4+1 vs 4+2, capacity savings, performance tradeoffs, where EC wins and where it costs. You can size a cluster correctly for EC requirements (6+ nodes for 4+1, 7+ for 4+2).

You know compression and dedup honestly: what they cost, what they deliver, when to enable them. You quote ranges, not marketing peaks. You know how Curator works: background scans, re-replication, EC conversion, ILM, capacity reclamation.

You can walk a customer through a node-failure scenario: detection, VM restart, Curator re-replication, the re-replication window, the second-failure exposure, the duration math. You can do capacity planning math: raw, RF, reservation, compression, dedup, effective usable.

You are now ready for networking. Module 6 takes the same depth treatment to AHV networking, Open vSwitch, virtual switches, VLANs, bridges, and Flow microsegmentation.