VMware ESXi

To use Ceph storage in your ESXi Hypervisors, we now recommend using NVMe-oF. The caveat here is that your ESXi needs to be up-to-date (at the time of writing we are using ESXi 7.0.3).

One you setup your NVMe-oF Gateway in croit:

1. Locate the NIC.

esxcli network nic list

2. Enable NVMe/TCP on the NIC.

# Substitue vmnic0 with the correct NIC
esxcli nvme fabrics enable --protocol TCP --device vmnic0

3. Locate the VMkernel NIC.

esxcli network ip interface list

4. Enable NVMe/TCP on the VMkernel NIC.

# Substitute vmk0 with the correct VMkernel NIC
esxcli network ip interface tag add --interface-name vmk0  --tagname NVMeTCP

5. Get the host NQN.

esxcli nvme info get

6. Add the host to NVMe-oF Gateway.

   • Navigate to Gateways -> NVMe-oF -> Hosts.
   • Click Add.
   • Enter the hypervisor NQN.
   • Add to your ESXi Subsystem.

7. Get the Adapter name.

esxcli nvme adapter list

8. Connect to the Gateway.

# Substitute vmhba65 with your nvmetcp adapter
# Substitue the NQN with your croit subsystem NQN
# Substitue IP with your Gateway nodes
# Repeat for all your Gateway nodes
esxcli nvme fabrics connect -a vmhba65 -i 172.31.134.101 -p 4420 -s nqn.2024-06.io.croit:ceph-gateway1
esxcli nvme fabrics connect -a vmhba65 -i 172.31.134.102 -p 4420 -s nqn.2024-06.io.croit:ceph-gateway1

9. Add a disk to the NVMe-oF Gateway.

   • Create an RBD.
   • Navigate to Gateways -> NVMe-oF -> Disks.
   • Click Add.
   • Select the RBD.
   • Important: At the time of writing, ESXi appears to only recognize NVMes with 512B block size.

10. Confirm the disk is visible in ESXi.

esxcli nvme namespace list
esxcli storage hpp device list

11. Confirm all paths are active.

esxcli storage hpp path list

12. Repeat 1-11 for all your hypervisors.

13. You can now create a VMFS on top of this disk.