Skip to content

HowTo use NFS for Persistent Storage

You can have some (or all) of your persistent volumes provisioned on your NAS server via NFS using csi-driver-nfs.

Prerequisites

  • The csi-driver-nfs driver requires existing and already configured NFSv3 or NFSv4 server.

High-level Setup Overview

diff --git a/metal/inventory/metal.yml b/metal/inventory/metal.yml
index aaaaaaa..bbbbbbb 100644
--- a/metal/inventory/metal.yml
+++ b/metal/inventory/metal.yml
metal:
     children:
         nas: null
         k3s: null
+nas:
+    hosts:
+        yggdrasil:
+            interface:
+                host: bifrost
+                interface: Gi0/4
+            ansible_host: 10.10.10.30
+            mac: aa:bb:cc:dd:ee:ff
+            data_drives: # NB! pre-partitioned
+                - id: ata-ST18000NM000J-FOOBAR_ABCDEF-part1
+                  brand: seagate
+            parity_drives: [] # NB! pre-partitioned
+                - id: ata-ST18000NM000J-FOOBAR_0123456-part1
+                  brand: seagate
+            network_interface: eno1
 k3s:
    children:
        control_plane:

!!! note NB! Unlike k8s nodes, the NAS OS is not installed via PXE. It's assumed a base server OS is already installed (both Debian-based and Fedora OSes are supported) on the NAS server.

  • Then run make metal

  • Install the driver on a Kubernetes cluster. See system/csi-driver-nfs for more details.

  • I use the static mode based on my own needs.

  • Declare a static PV for each NAS share. PersistentVolume is a cluster-scoped resource, so PVs live in the system layer — not in app charts — and are rendered from the nfs.volumes list in the csi-driver-nfs chart values:

./system/csi-driver-nfs/values.yaml
---
nfs:
  # TODO: use a DNS name once the NAS has a stable record (see docs/info/todo.md)
  server: 10.10.10.30
  basePath: /mnt/storage
  # Mount options applied to every static NFS PV below.
  mountOptions:
    - nfsvers=4.1
    # soft mount: return I/O errors instead of hanging indefinitely if the NAS goes down.
    # Without this, pods freeze and can't be killed or rescheduled until the NAS returns.
    - soft
    - timeo=600 # timeout per retry in deciseconds (60s)
    - retrans=2 # number of retries before returning error
    # 1MB read/write buffers for media streaming throughput.
    # Likely the negotiated default on modern kernels, but explicit to be deterministic.
    - rsize=1048576
    - wsize=1048576
  # One static PV per NAS share, named pv-nfs-<name>.
  # Consuming apps bind with a namespaced PVC that sets
  # volumeName: pv-nfs-<name> and storageClassName: "".
  volumes:
    - name: videos
      share: Videos # subdirectory of basePath exported by the NAS
      capacity: 1Ti
csi-driver-nfs:
  storageClass:
    create: false
./system/csi-driver-nfs/templates/pv-nfs.yaml
{{- range .Values.nfs.volumes }}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-{{ required "nfs.volumes[].name is required" .name }}
spec:
  capacity:
    storage: {{ required "nfs.volumes[].capacity is required" .capacity | quote }}
  accessModes:
    - ReadWriteMany
  # NAS data outlives any single consumer; never release the volume for reuse
  # (and never delete NFS contents) when the claim is deleted.
  persistentVolumeReclaimPolicy: Retain
  # No storageClassName: these are static PVs. Consuming PVCs must set
  # volumeName: pv-nfs-<name> and storageClassName: "" to bind.
  mountOptions:
    {{- toYaml $.Values.nfs.mountOptions | nindent 4 }}
  csi:
    driver: nfs.csi.k8s.io
    # volumeHandle just needs to be unique per share across the cluster;
    # server + real export path is used for readability.
    volumeHandle: "{{ $.Values.nfs.server }}{{ $.Values.nfs.basePath }}/{{ required "nfs.volumes[].share is required" .share }}"
    volumeAttributes:
      server: "{{ $.Values.nfs.server }}"
      share: "{{ $.Values.nfs.basePath }}/{{ .share }}"
{{- end }}
  • Create a static PVC in the consuming app's chart. The PVC binds to the PV via volumeName; storageClassName: "" must be the empty string (not omitted) so the default-StorageClass admission plugin doesn't assign a class and break binding:
./apps/jellyfin/templates/pvc-videos.yaml
---
# yaml-language-server: $schema=https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/master/persistentvolumeclaim.json
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-nfs-videos
  namespace: "{{ .Release.Namespace }}"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Ti
  # Static binding to the cluster-scoped NFS PV defined in system/csi-driver-nfs.
  volumeName: pv-nfs-videos
  # Empty string (not omitted): prevents the default-StorageClass admission
  # plugin from assigning a class, which would break binding to the class-less
  # static PV.
  storageClassName: ""

Note

Don't forget to create a Deployment as well, unless your template handles that automatically_

  • Use the PVC in your application, for example, Jellyfin:
./apps/jellyfin/values.yaml
  persistence:
    anothervideos:
      enabled: true
      type: persistentVolumeClaim
      existingClaim: pvc-nfs-videos
      advancedMounts:
        main:
          main:
            - path: /media/Videos
              readOnly: false

NFS Permissions and root_squash

The NFS server is configured with root_squash (the NFS default). This means any process connecting as root (uid 0) from a K8s node is mapped to the anonymous user on the NFS server. Non-root users pass through unchanged.

The anonymous uid/gid is configured via nfs_anon_uid/nfs_anon_gid (default 1000) and should match the fsGroup used by your application pods. This ensures that:

  • kubelet fsGroup chown works correctly — the kubelet runs as root, so its chown calls are squashed to uid 1000, which matches the target group
  • Root-running containers (e.g., linuxserver.io images that start as root before dropping to PUID/PGID) create files owned by uid 1000 on the NFS server
  • Non-root containers running as uid 1000 can read/write files created by either path above

Your pod spec should set fsGroup to match:

spec:
  securityContext:
    fsGroup: 1000

Tip

If your applications use a different uid (e.g., 568 for some Helm charts), override nfs_anon_uid and nfs_anon_gid in your inventory to match.

Reference