Deploying CephFS as a Self-Hosted GCP Filestore Alternative

CephFS is a POSIX-compliant distributed file system built on Ceph, deployable on Kubernetes via the Rook operator with dynamic CSI provisioning and optional NFS-Ganesha export for external clients, a self-managed, Kubernetes-native alternative to GCP Filestore. This guide deploys Rook + Ceph, creates a CephFS filesystem, provisions PVCs, exports over NFS, configures snapshots/backups, sets up CephX auth and tenant isolation, adds monitoring, and covers migrating off Filestore.

Concept Mapping

Components: Rook Operator (deploys/manages Ceph), MON/MGR (quorum + monitoring), OSDs (data on block storage), MDS (CephFS metadata), Ceph CSI driver (dynamic provisioning), NFS-Ganesha (NFS export), Prometheus + Grafana (monitoring).

Prerequisites: a Kubernetes cluster with 3+ worker nodes, each with a raw unformatted block storage volume attached; kubectl configured; Helm 3; basic Kubernetes storage familiarity.

Install the Rook Ceph Operator

$ mkdir -p ~/rook-ceph

$ cd ~/rook-ceph

$ kubectl get nodes

Confirm all nodes are Ready.

$ helm repo add rook-release https://charts.rook.io/release

$ helm repo update

$ helm install rook-ceph rook-release/rook-ceph --namespace rook-ceph --create-namespace --version v1.19.7

This installs the operator + CRDs. Pinned to v1.19.7 — a tested release that auto-provisions the rook-ceph.cephfs.csi.ceph.com CSI driver. If you use a newer version, confirm that driver exists before provisioning PVCs.

$ kubectl get pods -n rook-ceph

$ kubectl get crds | grep ceph.rook.io

Deploy the Ceph Cluster

$ nano ceph-cluster.yaml

apiVersion: ceph.rook.io/v1

kind: CephCluster

metadata:

  name: rook-ceph

  namespace: rook-ceph

spec:

  cephVersion:

    image: quay.io/ceph/ceph:v19.2.3

  dataDirHostPath: /var/lib/rook

  mon:

    count: 3

    allowMultiplePerNode: false

  mgr:

    count: 1

    allowMultiplePerNode: false

  dashboard:

    enabled: true

    ssl: false

  storage:

    useAllNodes: true

    useAllDevices: false

    deviceFilter: "^vd[b-z]"

  resources:

    mgr:

      requests:

        cpu: "100m"

        memory: "1Gi"

      limits:

        memory: "2Gi"

    mon:

      requests:

        cpu: "250m"

        memory: "512Mi"

      limits:

        memory: "1Gi"

  healthCheck:

    daemonHealth:

      mon:

        interval: 45s

      osd:

        interval: 60s

deviceFilter: "^vd[b-z]" tells Rook to use only additional virtio block devices (vdb, vdc, etc.) for OSDs, excluding the primary system disk (typically vda). Check your actual device names with lsblk on each node and adjust if needed.

$ kubectl apply -f ceph-cluster.yaml

$ kubectl get pods -n rook-ceph -w

Deployment can take several minutes while MON/MGR/OSD pods initialize — see the .