Lustre is an open-source, POSIX-compliant parallel distributed file system for HPC, ML training, and large-scale data processing, it separates metadata from data across dedicated server nodes, scaling aggregate throughput linearly as you add storage targets. AWS FSx for Lustre is the fully-managed equivalent, with per-TiB pricing and lock-in to AWS. This guide deploys a self-hosted Lustre cluster (1 Metadata Server + 2 Object Storage Servers + a client), configures striping/PFL, LNET, quotas/ACLs, Prometheus+Grafana monitoring, Pacemaker HA, IOR/mdtest benchmarks, and covers migrating off FSx.
Architecture
Lustre separates metadata (filenames, directory structure, permissions, stripe layout) from file data. The Metadata Server (MDS) hosts the Metadata Target (MDT); Object Storage Server (OSS) nodes host Object Storage Targets (OSTs) holding actual file data. Clients talk directly to both MDS and OSS over the LNET network protocol, so I/O flows in parallel across OSTs without a central gateway.
| AWS FSx Feature | Self-Hosted Lustre Equivalent |
|---|---|
| FSx for Lustre File System | Lustre file system (MDS + OSS cluster) |
| FSx Metadata | MDS + MDT |
| FSx Data Storage | OSS + OSTs |
| FSx SSD Storage Class | Lustre with LDISKFS on NVMe/SSD |
| FSx Intelligent-Tiering | Lustre HSM + Progressive File Layout (PFL) |
| FSx S3 Data Repository | Lustre HSM with an S3-compatible backend |
| FSx Auto Import/Export | Robinhood Policy Engine or lhsmtool_posix |
| FSx Throughput Scaling | Adding OSS/OST pairs |
| FSx Client Mount | Lustre kernel client module |
| CloudWatch Metrics | Lustre Jobstats + Prometheus exporter |
Prerequisites: 3 Linux servers (1 MDS + 2 OSS) plus a separate client server, all non-root sudo; a dedicated block storage volume on the MDS (≥10GB) and each OSS (≥20GB each); all nodes connected over a private network (10GbE minimum, InfiniBand recommended for production HPC); comfort with kernel modules and matching
kernel-develto the running kernel.
Install Lustre Packages
Lustre 2.15 LTS needs a Whamcloud-patched kernel. This uses the Enterprise Linux 8 build (Rocky/AlmaLinux/RHEL 8), check the ). lfs hsm_archive copies to S3, lfs hsm_release frees the local copy, reads transparently restore from S3. .
SOCIAL SHARE CARD GENERATOR