As large language models (LLMs) gain popularity, GPU computing resources are becoming increasingly scarce. The traditional strategy of scaling compute power based on storage demand now needs to shift to a model where storage adapts to compute availability. To ensure data consistency and simplify management, enterprises often rely on a centralized object storage point in a public cloud of a specific region to store all model data. However, when it comes to scheduling compute tasks across regions, manual data copying and migration are not only costly but also bring about significant complexity, especially regarding permissions and maintenance.
The mirror file system feature in , comprising a client, object storage, and a metadata engine.
The key difference is that while the Community Edition typically uses third-party databases like Redis, TiKV, or PostgreSQL as its metadata engine, the Enterprise Edition features a proprietary, high-performance metadata service. The metadata engine consists of one or more Raft groups, as shown below:
At this point, the metadata service of the mirror is part of the same Raft group as the source metadata service, but its role is a learner. When metadata updates at the source, the service automatically pushes change logs to the mirror, where they are replayed. This means the existence of the mirror file system does not impact the performance of the source file system. However, the metadata version on the mirror will lag slightly behind.
Data mirroring is also done asynchronously, with designated nodes automatically synchronizing the data. The difference is that, for clients in the mirrored region, only the local metadata is accessed, but both regions' object storage can be accessed. When reading data, the client prioritizes reading from the local region. If the required object cannot be found, it will try to retrieve it from the source region.
Generally, since data tends to be large and costly to duplicate, a recommended approach is to only mirror the metadata while setting up a distributed caching system in the mirrored region to speed up data access. Here's a diagram showing the setup:
Here’s an example of creating a new file (a create request) to explain the workflow in more detail.
and community on Slack.
SOCIAL SHARE CARD GENERATOR