Since the beginning of the project, the guarantee of zero downtime and the technological independence of the fdmon (Fast Deployment Monitoring) solution have been essential elements of our specifications.
In 2019, we released, notably for our monitoring as-a-service platform, a version of fdmon without any database engine, which was based only on a distributed and resilient file system, such as CEPH or GlusterFS, and could support high network latencies and poor disk performances.
Today we are excited to announce that fdmon has its own distributed file system that we have called API-FS. API-FS is the result of a development from scratch that has prioritized the needs of fdmon in terms of storage capabilities, which were not completely met by solutions such as MinIO, S3 or equivalent.
All existing on-premises fdmon platforms will be gradually migrated to API-FS.
As its name might suggest, this distributed and resilient file system is entirely based on APIs which implement all of the fundamental functions of a file system that must meet the needs of the fdmon solution.
The robustness of API-FS as well as its scalability and resilience are linked to three principles:
- The simplicity of its architecture and its redundancy, load distribution and reconciliation algorithms (the source code of API-FS has less than 2’000 lines !)
- Intelligent management of network latency issues
- Optimization of traffic between the components of an API-FS cluster
An API-FS cluster can be distributed across the world and organized into pairs of Hubs, each Hub being able to rely on any type of underlying storage technology without limit in terms of capacity.
API-FS provides compression, deduplication and encryption features. As an option, the “Permanent Snapshotting” of API-FS allows to rollback any file or all the cluster to any point of time in the past, without notion of time granularity.
API-FS is more scalable than GlusterFS or CEPH. An unlimited number of fdmon components (fdmon Analytics, Front-End or Trending nodes) can connect to the same API-FS cluster. This makes it possible to build very large and performant monitoring and orchestration platforms (more than 1’000’000 monitored components) with a guarantee of absolute availability.
By design, API-FS does not need any consensus algorithm, nor any quorum, nor any component that acts as a “master”, to guarantee data access and data integrity.
More details will be provided in our next posts.
Leave a Reply