Skip to content

Multi-cluster

Union Cloud supports the configuration of multi-cluster data planes in order to provide isolation either at the Flyte domain or Flyte project level.

When configuring your data plane you can map each cluster to either a Flyte domain or a Flyte project (more specifically, a Flyte project-domain pair).

  • If you choose to map each cluster to a Flyte domain, then you would have one each for each domain. For example:
    • development
    • staging
    • production
  • If you choose to map each cluster to a Flyte project, then you would have one for each project-domain pair. For example:
    • project-1-development
    • project-2-development
    • project-3-development
    • project-1-staging
    • project-2-staging
    • project-3-staging
    • project-1-production
    • project-2-production
    • project-3-production

The precise set of the clusters, of course, depends on the number of Flyte domains and projects that you have.

Limitations of project per cluster

Note that if you choose to map each Flyte project to its own cluster, you will need to define the set of such projects upfront. This is because the Flyte project will have to be created when the cluster is set up.

If you also want the ability to create projects on demand, this can be supported buy having an additional default cluster. Any projects created after onboarding will be created in that default cluster.

Data and metadata isolation

Each cluster is isolated within its own AWS account or Google project, and therefore provides the level of compute and data isolation intrinsic to that arrangement. Specifically, execution-time isolation per cluster is maintained for both compute and user data stored in blob store (or other configured storage).

In addition, metadata specific to the internals of Flyte can be either isolated or shared across clusters, depending on the configuration you choose.

Specifically, the sharing of metadata is controlled by the cluster pool to which a cluster belongs. If two clusters are in the same cluster pool, then they must share the same metadata bucket. If they are in different cluster pools, then they must have different metadata buckets. You could, for example, have a single metadata bucket for all your development clusters, and a separate one for all your production clusters, by grouping the clusters into cluster pools accordingly. Alternatively you could have a separate metadata bucket for each cluster, by putting each cluster in its own cluster pool.

You specify the cluster pool to which a cluster belongs when you configure your data plane with the help of the Union Cloud team.