Registering workflows

The numerous ways in which workflows can be deployed and run can be somewhat confusing. Here is a full overview.

Local Python

$ pyflyte run my_file.py my_workflow

• Task code runs in your local Python environment.
• Supports any Python dependencies you wish, since you have full control of your local environment.
• Does not support features that require running in a cluster (S3, plugins, etc.).
• Single workflow runs immediately.
• Useful for quick testing during the development cycle.

Local cluster with default image

$ pyflyte run --remote my_file.py my_workflow

Where pyflyte is configured to point to the local cluster started with uctl demo start.

• Task code runs in the environment of the default image in your local cluster.
• Python code is dynamically overlaid into the container at runtime.
• Only supports Python code whose dependencies are installed in the default image (see here).
• Includes a local S3.
• Supports some plugins but not all.
• Single workflow runs immediately.
• Workflow is registered to a default project.
• Useful for demos.

Local cluster with custom image

$ pyflyte run --remote \
              --image my_cr.io/my_org/my_image:latest \
              my_file.py \
              my_workflow

Where pyflyte is configured to point to the local cluster started with uctl demo start.

• Task code runs in the environment of your custom image (my_cr.io/my_org/my_image:latest) in your local cluster.
• Python code is dynamically overlaid into the container at runtime
• Supports any Python dependencies you wish, since you have full control of the image.
• Includes a local S3.
• Supports some plugins but not all.
• Single workflow runs immediately.
• Workflow is registered to a default project.
• Useful for advanced testing during the development cycle.

Remote cluster with custom image

$ pyflyte run --remote \
              --image my_cr.io/my_org/my_image:latest \
              my_file.py \
              my_workflow

Where pyflyte is configured to point to your Union data plane.

• Task code runs in the environment of your custom image (my_cr.io/my_org/my_image:latest) in Union.
• Python code is dynamically overlaid into the container at runtime.
• Supports any Python dependencies you wish, since you have full control of the image.
• Full support for all features (S3, all plugins, etc.).
• Single workflow runs immediately on invocation of pyflyte command.
• Workflow is registered to a default project.
• Useful for advanced testing during the development cycle.

Remote cluster using fast registration

$ pyflyte register workflows \
       --project my_project \
       --image my_cr.io/my_org/my_image:latest

Where pyflyte is configured to point to your Union data plane.

• Task code runs in the environment of your custom image (my_cr.io/my_org/my_image:latest) in Union.
• Python code is dynamically overlaid into the container at runtime.
• Supports any Python dependencies you wish, since you have full control of the image.
• Includes full support for all features (S3, all plugins, etc.).
• Multiple workflows are registered but do not run immediately.
• Workflows are registered to the specified project.
• Workflows are run from the Web interface.
• Useful for advanced testing during the development cycle (possible but not recommended for production deployment).

Remote cluster using standard registration

First, package your workflows:

$ pyflyte --pkgs workflows \
          package \
          --image my_cr.io/my_org/my_image:latest

Then, register them:

$ uctl register files \
       --project onboarding \
       --domain development \
       --archive flyte-package.tgz \
       --version 1.0

Where pyflyte and uctl are configured to point to your Union data plane.

• Task code runs in the environment of your custom image (my_cr.io/my_org/my_image:latest) in Union.
• Python code is built into the image and not dynamically overlaid into the container, thus preserving immutability.
• Supports any Python dependencies you wish, since you have full control of the image.
• Includes full support for all features (S3, all plugins, etc.).
• Multiple workflows are registered but do not run immediately.
• Workflows are registered to the specified project.
• Workflows are run from the Web interface.
• Recommended for production deployment.