{"serverless": "Serverless", "byoc": "BYOC"}

["serverless", "byoc"]

serverless

Remote dependencies with ImageSpec

During the development cycle you will want to be able to run your workflows both locally on your machine and remotely on Union, so you will need to ensure that the required dependencies are installed in both environments.

Here we will explain how to set up the dependencies for you workflow to run remotely on Union. For information on how to make your dependencies available locally, see Local dependencies.

ImageSpec

When a workflow is deployed to Union, each task is set up to run in its own container in the Kubernetes cluster. You specify the dependencies as part of the definition of the container image to be used for each task using the ImageSpec class.

In the template code generated when you did union init, you will see a ImageSpec block in the script. The relevant part for our purposes is:

"""Basic Union workflow template."""

from flytekit import task, workflow, ImageSpec

# ImageSpec defines the container image used for the Kubernetes pods that run the tasks in Union.
image_spec = ImageSpec(

    # The name of the image
    name="basic-union-image",

    # Use the requirements.txt to define the packages to be installed in the the image
    requirements="requirements.txt",

The init template includes all available ImageSpec parameters. Apart from the two parameters above (name and requirements), they are all commented out. The two uncommented ones are the only ones needed for this example.

Building the task container image

When you register your workflow code, Union builds the container images specified by the ImageSpec blocks using its ImageBuilder service in the cloud. These images are then stored in Union’s own container registry. All of this is done transparently and does not require any set up by the user.

When a task that uses that image is executed on Union, the image will be pulled from Union’s native registry and installed in the container that runs the task.

Note

Transparent building and storing of images with ImageBuilder is currently only supported on Union Serverless. For information on how to set up image handling on Union BYOC, see the BYOC version of this page and Setting up container image handling.