Compile time and serialization settings#

class flytekit.configuration.Image(name, fqn, tag=None, digest=None)#

Image is a structured wrapper for task container images used in object serialization.

Attributes:

name (str): A user-provided name to identify this image. fqn (str): Fully qualified image name. This consists of

  1. a registry location

  2. a username

#. a repository name For example: hostname/username/reponame

tag (str): Optional tag used to specify which version of an image to pull digest (str): Optional digest used to specify which version of an image to pull

Parameters:

  • name (str)

  • fqn (str)

  • tag (str | None)

  • digest (str | None)

property full: str#

” Return the full image name with tag or digest, whichever is available.

When using a tag the separator is : and when using a digest the separator is @.

static look_up_image_info(name, image_identifier, allow_no_tag_or_digest=False)#

Creates an Image object from an image identifier string or a path to an ImageSpec yaml file.

This function is used when registering tasks/workflows with Admin. When using the canonical Python-based development cycle, the version that is used to register workflows and tasks with Admin should be the version of the image itself, which should ideally be something unique like the git revision SHA1 of the latest commit.

Parameters:

  • name (str)

  • image_identifier (str) – Either the full image identifier string e.g. somedocker.com/myimage:someversion123 or a path to a file containing a ImageSpec.

  • allow_no_tag_or_digest (bool)

Return type:

Image

property version: str | None#

Return the version of the image. This could be the tag or digest, whichever is available.

class flytekit.configuration.ImageConfig(default_image=None, images=None)#

We recommend you to use ImageConfig.auto(img_name=None) to create an ImageConfig. For example, ImageConfig.auto(img_name=””ghcr.io/flyteorg/flytecookbook:v1.0.0””) will create an ImageConfig.

ImageConfig holds available images which can be used at registration time. A default image can be specified along with optional additional images. Each image in the config must have a unique name.

Attributes:

default_image (Optional[Image]): The default image to be used as a container for task serialization. images (List[Image]): Optional, additional images which can be used in task container definitions.

Parameters:

  • default_image (Image | None)

  • images (List[Image] | None)

classmethod auto(config_file=None, img_name=None)#

Reads from config file or from img_name Note that this function does not take into account the flytekit default images (see the Dockerfiles at the base of this repo). To pick those up, see the auto_default_image function..

Parameters:

  • config_file (str | ConfigFile | None)

  • img_name (str | None)

Returns:

Return type:

ImageConfig

find_image(name)#

Return an image, by name, if it exists.

Return type:

Image | None

classmethod from_images(default_image, m=None)#

Allows you to programmatically create an ImageConfig. Usually only the default_image is required, unless your workflow uses multiple images

ImageConfig.from_dict(
    "ghcr.io/flyteorg/flytecookbook:v1.0.0",
     {
          "spark": "ghcr.io/flyteorg/myspark:...",
          "other": "...",
     }
)
Returns:

Parameters:

  • default_image (str)

  • m (Dict[str, str] | None)

static validate_image(_, param, values)#

Validates the image to match the standard format. Also validates that only one default image is provided. a default image, is one that is specified as default=<image_uri> or just <image_uri>. All other images should be provided with a name, in the format name=<image_uri> This method can be used with the CLI

Parameters:

  • _ (Any) – click argument, ignored here.

  • param (str) – the click argument, here should be “image”

  • values (tuple) – user-supplied images

Returns:

Return type:

ImageConfig

class flytekit.configuration.SerializationSettings(image_config, project=None, domain=None, version=None, env=None, git_repo=None, python_interpreter='/opt/venv/bin/python3', flytekit_virtualenv_root=None, fast_serialization_settings=None, source_root=None)#

These settings are provided while serializing a workflow and task, before registration. This is required to get runtime information at serialization time, as well as some defaults.

Attributes:

project (str): The project (if any) with which to register entities under. domain (str): The domain (if any) with which to register entities under. version (str): The version (if any) with which to register entities under. image_config (ImageConfig): The image config used to define task container images. env (Optional[Dict[str, str]]): Environment variables injected into task container definitions. flytekit_virtualenv_root (Optional[str]): During out of container serialize the absolute path of the flytekit

virtualenv at serialization time won’t match the in-container value at execution time. This optional value is used to provide the in-container virtualenv path

python_interpreter (Optional[str]): The python executable to use. This is used for spark tasks in out of

container execution.

entrypoint_settings (Optional[EntrypointSettings]): Information about the command, path and version of the

entrypoint program.

fast_serialization_settings (Optional[FastSerializationSettings]): If the code is being serialized so that it

can be fast registered (and thus omit building a Docker image) this object contains additional parameters for serialization.

source_root (Optional[str]): The root directory of the source code.

Parameters:

  • image_config (ImageConfig)

  • project (str | None)

  • domain (str | None)

  • version (str | None)

  • env (Dict[str, str] | None)

  • git_repo (str | None)

  • python_interpreter (str)

  • flytekit_virtualenv_root (str | None)

  • fast_serialization_settings (FastSerializationSettings | None)

  • source_root (str | None)

class Builder(project: 'str', domain: 'str', version: 'str', image_config: 'ImageConfig', env: 'Optional[Dict[str, str]]' = None, git_repo: 'Optional[str]' = None, flytekit_virtualenv_root: 'Optional[str]' = None, python_interpreter: 'Optional[str]' = None, fast_serialization_settings: 'Optional[FastSerializationSettings]' = None, source_root: 'Optional[str]' = None)#
Parameters:

  • project (str)

  • domain (str)

  • version (str)

  • image_config (ImageConfig)

  • env (Dict[str, str] | None)

  • git_repo (str | None)

  • flytekit_virtualenv_root (str | None)

  • python_interpreter (str | None)

  • fast_serialization_settings (FastSerializationSettings | None)

  • source_root (str | None)

static default_entrypoint_settings(interpreter_path)#

Assumes the entrypoint is installed in a virtual-environment where the interpreter is

Parameters:

interpreter_path (str)

Return type:

EntrypointSettings

new_builder()#

Creates a SerializationSettings.Builder that copies the existing serialization settings parameters and allows for customization.

Return type:

Builder

property serialized_context: str#
Returns:

returns the serialization context as a base64encoded, gzip compressed, json strinnn

should_fast_serialize()#

Whether or not the serialization settings specify that entities should be serialized for fast registration.

Return type:

bool

static venv_root_from_interpreter(interpreter_path)#

Computes the path of the virtual environment root, based on the passed in python interpreter path for example /opt/venv/bin/python3 -> /opt/venv

Parameters:

interpreter_path (str)

Return type:

str

with_serialized_context()#

Use this method to create a new SerializationSettings that has an environment variable set with the SerializedContext This is useful in transporting SerializedContext to serialized and registered tasks. The setting will be available in the env field with the key SERIALIZED_CONTEXT_ENV_VAR :return: A newly constructed SerializationSettings, or self, if it already has the serializationSettings

Return type:

SerializationSettings

class flytekit.configuration.FastSerializationSettings(enabled=False, destination_dir=None, distribution_location=None)#

This object hold information about settings necessary to serialize an object so that it can be fast-registered.

Parameters:

  • enabled (bool)

  • destination_dir (str | None)

  • distribution_location (str | None)