Customizing tasks and workflows#

class flytekit.TaskMetadata(cache=False, cache_serialize=False, cache_version='', cache_ignore_input_vars=(), interruptible=None, deprecated='', retries=0, timeout=None, pod_template_name=None)#

Metadata for a Task. Things like retries and whether or not caching is turned on, and cache version are specified here.

See the IDL for the protobuf definition.

Args:

cache (bool): Indicates if caching should be enabled. See Caching cache_serialize (bool): Indicates if identical (ie. same inputs) instances of this task should be executed in serial when caching is enabled. See Caching cache_version (str): Version to be used for the cached value cache_ignore_input_vars (Tuple[str, …]): Input variables that should not be included when calculating hash for cache interruptible (Optional[bool]): Indicates that this task can be interrupted and/or scheduled on nodes with

lower QoS guarantees that can include pre-emption. This can reduce the monetary cost executions incur at the cost of performance penalties due to potential interruptions

deprecated (str): Can be used to provide a warning message for deprecated task. Absence or empty str indicates

that the task is active and not deprecated

retries (int): for retries=n; n > 0, on failures of this task, the task will be retried at-least n number of times. timeout (Optional[Union[datetime.timedelta, int]]): the max amount of time for which one execution of this task

should be executed for. The execution will be terminated if the runtime exceeds the given timeout (approximately)

pod_template_name (Optional[str]): the name of existing PodTemplate resource in the cluster which will be used in this task.

Parameters:

  • cache (bool)

  • cache_serialize (bool)

  • cache_version (str)

  • cache_ignore_input_vars (Tuple[str, ...])

  • interruptible (bool | None)

  • deprecated (str)

  • retries (int)

  • timeout (timedelta | int | None)

  • pod_template_name (str | None)

to_taskmetadata_model()#

Converts to _task_model.TaskMetadata

Return type:

TaskMetadata

class flytekit.Resources(cpu=None, mem=None, gpu=None, ephemeral_storage=None)#

This class is used to specify both resource requests and resource limits.

Resources(cpu="1", mem="2048")  # This is 1 CPU and 2 KB of memory
Resources(cpu="100m", mem="2Gi")  # This is 1/10th of a CPU and 2 gigabytes of memory
Resources(cpu=0.5, mem=1024) # This is 500m CPU and 1 KB of memory

# For Kubernetes-based tasks, pods use ephemeral local storage for scratch space, caching, and for logs.
# This allocates 1Gi of such local storage.
Resources(ephemeral_storage="1Gi")

Note

Persistent storage is not currently supported on the Union backend.

Please see the User Guide for detailed examples. Also refer to the K8s conventions.

Parameters:

  • cpu (str | int | float | None)

  • mem (int | str | None)

  • gpu (int | str | None)

  • ephemeral_storage (int | str | None)

class flytekit.WorkflowFailurePolicy(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)#

Defines the behavior for a workflow execution in the case of an observed node execution failure. By default, a workflow execution will immediately enter a failed state if a component node fails.

FAIL_AFTER_EXECUTABLE_NODES_COMPLETE = 1#

Will proceed to run any remaining runnable nodes once a component node fails.

FAIL_IMMEDIATELY = 0#

Causes the entire workflow execution to fail once a component node fails.

class flytekit.PodTemplate(pod_spec=None, primary_container_name='primary', labels=None, annotations=None)#

Custom PodTemplate specification for a Task.

Parameters:

  • pod_spec (V1PodSpec | None)

  • primary_container_name (str)

  • labels (Dict[str, str] | None)

  • annotations (Dict[str, str] | None)