Configuration#

Configuration uses#

Configuration usage can roughly be bucketed into the following areas:

  • Compile-time settings: These are settings like the default image and named images, where to look for Flyte code, etc.

  • Platform settings: Where to find the Flyte backend (Admin DNS, whether to use SSL).

  • Registration runtime settings: These are things like the K8s service account to use, a specific S3/GCS bucket to write off-loaded data (dataframes and files) to, notifications, labels & annotations, etc.

  • Data access settings: Settings such as custom S3 endpoint settings, backoff/retry behavior for accessing S3 or GCS, key and password, and so on.

  • Other settings: statsd configuration, which is a run-time applicable setting, but is not necessarily relevant to the Flyte platform.

Configuration sources#

Command line arguments#

We recommended using command line arguments to set configuration values most of the time. See the union CLI documentation for a full list of arguments for each command.

Configuration files#

YAML format#

If you are using a configuration file, we recommend YAML format. The ~/.union/config.yaml configuration file contains settings for both uctl and flytekit. To create a boilerplate ~/.union/config.yaml file, run uctl config init, and to learn about all of the configuration YAML options, run uctl --help.

See example config.yaml file
config.yaml#
admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///flyte.mycorp.io
  authType: Pkce
  insecure: true
  clientId: propeller
  scopes:
    - all
storage:
  connection:
    access-key: minio
    endpoint: http://localhost:30084
    secret-key: miniostorage
images:
  xyz: docker.io/xyz:latest
  abc: docker.io/abc
  bcd: docker.io/bcd@sha256:26c68657ccce2cb0a31b330cb0hu3b5e108d467f641c62e13ab40cbec258c68d

INI format#

Warning

The INI format configuration is considered a legacy configuration format. We recommend using the YAML format instead if you’re using a configuration file.

A configuration file for flytekit. By default, flytekit will look for a file in two places:

  1. First, flytekit looks in the Python interpreter’s working directory for a file named flytekit.config.

  2. Next, flytekit looks in the home director as detected by Python for a file named ~/.flyte/config.

See example flytekit.config file
flytekit.config#
[sdk]
workflow_packages=module1,module2

[platform]
url=flyte.mycorp.io
insecure=true

[auth]
kubernetes_service_account=demo
raw_output_data_prefix=s3://my-bucket

[images]
xyz=docker.io/xyz:latest
abc=docker.io/abc

Python objects#

Config object#

Note

You can use a Config object directly, for example, when initializing a UnionRemote object. See Creating a UnionRemote object for examples on how to specify a Config object.

Config

This the parent configuration object and holds all the underlying configuration object types.

Compile time and serialization settings#

Image

Image is a structured wrapper for task container images used in object serialization.

ImageConfig

We recommend you to use ImageConfig.auto(img_name=None) to create an ImageConfig.

SerializationSettings

These settings are provided while serializing a workflow and task, before registration.

FastSerializationSettings

This object hold information about settings necessary to serialize an object so that it can be fast-registered.

Note

These are compile-time and serialization settings that are usually passed in as flags to commands like union package or union register.

The image configurations are typically either passed in via an --image flag, or can be specified in a YAML or ini configuration file.

Execution time settings#

You usually won’t need to worry about these configurations, as they are typically set by FlytePropeller or FlyteAdmin. The configurations below are useful for authenticating to a Union backend, configuring data access credentials, secrets, and statsd metrics.

PlatformConfig

This object contains the settings to talk to a Union backend (the DNS location of your Admin server basically).

StatsConfig

Configuration for sending statsd.

SecretsConfig

Configuration for secrets.

S3Config

S3 specific configuration

GCSConfig

Any GCS specific configuration.

DataConfig

Any data storage specific configuration.

Environment variables#

You can specify environment variables at compile time, but when your task is run, FlytePropeller will also set configuration to ensure correct interaction with the platform. The environment variables must be specified with the format FLYTE_{SECTION}_{OPTION}, all in upper case. For example, to specify the PlatformConfig.endpoint setting, the environment variable would be FLYTE_PLATFORM_URL.

Note

Environment variables won’t work for specifying an image, which needs to be specified with the union package --image ... option or in a configuration file.