Shell task#

class flytekit.extras.tasks.shell.ShellTask(*args, **kwargs)#
execute(**kwargs)#

Executes the given script by substituting the inputs and outputs and extracts the outputs from the filesystem

Return type:

Any

post_execute(user_params, rval)#

Post execute is called after the execution has completed, with the user_params and can be used to clean-up, or alter the outputs to match the intended tasks outputs. If not overridden, then this function is a No-op

Args:

rval is returned value from call to execute user_params: are the modified user params as created during the pre_execute step

Parameters:
Return type:

Any

pre_execute(user_params)#

This is the method that will be invoked directly before executing the task method and before all the inputs are converted. One particular case where this is useful is if the context is to be modified for the user process to get some user space parameters. This also ensures that things like SparkSession are already correctly setup before the type transformers are called

This should return either the same context of the mutated context

Parameters:

user_params (ExecutionParameters)

Return type:

ExecutionParameters

class flytekit.extras.tasks.shell.OutputLocation(var, var_type, location)#
Args:

var: str The name of the output variable var_type: typing.Type The type of output variable location: os.PathLike The location where this output variable will be written to or a regex that accepts input

vars and generates the path. Of the form "{{ .inputs.v }}.tmp.md". This example for a given input v, at path /tmp/abc.csv will resolve to /tmp/abc.csv.tmp.md

Parameters:

  • var (str)

  • var_type (Type)

  • location (PathLike | str)