Composing tasks¶

Composing tasks into sequences¶

You can compose tasks into a new task that runs them in order by declaring a sequence task.

For example:

[tool.poe.tasks]

_publish = "poetry publish"
release = [
  { cmd = "pytest --cov=src" },
  { script = "devtasks:build" },
  { ref = "_publish" },
]

And here’s an example that uses the sequence task type explicitly in a task definition:

[tool.poe.tasks._publish]
cmd = "poetry publish"

[tool.poe.tasks.release]
sequence = [
  { cmd = "pytest --cov=src" },
  { script = "devtasks:build" },
  { ref = "_publish" }
]

Composing tasks to run in parallel¶

You can compose tasks to run concurrently using the parallel task type.

For example:

[tool.poe.tasks.lint]
parallel = ["mypy", "pylint"]

Sequence and parallel tasks can also be combined, by default a list inside a sequence is interpreted as parallel task. For example:

[tool.poe.tasks.check-code]
sequence = [
  ["mypy", "pylint"], # First these two tasks run in parallel
  "pytest"            # then this task runs after both complete
]

Similarly a list inside a parallel task is interpreted as sequence task. For example:

[tool.poe.tasks.check-code]
parallel = [
  "mypy",
  "pylint",
  [ # These tasks will run in sequence, concurrently with the others
    "build-deps",
    "pytest"
  ]
]

Composing tasks into graphs¶

You can define tasks that depend on other tasks, and optionally capture and reuse the output of those tasks, thus defining an execution graph of tasks. This is done by using the deps task option, or if you want to capture the output of the upstream task to pass it to the present task then specify the uses option, as demonstrated below.

[tool.poe.tasks]
_website_bucket_name.shell = """
  aws cloudformation describe-stacks \
    --stack-name $AWS_SAM_STACK_NAME \
    --query "Stacks[0].Outputs[?(@.OutputKey == 'FrontendS3Bucket')].OutputValue" \
  | jq -cr 'select(0)[0]'
"""

[tool.poe.tasks.build-backend]
help = "Build the backend"
sequence = [
  {cmd = "poetry export -f requirements.txt --output src/requirements.txt"},
  {cmd = "sam build"},
]

[tool.poe.tasks.build-frontend]
help = "Build the frontend"
cmd = "npm --prefix client run build"

[tool.poe.tasks.shipit]
help = "Build and deploy the app"
sequence = [
  "sam deploy --config-env $SAM_ENV_NAME",
  "aws s3 sync --delete ./client/build s3://${_bucket_name}"
]
default_item_type = "cmd"
deps = ["build-frontend", "build-backend"]
uses = { _bucket_name = "_website_bucket_name" }

In this example the shipit task depends on the build-frontend build-backend, which means that these tasks get executed before the shipit task. It also declares that it uses the output of the hidden _website_bucket_name task, which means that this also gets executed, but its output is captured and then made available to the shipit task as the private variable _bucket_name.

Normally the key in the uses table will be set as an environment variable for the task, but if as in this example it has the form of a private variable (lowercase and prefixes with _) then it will only be available to reference in config (or in a cmd task), and will not be available on the environment (which prevents it from being used directly in a shell task).

Important

Note that captured output that is exposed as an environment variable via the uses is compacted to have new lines removed. This is similar to how interpolated command output is treated by bash.

Sourcing variables from dependency task output¶

Whereas uses captures task output into a named variable, the uses_env option captures the task output and parses variables from it like an env file (dotenv syntax) and exposes the resulting variables to the task at runtime. This lets a single upstream task provide zero or more variables, which it names itself, rather than the host task naming one variable per task.

This is useful for loading several related variables produced together by one command, such as credentials from a tool like aws-vault:

[tool.poe.tasks._aws-creds]
cmd  = "aws-vault export ${_profile}"
args = [{ name = "_profile", default = "dev", help = "AWS profile to use" }]

[tool.poe.tasks.deploy]
help = "Deploy with credentials sourced from aws-vault"
cmd = "terraform apply"
uses_env = "_aws-creds --profile ${_profile}"
args = [{ name = "_profile", default = "dev", help = "AWS profile to use" }]

Here _aws-creds returns several AWS_*=... lines, which are then exposed in the environment of the deploy task. uses_env also accepts a list of task invocations (including parameter expansions), applied in order so that later entries override earlier ones. Notice that the _aws-creds task optionally accepts an argument to pass to aws-vault to choose which AWS profile to use.

Note

As is conventional in poethepoet, lowercase variables prefixed with _ (e.g. _my_private_var) are considered private — accessible within task config but not exposed on the subprocess environment at runtime.