Pipelines

Pipeline is a set of Tasks executing in selected order, with optional addition of error handling. Modifiers are changing behavior of Task execution, by implementing fallbacks, retries and error notifications.

Tip

Modifiers can be used together e.g. @retry + @rescue , @retry + @error

Basic pipeline

Basically the pipeline is a set of Tasks, it does not need to define any error handling.

Tip

Treat Pipeline as a shell command invocation - in practice a Pipeline is an alias, it is similar to a command executed in command line but a little bit more advanced.

The comparison isn’t abstract, that’s how Pipelines works and why there are shell examples of Pipelines.

YAML Python Shell

version: org.riotkit.rkd/yaml/v2

# ...

pipelines:
    :perform:
        tasks:
            - task: :start
            - task: :do-something
            - task: :stop

@retry

Simplest modifier that retries each failed task in a block up to maximum of N times.

The example actually combines @retry + @rescue . But @retry can be used alone.

Warning

When retrying a Pipeline inside of a Pipeline, then all that child Pipeline Tasks will be repeated, it will work like a @retry-block for inherited Pipeline.

Syntax:

YAML Python Shell

version: org.riotkit.rkd/yaml/v2

# ...

pipelines:
    :start:
        tasks:
            - block:
                  retry: 1  # retry max. 1 time
                  rescue: [':app:clear-cache', ':app:start']
                  tasks:
                      - task: [':db:start']
                      - task: [':app:start']
            - task: [':logs:collect', '--app', '--db', '--watch']

Example workflow:

@retry-block

Works in similar way as @retry , the difference is that if at least one task fails in a block, then all tasks from that blocks are repeated N times.

Example workflow:

@error

Executes a Task or set of Tasks when error happens. Does not affect the final result. After error task is finished the whole execution is stopped, no any more task will execute.

Syntax:

YAML Python Shell

version: org.riotkit.rkd/yaml/v2

# ...

pipelines:
    :upgrade:
        tasks:
            - task: ":db:backup"
            - task: ":db:stop"
            - block:
                  error: [':notify', '--msg="Failed"']
                  tasks:
                      - task: [':db:migrate']
            - task: [":db:start"]
            - task: [":notify", '--msg', 'Finished']

Example workflow:

@rescue

Defines a Task that should be ran, when any of Task from given block will fail. Works similar as @error , but with difference that @rescue changes the result of pipeline execution.

Tip

When @rescue succeeds, then we assume that original Task that failed is now ok.

Warning

When rescuing a whole Pipeline inside other Pipeline, then failing Task will be rescued and the rest of Tasks from child Pipeline will be skipped.

Example workflow:

Order of modifiers execution

1. @retry: Each task is repeated until success or repeat limit

2. @retry-block: Whole block is repeated until success or repeat limit

3. @rescue: A rescue attempt of given Task or inherited Pipeline is attempted

4. @error: An error notification is sent, when all previous steps failed

Pipeline in Pipeline

A Pipeline inside a Pipeline is when we have defined a Pipeline, and one of it’s called Tasks is other Pipeline.

version: org.riotkit.rkd/yaml/v2

# ...

pipelines:
    :prepare_disk_space:
        tasks:
            - task: ":db:clear_buffers"
            - task: ":db:clear_temporary_directory"

    :upgrade:
        tasks:
            - task: ":db:backup"
            - task: ":db:stop"
            - task: ":prepare_disk_space"   # HERE IS OUR INHERITED PIPELINE
            - block:
                  error: [':notify', '--msg="Failed"']
                  tasks:
                      - task: [':db:migrate']
            - task: [":db:start"]
            - task: [":notify", '--msg', 'Finished']

Pipeline in Pipeline - how modifiers behave

Having Pipeline called inside other Pipeline, the inherited one is treated similar to a Task.

:pipeline_1 (@retry, @error, @rescue)
    :task_1
    :pipeline_2 (@retry, @rescue, ...)
        :subtask_1
        :subtask_2
    :task_3

When :pipeline_2 fails then at first - :pipeline_2 modifiers are called. In case, when :pipeline_2 modifiers didn’t rescue the Pipeline, then modifiers from parent level :pipeline_1 are called.

Warning

When modifiers on main level of Pipeline fails, then parent Pipeline modifiers are inherited that behaves differently.

1. @retry from parent becomes a @retry-block of whole Pipeline (we retry a Pipeline now)

2. @rescue after rescuing a Task inside child Pipeline is skipping remaining Tasks in child Pipeline

Python syntax reference (API)

class rkd.core.api.syntax. Pipeline ( name : str , to_execute : List [ Union [ str , rkd.core.api.contract.PipelinePartInterface ] ] , env : Optional [ Dict [ str , str ] ] = None , description : str = '' )

Task Caller

Has a name like a Task, but itself does not do anything than calling other tasks in selected order

class rkd.core.api.syntax. PipelineTask ( task : str , * task_args )

Represents a single task in a Pipeline

from rkd.core.api.syntax import Pipeline

PIPELINES = [
    Pipeline(
        name=':build',
        to_execute=[
            Task(':server:build --with-bluetooth'),
            Task(':client:build', '--with-bluetooth')
        ]
    )
]

class rkd.core.api.syntax. PipelineBlock ( tasks : List [ rkd.core.api.syntax.PipelineTask ] , retry : Optional [ int ] = None , retry_block : Optional [ int ] = None , error : Optional [ str ] = None , rescue : Optional [ str ] = None )

Represents block of tasks

Example of generated block:

{ @ retry 3} :some-task {/@}

from rkd.core.api.syntax import Pipeline, PipelineTask as Task, PipelineBlock as Block, TaskDeclaration

Pipeline(
    name=':error-handling-example',
    description=':notify should be invoked after "doing exit" task, and execution of a BLOCK should be interrupted',
    to_execute=[
        Task(':server:build'),
        Block(error=':notify -c "echo 'Build failed'"', retry=3, tasks=[
            Task(':docs:build', '--test'),
            Task(':sh', '-c', 'echo "doing exit"; exit 1'),
            Task(':client:build')
        ]),
        Task(':server:clear')
    ]
)