Passing Variables Through GitLab Pipelines

During working with GitLab multi-project pipelines and parent-child pipelines, I have encountered the problem how to pass variables through these pipelines. The GitLab documentation describes very well how to pass variables to a downstream pipeline. My challenge is how to pass variables from child to parent pipeline and how the parent pipeline can pass these variables to a downstream pipeline, that it describes in another GitLab project. Let's start, how to publish the variable that are defined in a child pipeline.

Publishing Variables of a Child Pipeline

Assume that we have a GitLab project with the following structure for the pipelines.

1.
2├── pipelines
3│   └── child-pipeline.yml
4└── .gitlab-ci.yml

The parent pipeline, defined in .gitlab-ci.yml, triggers the child pipeline, that is defined in pipelines/child-pipeline.yml.

1# .gitlab-ci.yml
2stages:
3  - a
4
5trigger-child-pipeline-job:
6  stage: a
7  trigger:
8    include: pipelines/child-pipeline.yml
9    strategy: depend

The child pipeline pipelines/child-pipeline.yml defines the variables and publishes them via the report artifact dotenv.

 1stages:
 2  - define-env
 3
 4define-env-job:
 5  stage: define-env
 6  script:
 7    - echo "MODULE_A_VERSION=1.0.0" >> .env
 8  artifacts:
 9    reports:
10      dotenv: .env

Dotenv is a standardized way to handle environment variables. Following the dotenv concept, the environment variables are stored in a file that have the following structure.

1# .env
2VARIABLE_NAME=variable-value
3MODULE_A_VERSION=1.0.0

For more information, please visit the dotenv homepage.

Let's go to the next step, how to consume this variable in the parent pipeline.

Consuming Variables From a Child Pipeline in a Parent Pipeline

The first challenge is how the parent pipeline can consume the variable, that is defined in the child pipeline (in our sample, it is the variable MODULE_A_VERSION). The child pipeline publishes its variable via a report artifact. This artifact can be used by the parent pipeline via the needs keyword. Unfortunately, it is not enough to reference the job name of the child pipeline that creates the report artifact. You also have to add a reference to the project that contains the parent and the child pipeline. Now, the parent pipeline can use the variable that is stored in the report artifact.

 1stages:
 2  - a
 3  - b
 4
 5trigger-child-pipeline-job:
 6  stage: a
 7  trigger:
 8    include: pipelines/child-pipeline.yml
 9    strategy: depend
10
11consume-env-from-child-pipeline-job:
12  stage: b
13  script:
14    - "echo $MODULE_A_VERSION"
15  needs:
16    - project: sparsick/gitlab-ci-passing-variable-pipeline
17      job: define-env-job
18      ref: main
19      artifacts: true

The next challenge is to consume this variable in a downstream pipeline that is defined in another project.

Consuming Variable From a Child Pipeline in a Downstream Pipeline of the Parent Pipeline

It exists two ways how a downstream pipeline can consume a variable from a child pipeline of its upstream pipeline.

The first way works similarly that I described in the above section. Assume, that we have the following parent pipeline that triggered a child pipeline and a downstream pipeline in another project.

 1# .gitlab-ci.yaml
 2stages:
 3  - a
 4  - b
 5
 6trigger-child-pipeline-job:
 7  stage: a
 8  trigger:
 9    include: pipelines/child-pipeline.yml
10    strategy: depend
11
12trigger-another-pipeline-job:
13  stage: b
14  trigger: sparsick/gitlab-ci-passing-variable-downstream-pipeline

The variable MODULE_A_VERSION is defined in the child pipeline like I described in the above section. The variable can be consumed by the downstream pipeline in the same way as the parent pipeline, that I described in the above section.

 1# .gitlab-ci.yaml of the downstream pipeline
 2stages:
 3  - print-env
 4
 5print-env-from-a-child-pipeline-of-the-upstream-job:
 6  stage: print-env
 7  script:
 8    - echo $MODULE_A_VERSION
 9  needs:
10    - project: sparsick/gitlab-ci-passing-variable-pipeline
11      job: define-env-job
12      ref: main
13      artifacts: true

This approach has a big disadvantage. If the job/variable/project/branch of the upstream pipeline changes its name, the downstream pipeline doesn't recognize this change automatically, and it couldn't work anymore as expected.

A second way solves this disadvantage. Here, the variable value is passed via a new variable to the downstream pipeline. Assume, that we have the following parent pipeline that triggered a child pipeline and a downstream pipeline in another project and pass a variable to the downstream pipeline.

 1# .gitlab-ci.yaml
 2stages:
 3  - a
 4  - b
 5
 6trigger-child-pipeline-job:
 7  stage: a
 8  trigger:
 9    include: pipelines/child-pipeline.yml
10    strategy: depend
11
12trigger-another-pipeline-with-var-job:
13  stage: b
14  variables:
15    ARTIFACT_VERSION: "1.0.0"
16  trigger: sparsick/gitlab-ci-passing-variable-downstream-pipeline

The downstream pipeline can use the ARTIFACT_VERSION variable in the common way.

1# .gitlab-ci.yaml of the downstream pipeline
2stages:
3  - print-env
4
5print-env-from-upstream-job:
6  stage: print-env
7  script:
8    - echo $ARTIFACT_VERSION

Now, I want, that the value of the variable MODULE_A_VERSION of the child pipeline is pass to the downstream pipeline. My first idea was to add with needs a dependency like I used it above in the consume-env-from-child-pipeline-job job. But this is invalid because trigger and needs with a reference to a project can't be used together in the same job.

 1# invalid job definition
 2trigger-another-pipeline-with-var-job:
 3  stage: b
 4  variables:
 5    ARTIFACT_VERSION: "$MODULE_A_VERSION"
 6  trigger: sparsick/gitlab-ci-test-downstream
 7  needs:
 8    - project: sparsick/gitlab-ci-test
 9      job: define-env-job
10      ref: main
11      artifacts: true

Therefore, I have to take a detour via a new job that read the variable from the child and create a new dotenv report artifact.

 1stages:
 2  - a
 3  - b
 4  - c
 5
 6trigger-child-pipeline-job:
 7  stage: a
 8  trigger:
 9    include: pipelines/child-pipeline.yml
10    strategy: depend
11
12consume-env-from-child-pipeline-job:
13  stage: b
14  script:
15    - echo "MODULE_A_VERSION=$MODULE_A_VERSION" >> .env
16  needs:
17    - project: sparsick/gitlab-ci-test
18      job: define-env-job
19      ref: main
20      artifacts: true
21  artifacts:
22    reports:
23      dotenv: .env
24
25trigger-another-pipeline-with-var-job:
26  stage: c
27  variables:
28    ARTIFACT_VERSION: "$MODULE_A_VERSION"
29  needs:
30    - job: consume-env-from-child-pipeline-job
31      artifacts: true
32  trigger: sparsick/gitlab-ci-passing-variable-downstream-pipeline

You can find the whole example on GitLab.

When you have another or better approach how to solve this described problem, let me know and please write a comment.