Passing Variables Through GitLab Pipelines
During working with GitLab multi-project pipelines and parent-child pipelines, I have encountered the problem how to pass variables through these pipelines. The GitLab documentation describes very well how to pass variables to a downstream pipeline. My challenge is how to pass variables from child to parent pipeline and how the parent pipeline can pass these variables to a downstream pipeline, that it describes in another GitLab project. Let's start, how to publish the variable that are defined in a child pipeline.
Publishing Variables of a Child Pipeline
Assume that we have a GitLab project with the following structure for the pipelines.
1.
2├── pipelines
3│ └── child-pipeline.yml
4└── .gitlab-ci.yml
The parent pipeline, defined in .gitlab-ci.yml, triggers the child pipeline, that is defined in pipelines/child-pipeline.yml.
1# .gitlab-ci.yml
2stages:
3 - a
4
5trigger-child-pipeline-job:
6 stage: a
7 trigger:
8 include: pipelines/child-pipeline.yml
9 strategy: depend
The child pipeline pipelines/child-pipeline.yml defines the variables and publishes them via the report artifact dotenv.
1stages:
2 - define-env
3
4define-env-job:
5 stage: define-env
6 script:
7 - echo "MODULE_A_VERSION=1.0.0" >> .env
8 artifacts:
9 reports:
10 dotenv: .env
Dotenv is a standardized way to handle environment variables. Following the dotenv concept, the environment variables are stored in a file that have the following structure.
1# .env
2VARIABLE_NAME=variable-value
3MODULE_A_VERSION=1.0.0
For more information, please visit the dotenv homepage.
Let's go to the next step, how to consume this variable in the parent pipeline.
Consuming Variables From a Child Pipeline in a Parent Pipeline
The first challenge is how the parent pipeline can consume the variable, that is defined in the child pipeline (in our sample, it is the variable MODULE_A_VERSION).
The child pipeline publishes its variable via a report artifact.
This artifact can be used by the parent pipeline via the needs keyword.
Unfortunately, it is not enough to reference the job name of the child pipeline that creates the report artifact.
You also have to add a reference to the project that contains the parent and the child pipeline.
Now, the parent pipeline can use the variable that is stored in the report artifact.
1stages:
2 - a
3 - b
4
5trigger-child-pipeline-job:
6 stage: a
7 trigger:
8 include: pipelines/child-pipeline.yml
9 strategy: depend
10
11consume-env-from-child-pipeline-job:
12 stage: b
13 script:
14 - "echo $MODULE_A_VERSION"
15 needs:
16 - project: sparsick/gitlab-ci-passing-variable-pipeline
17 job: define-env-job
18 ref: main
19 artifacts: true
The next challenge is to consume this variable in a downstream pipeline that is defined in another project.
Consuming Variable From a Child Pipeline in a Downstream Pipeline of the Parent Pipeline
It exists two ways how a downstream pipeline can consume a variable from a child pipeline of its upstream pipeline.
The first way works similarly that I described in the above section. Assume, that we have the following parent pipeline that triggered a child pipeline and a downstream pipeline in another project.
1# .gitlab-ci.yaml
2stages:
3 - a
4 - b
5
6trigger-child-pipeline-job:
7 stage: a
8 trigger:
9 include: pipelines/child-pipeline.yml
10 strategy: depend
11
12trigger-another-pipeline-job:
13 stage: b
14 trigger: sparsick/gitlab-ci-passing-variable-downstream-pipeline
The variable MODULE_A_VERSION is defined in the child pipeline like I described in the above section.
The variable can be consumed by the downstream pipeline in the same way as the parent pipeline, that I described in the above section.
1# .gitlab-ci.yaml of the downstream pipeline
2stages:
3 - print-env
4
5print-env-from-a-child-pipeline-of-the-upstream-job:
6 stage: print-env
7 script:
8 - echo $MODULE_A_VERSION
9 needs:
10 - project: sparsick/gitlab-ci-passing-variable-pipeline
11 job: define-env-job
12 ref: main
13 artifacts: true
This approach has a big disadvantage. If the job/variable/project/branch of the upstream pipeline changes its name, the downstream pipeline doesn't recognize this change automatically, and it couldn't work anymore as expected.
A second way solves this disadvantage. Here, the variable value is passed via a new variable to the downstream pipeline. Assume, that we have the following parent pipeline that triggered a child pipeline and a downstream pipeline in another project and pass a variable to the downstream pipeline.
1# .gitlab-ci.yaml
2stages:
3 - a
4 - b
5
6trigger-child-pipeline-job:
7 stage: a
8 trigger:
9 include: pipelines/child-pipeline.yml
10 strategy: depend
11
12trigger-another-pipeline-with-var-job:
13 stage: b
14 variables:
15 ARTIFACT_VERSION: "1.0.0"
16 trigger: sparsick/gitlab-ci-passing-variable-downstream-pipeline
The downstream pipeline can use the ARTIFACT_VERSION variable in the common way.
1# .gitlab-ci.yaml of the downstream pipeline
2stages:
3 - print-env
4
5print-env-from-upstream-job:
6 stage: print-env
7 script:
8 - echo $ARTIFACT_VERSION
Now, I want, that the value of the variable MODULE_A_VERSION of the child pipeline is pass to the downstream pipeline.
My first idea was to add with needs a dependency like I used it above in the consume-env-from-child-pipeline-job job.
But this is invalid because trigger and needs with a reference to a project can't be used together in the same job.
1# invalid job definition
2trigger-another-pipeline-with-var-job:
3 stage: b
4 variables:
5 ARTIFACT_VERSION: "$MODULE_A_VERSION"
6 trigger: sparsick/gitlab-ci-test-downstream
7 needs:
8 - project: sparsick/gitlab-ci-test
9 job: define-env-job
10 ref: main
11 artifacts: true
Therefore, I have to take a detour via a new job that read the variable from the child and create a new dotenv report artifact.
1stages:
2 - a
3 - b
4 - c
5
6trigger-child-pipeline-job:
7 stage: a
8 trigger:
9 include: pipelines/child-pipeline.yml
10 strategy: depend
11
12consume-env-from-child-pipeline-job:
13 stage: b
14 script:
15 - echo "MODULE_A_VERSION=$MODULE_A_VERSION" >> .env
16 needs:
17 - project: sparsick/gitlab-ci-test
18 job: define-env-job
19 ref: main
20 artifacts: true
21 artifacts:
22 reports:
23 dotenv: .env
24
25trigger-another-pipeline-with-var-job:
26 stage: c
27 variables:
28 ARTIFACT_VERSION: "$MODULE_A_VERSION"
29 needs:
30 - job: consume-env-from-child-pipeline-job
31 artifacts: true
32 trigger: sparsick/gitlab-ci-passing-variable-downstream-pipeline
You can find the whole example on GitLab.
When you have another or better approach how to solve this described problem, let me know and please write a comment.