About CICD library projects

GitLab CICD provides a powerful general-purpose automation platform for running a wide variety of workloads, including Continuous Integration, Continuous Delivery, ModelOps, and GitOps. The platform also includes a powerful collection of templates and components to configure CICD rapidly for common use cases.

Organizations sometimes require specific, highly-customized configurations to meet sophisticated business and technical requirements. In such cases, a CICD library project provides a useful mechanism to develop, manage, and maintain pipeline configuration, job images, and scripts.

Similar to the CICD Catalog, but different. A CICD library project leverages key technology within the Components initiative at GitLab, and in some cases even uses components from the catalog, but is managed within an organization for use in its projects.

Always a work in progress. Always a draft.

Framework

The ProCICD initiative includes a rough "framework" that can be used as a starting point for a CICD library project. Parts of the framework include:

• ProCICD guide - This document
• ProCICD base library - Starting point for custom CICD libraries
• Token tool - Script to encode a GitLab deploy token
• test projects - A few projects designed to exercise specific capabilities of the framework

Terminology

ProCICD uses specific terminology, some of which differs slightly from GitLab convention.

• CICD - Continuous Integration and Continuous Delivery. Sometimes written "CI/CD" or "CI-CD" but we prefer to leave out the punctuation to emphasize that GitLab pipelines perform all operations under one automation system (and it saves a keystroke).
• Element - A YAML file containing GitLab CICD job definitions and other pipeline configuration, referenced via include:project: and include:file:. We chose the term "element" to differentiate from "templates" and "components" (although the meaning overlaps).
• Object - Any key-value pair in YAML, including the entire value.
• Base job - CICD job whose names begin with a dot ("."), referred to as "hidden job" in the official GitLab documentation, that act in a similar way to base classes in object-oriented programming (the word "hidden" implies invisible activity so can confuse us).
• Pipeline rules - Rules that determine whether an entire pipeline is run, and sets variables based on conditions. Pipeline rules use workflow:rules: objects in YAML but we find "pipeline rules" a more accurate term than "workflow rules".
• Delivery - Includes "deployment" and/or "distribution" (see below); conveniently, all 3 words start with the letter "D".
• Deploy - Push an application to a server/hosting/cluster environment where it will operate.
• Distribute - Push an application or library to a storage location such as a container or package registry.
• Release - Generally refers to version numbering and Git tagging operations, distinct from actual deployment and distribution

GitLab CICD documentation and practice uses two other terms that lie outside the scope of custom CICD library contents, which is why we use the term "element" to refer to YAML files in the library itself. Still, understanding helps.

• Template - GitLab-produced YAML that comes with the GitLab system, viewable on GitLab.com, referenced via include:template:. The template collection will soon be deprecated in favour of components (below).
• Component - Newer replacement for "templates" that may be developed and contributed by the GitLab community, viewable in the components catalog, referenced via include:component:.

The framework suggests a gitlab directory for "wrapping" templates and components if required.

Document and element statuses

We've tried to standardize the "status" at the tops of pages (in this guide) and elements (in the base library). The statuses are:

• Sketchy: Author brainstormed ideas and recorded them - read with caution
• Rough: Undergone lightweight review and/or testing - reference it to discern broad themes or extract examples
• Draft: Some structure and useful information - test and refine before use
• Stable: Someone other than the author has read or used this - proceed with confidence

By Francis Potter, based on over 5 years of experience as a GitLab CICD professional and work on personal projects.

Freely distributed. None of the content herein represents official GitLab marketing or documentation. Documents, code, and configuration might change at any time without notice. Intended for customization and use by experienced GitLab practitioners.

Guide published using MDBook and the ProCICD base library itself.

General guidelines

Based on experience, the guidelines below apply to all uses of GitLab CICD, regardless of whether in a library project.

Know the pipeline types

GitLab CICD runs three distinct types of pipelines:

• Branch pipelines, which run by default on any push that changes a branch, as well as most manual and scheduled pipeline runs
• Tag pipelines, which run by default on any push that changes a tag
• Merge Request pipelines which run under some conditions on pushes to a branch managed by a merge request, as well as manual pipeline runs from the merge request form

Without attention, a project can end up with multiple pipelines running on the same push, pipelines triggering other pipelines in a loop, or simple confusion. The patterns described under pipeline conditions help to simplify things.

Leverage stages for visual flow and dependency openness

Both needs: and stage: can be used to define the order in which jobs run. Some guidelines:

• Without needs: or dependencies: a job runs when all the previous stages finish, and gets all the artifacts from all previous stages
• If a job has any needs: at all, it will run when the needed jobs are done, potentially out of stage order - also then only gets artifacts from listed jobs
• Use dependencies: to optimize artifact downloads (rarely necessary)
• needs: can be used within a stage, but can look a little confusing in the web UI (where jobs are sorted alphabetically by default)
• stage: allows control over how jobs are grouped together in both the pipeline page and the pipelines list
• Define meaningful stages and let the artifacts flow to take advantage of the setenv/getenv pattern

Use base jobs to avoid global defaults

In CICD YAML, the globally-defined image: keyword is deprecated in favour of the default: keyword. Just keep in mind that the default: will apply to every job in the pipeline that doesn't define it's own image (or control defaults). Once the use of GitLab CICD becomes more sophisticated, with imports: and components, the global settings become confusing. Instead, use base jobs (called hidden jobs in the documentation) to reuse configuration. A simple example:

.maven:
  image: maven:3.8.7-openjdk-18-slim
  variables:
    MAVEN_CLI_OPTS: >-
      -Dmaven.repo.local=.m2/repository
  cache:
    paths:
      - .m2/repository
    key: maven

maven-build:
  extends: [.maven]
  script:
    - mvn build ${MAVEN_CLI_OPTS}

maven-deploy:
  extends: [.maven]
  ...

Avoid Git pushes in pipelines

I advise against performing Git operations (other than tagging) in a CICD pipeline, for several reasons:

• The Git protocol is designed for source code repositories; CICD operations happen strictly downstream
• GitLab's Job artifacts, Package registry, and Container registry capabilities provide managed, access-control mechanisms to store downstream information, avoiding the need to commit files such as build products back into a Git repository
• GitLab's Releases and Environments capabilities provide mechanisms to track, control, and roll back deployments, avoiding the need for a "release tracking branch"
• Without careful attention to detail, it's possible to accidentally set up CICD loops, where a Git push triggers a CICD run, which triggers another Git push, etc

In the event a Git push operation seems necessary, the operation would require either a deploy key with "Read-write" permission or an access token with write_repository permission. Access tokens expire (maximum expiration period of a year) and must be rotated. Read the documentation in detail before selecting and implementing an approach.

Base library

The ProCICD base library can be used as a starting point for custom CICD libraries.

Getting Started

To begin, copy the base library into a gropu within the organization's control. Some options:

• Clone it - allows easy upstreaming of changes but exposes more of the history of ProCICD
  1. Use git clone to make a local copy of procicd/lib.
  2. Use git remote to add a new remote for the custom library project.
  3. Make initial adjustments (below)
  4. Push to the new remote
• Copy it - makes upstreaming changes more difficult, but keeps the division clean.
  1. git clone git@gitlab.com:your/new/library.git (replace path to new GitLab project for library)
  2. cd library (replace with correct name)
  3. git archive --remote git@gitlab.com:procicd/lib.git v1.2.2 --output procicd.tar (replace version)
  4. tar xfv procicd.tar
  5. rm procicd.tar
  6. git add -a
  7. git commit -m 'ProCICD v1.2.2'
  8. git push

In either case, edit the README to replace procicd/lib with the path to the custom library project itself. Something like the following (on MacOS) will work:

find ./ -type f -exec sed -i '' -e 's@procicd/lib@your/new/library@g' {} +

Directories

Within a CICD library (including the base library), we suggest the following directories:

• docker: Boilerplate job for docker builds. Used for custom job image prebuilds; can also be used in projects if needed.
• gitlab: Placeholder/example directory for wrappers around the standard instance-level "templates" as needed for extension or customization.
• images: Custom job images for prebuild (including Dockerfiles and .gitlab-ci.yml files for performing the builds).
• java: Elements developed for Java and Maven projects. Maturity: low
• python: Elements developed for Python projects. Maturity: high
• release: Commonly used elements related to version numbering and release tagging, available for use as needed.
• scratch: Mostly outdated reference material such as scripts from legacy CICD systems - don't use them in include:, and consider deleting them when no longer of use.
• static: Elements for static site generation (GitLab Pages). Maturity: medium
• util: Commonly used elements with broad utility, including:
  • save/load CICD state and generic registry packages.
  • all-global.gitlab-ci.yml - Include it in everything. Described below.
• Name other directories in the library after the underlying technology (e.g. ruby).
• The root of the library project contains a few special files:
  • .gitlab-ci.yml - The CICD pipeline definition for the library itself, which is capable of prebuilding custom job images (when they change) and handling release tagging of the library as well as pushing constants.yml to the generic package registry.
  • constants.yml - Values required by various jobs and prepackaged for download. (TODO)
  • VARIABLES.md - Index to global variables.

We left in some boilerplate elements not actually used today, in case they become useful in the future.

Elements (YAML files)

The library contains several specific types of elements (YAML files), distinguished by file name, indicating their intended use.

• pipe: complete pipeline definition, ready to be referenced as an include: (with or without inputs:) in a project's CICD YAML. pipe elements may contain one or more phase or job elements. Example: python/pypi-library-pipe.gitlab-ci.yml.
• phase: set of jobs that serve a common purpose within a pipeline, which might cross multiple stages, often pass values between each other in artifacts, and are typically referenced from a pipe element with include:local:. phase elements may contain one or more job elements.
• job: exactly one job, which typically contains a job-label input that gets suffixed to job names to support repetition (see below). Example: python/poetry-build-job.yml.
• base: only base ("hidden") jobs (starting with '.') to define YAML objects shared by jobs in multiple elements. Example: python/venv-base.gitlab-ci.yml.
• global: elements that may be included only once in a pipeline, to include project-wide jobs or root variables: with options: and/or description:. Example: release/version-patch-scheme-global.gitlab-ci.yml.

Use the following format for element filenames: <parts>-<type>.gitlab-ci.yml where "parts" are all-lower hyphen-separated name parts and "type" is one of the types described above.

The full .gitlab-ci.yml filename extension (different from the GitLab "components" convention) allows Visual Studio Code (with the GitLab extension) to see the files as GitLab CICD configuration and handles linting appropriately.

Includes and inputs

Library elements are designed to be include:'d in project CICD YAML configurations, and elements might include: other elements, which might in turn include: yet more elements. So inclusion is nested.

Included elements use the inputs: mechanism (introduced as part of the "components" intiative at GitLab) to accept parameters, similar to function arguments in a traditional programming language.

The library uses the kabab-case naming convention for inputs: (hyphen-separated, all lower).

Try to avoid propagating inputs (i.e. having an input reference another input).

CICD Variables

In GitLab CICD, Variables are named values that may be set in group settings, project settings, on the Run Pipeline form, via API, via Git push options, globally in YAML, within a rules: item, within a job, or inside a script (among other ways). Variables can be used for if expressions and in scipts as shell environment variables, among other ways.

If this is your first time working with variables in CICD pipelines (or just for a refresher), consult the guide to variable precedence in the docs.

Because of the flexibility and power of CICD variables, the library applies some conventions to help make sense of them.

• Use the ALL_CAPS_WITH_UNDERSCORES convention for all CICD variables and environment variables defined in a script item (clearly differentiates them from inputs to avoid confusion).
• For variables set at project or group level and required by a pipeline, define them as a global variable (root level of YAML, either in the all-global element or in one of the pipe elements) so other engineers can see the variable is expected - typically the values may be an empty string, or contain a default value. Example: DEFAULT_JOB_IMAGE_REGISTRY in util/all-global.gitlab-ci.yml.
• Note: Because we set default empty string values for variables, to test for a null value, test for '' rather than null in rules:.

See VARIABLES.md for details on global variables required by the library.

Jobs

A few thoughts on best practices for individual jobs:

• Use base jobs and extends: to minimize duplication
• Use numbers at the beginning of job names where necessary to force ordering in the web UI when viewing by stage.
• Use the display function to highlight key values in job log outputs.
• Job-level CICD variables make a handy place to keep error message text.

Library versioning

The library supports versioning in multiple ways:

• CICD element (YAML) versioning uses Git tags and semantic versioning; refs can be used in the include: object in projects to specify a version of the library. To increment version: Set the VERSION_INCEREMENT variable in the Run Pipeline form and run a fresh pipeline.
• Custom job image versions can be changed with the CUSTOM_JOB_IMAGE_VERSION variable, though usually not necessary - helpful for e.g. clearing Docker caches. Projects always get the latest version with the associated value, regardless of which version of the library they reference.
• constants.yml goes into the generic package registry. Projects always get the very latest version, regardless of which version of the library they reference.

Patterns

The ProCICD framework uses several patterns, akin to software design patterns, that reflect useful approaches to addressing common CICD requirements. In many cases, the elements in the base library can be copied and used as provided to implement the pattern; in other cases custom implementations are required.

Documented patterns include:

• Centralized Globals
• Including other files
• Custom job images
• Job labels
• CICD State
• Constants
• Variable setenv/getenv
• Version numbering
• Phase skipping
• Pipeline conditions
• Static docs sites

TODO: Add pattern for variables as error messages, outputting commands before running, etc

Centralized Globals

STATUS Sketchy: Author brainstormed ideas and recorded them - read with caution

We use the term "globals" to refer to YAML objects at the "root" level (leftmost column) of an element's YAML file that is NOT a job. Examples of "global" objects include:

• stages:
• default:
• workflow:
• global variables:
• base jobs (starting with ".")

Globals apply to all the jobs in a pipeline, including other elements added with include:. Furthermore, few of them are extensible (stages: for example), so different instances of the object in different elements might unintentionally override each other.

The all-global element

The base library includes an element at util/all-global intended to hold global values for use in all pipelines.

Recommendations

• Include util/all-global in every -pipe element

Usage

See the python/pypi-library-pipe element for an example of usage:

include:
  - local: 'util/all-global.gitlab-ci.yml'

Contents

The util/all-global element contains functionality to support patterns described in this guide throughout the library:

• Constants pattern
• SetEnv and GetEnv pattern
• Custom job images pattern
• Pipeline conditions pattern

It also includes:

• Definitions of stages to be used across all pipeline definitions - a longer and more granular, but still generalized, list of stages than provided by default in GitLab, to allow for more sophisticated pipelines. Ohter elements in the base library reference the stages defined here and expect them to occur in that order.
• A handy display function to highlight important output in job logs
• Default job image setting for alpine

The root-level default antipattern

TODO

Including other files

STATUS Sketchy: Author brainstormed ideas and recorded them - read with caution

Often it's necessary to include files other than CICD YAML (such as scripts and configuration files) for use in a job. The include: mechanism only supports CICD YAML. Here are other options, which might apply in different situations:

Script text in YAML

Some shell scripts are too long or complicated to go into a YAML array nicely, but still belong in the CICD element. Try including the script text in the element itself at the end using a single YAML multiline string in a YAML anchor

.my-script: &my-script |
  function my_function() {
    echo "Hello $1!"
  }

my-job:
  script:
    - *my-script
    - my_function 'World'

In a long CICD element, sometimes it's easier to keep the big script block at the end of the element, after all the YAML. But an anchor definition has to appear before its reference. Consider putting a base job at the end of the file (only works if the multiline script is the only entry in before_script).

my-job:
  extends: [.my-script]
  script:
    - my_function 'World'

# ... other configuration ...

.my-script:
  before_script:
    - |
        function my_function() {
            echo "Hello $1!"
        }

        # ... other script ...

As an example, the Salesforce.com team provided a set of Bash functions in a multiline YAML anchor for the SFDX integration project (example).

Other techniques

• Put the whole script in a group CICD variable - not recommended
• Prebuild a custom job image
• Publish to the generic package registry
• Use needs:project:

Custom job images

STATUS Sketchy: Author brainstormed ideas and recorded them - read with caution

Why custom job images

...

Suggested guidelines

1. Keep the source code (Dockerfile) for custom job images in the CICD library project to manage all the configuration related to one operation in the same place.
2. Publish custom job images to the GitLab container registry -- even if the company has a policy of storing all container images in an external platform, it's MUCH simpler to keep custom job images -- which are both created and consumed by GitLab -- in GitLab itself.
3. Dockerfile...
4. job image version...
5. how to build in an MR...

Proposed conventions

• Directory structure: images/ at the root of the library, with a subfolder for each image
• Dockerfile and README.md (minimum) in each image folder, along with other files required
• .gitlab-ci.yml fiel within each image directory with instructions for building that image -- typically a reference to docker/build-job.gitlab-ci.yml with inputs:.
• In the main .gitlab-ci.yml for the library, an include: entry for each image to be built.

See the MDBook image in the base library for a simple example, or Maven for one that includes other files.

Using a custom job image

1. extend custom-job-image-registry base
2. Understand the JOB IMAGE variables

Author notes

• Better approach to versioning of custom images
• Workarounds for no variables in MR pipelines?
• Here explain the logic in root CI config and in the build job

Job labels

STATUS: Draft: Some structure and useful information - test and refine before use

Opportunity: Include the same CICD element multiple times

Example: some teams need to deploy to multiple environments or clusters, with slightly different settings for each, but a similar script. Inputs make it easy to keep the configuration DRY while keeping the parameters local.

A client needed to deploy to a custom platform across multiple environments and regions. To avoid repetition, we include the same element more than once.

In deploy-phase.gitlab-ci.yml:

  - local: deploy-job.gitlab-ci.yml
    inputs:
      region: west
      env: stage
  - local: deploy-job.gitlab-ci.yml
    inputs:
      region: east
      env: stage
  - local: deploy-job.gitlab-ci.yml
    inputs:
      region: west
      env: prod
  # ...etc...

Problem: Job name collision

Job names are global to the entire pipeline, across all include: elements, including base jobs. In the example above, if the deploy-job element included a job such as deploy, YAML interpolation would overwrite the job name each time, and only one job would appear in the pipeline. The impact confuses some developers.

Solution: Reference unique inputs in job names

In deploy-job.gitlab-ci.yml:

spec:
  inputs:
    region:
    env:
---
deploy-$[[inputs.env]]-$[[inputs.region]]:
  # ... other aspects of job definition ...
  script:
    # ... other parts of script ...
    - mvn deploy -Denv=$[[inputs.env]] -Dregion=$[[inputs.region]]

The jobs appear in the pipeline UI and API responses with unique names such as deploy-dev-west.

Hot tip: Use explicit job labels

For many job elements, multiple use cases aren't apparent when the job is created. But others might come up with reasons to use a job more than once. Consider using an explicit job label in every job element in a library.

In some-test-job.gitlab-ci.yml:

spec:
  inputs:
    job-label:
      default: job
      description: >-
        Set to differentiate jobs if using this element more than once in a pipeline.
    # ... other inputs ...
---
some-test-$[[inputs.job-label]]:
    # ... job definition ...

The base library usess the pattern extensively (example).

CICD state

STATUS Sketchy: Author brainstormed ideas and recorded them - read with caution

The library supports a mechanism for saving the state of CICD pipelines in the middle for later reuse within pipelines of the same ref. For example, a project's built package can be saved for deployment in a later pipeline.

To save and load state, include cicd-state-job.gitlab-ci.yml.

The CICD state mechanism saves all the artifacts available in the pipeline at the point of saving, placing them into the GitLab generic package registry indexed by Git ref. The loading job then pulls the latest entry from the registry.

Constants

STATUS Sketchy: Author brainstormed ideas and recorded them - read with caution

CICD Variables are, in fact, variable: they can differ from one pipeline run to the next. Also, variable precedence means that values defined in CICD YAML can be overwritten at the group/project level in GitLab for a specific pipeline run, sometimes overriding important logic.

Sometimes we use CICD variables to represent constants, that is:

• The value stays the same during a pipeline
• The value is the same for each pipeline run
• The value is the same for pipelines run in multiple projects accesssing a library

In these cases, sometimes we use a group-level variable, creating a tight coupling between database data (the variable setting) and configuration (in the YAML). Such a situation can confuse future operators, who have to search to figure out where the variable it set. An alternative is to set variables in the CICD library itself, which results in a situation akin to hard-coding values in code. Instead of both, we recommend using a single file within the CICD library to hold all the constants, and the bas library provides a mechanism to do so.

The Constants pattern says:

Manage site-wide constants (not true variables, not secrets) in a single YAML file

The constants.yml file

The base library contains everything necessary to manage constants as described by the pattern:

• A placeholder constants.yml file - put your values here
• A function constant to read from the YAML file in a Bash script at runtime using YQ
• An include: object using util/save-package-file-job to save the constants.yml file to a generic package in the library (part of the library's own .gitlab-ci.yml element)
• An include: object using util/load-package-file-job to load the constants.yml file at the beginning of every pipeline run to make the values available to scripts within the pipeline (part of all-global)

Shortcomings

• Constants loaded this way can only be used within scripts - they are not available at the time of YAML interpretation so can't be used in environment names, etc.

TODO Needs more examples and clarity!

SetEnv and GetEnv

STATUS Sketchy: Author brainstormed ideas and recorded them - read with caution

Instead of using artifacts:reports:dotenv:, use the setenv/getenv pattern defined in the all-global element. It relies on job artifacts, which are more tightly constrained and can be saved.

TODO

Version numbering

STATUS Stable: Someone other than the author has read or used this - proceed with confidence

Managing version numbers in software projects requires balancing several competing needs: maintaining a clear history of releases, ensuring build artifacts are properly labeled, and keeping version information accessible to running applications.

While it's tempting to store version numbers directly in source files and commit them to the repository, this approach creates unnecessary commits, complicates merges, and makes it harder to audit changes later.

This document describes a CICD-based approach where version numbers are generated and managed by the pipeline itself, using Git tags as the source of truth and performing any necessary version substitutions during the build process.

Doing so keeps the version numbers out of committed code while ensuring all build artifacts and deployments are properly versioned and correlated to Git tags.

Principles

• Semantic versioning (major.minor.patch) and its variants apply to almost any type of Git repo - definitely to code libraries, but also to web apps, mobile apps, infrastructure-as-code, and documentation.
• Git tags are the standard way to store canonical version information and correlate the version number to a specific state of the code base.
• GitLab Releases are the standard way to store metadata about a version, including release notes.
• Apps sometimes need to know their version number, at build time, at runtime (such as to show the version number on a help page), or both.
• Sometimes certain configuration files must contain a reference to the project's version number, for example pyproject.toml, terraform.tfvars, or pom.xml.
• It's best if version numbering happens downstream of source code, and the only record of a version number lives in the Git tags themselves. In other words, resist the temptation to add new commits just to bump a version - see Avoid Git pushes in pipelines.
• For configuration files that require version numbers, it's better to replace the version number in the files at build-time (based on the Git tag) than to store the version number itself in the repo.

Library

The ProCICD base library (and many private CICD libraries) uses the VerNum tool for semantic version number management and processing. Refer to the VerNum README for details on the tool itself.

Capabilities include:

• Flexible technology support: Works with any technology or framework
• Embeds common practices: Uses Git tags, GitLab release objects, and CICD
• Manual and automated releases: Trigger version increments either manually through pipeline variables or automatically using commit messages
• Rich release documentation: Automatically generates changelogs from commit messages and maintains activity logs
• Safe version management: Checks for existing releases, protected branch requirements, and different handling for push events versus manual triggers
• Protected release process: Releases only occur on protected default branches, preventing accidental version increments

Configuration

To use version numbering (assuming a CICD library derived from the base library) add the following to a pipeline:

include:
  - local: 'util/all-global.gitlab-ci.yml'
  - local: 'release/release-global.gitlab-ci.yml'

If version numbering is the only functionality required from ProCICD, then include the release pipe in your project:

include:
    - project: your/library
      file: release/release-pipe.gitlab-ci.yml

The above approaches will add a set of jobs that:

• Output the latest version (based on Git tags that match the pattern v1.2.3)
• Increment the version according to an increment provided as a variable
• Place the new version number in a job artifact (using setenv) for use by later jobs
• Add a GitLab release object and Git tag for the new version (using release-cli)

Jobs that need to access the version being released can extend the .release base job and reference the RELEASE_VERSION variable in their script blocks.

Performing a release

To trigger a release, you can use one of two variables:

• VERSION_INCREMENT: Increment the version number. Provide the value patch, minor, or major
• VERSION_RERELEASE: Perform release/deployment operations on an already created/tagged version

Treat these variables as momentary and keep them out of the settings. Provide the value in the "Run Pipeline" form or use the glab CLI:

glab ci run --variables VERSION_INCREMENT:minor

Or attach the variable to a push operation:

git push -o ci.variable="VERSION_INCREMENT=alpha"

Auto-release

Most teams like to control when release number increments happen at pipeline runtime using variables. Alternatively, the framework supports automatic version number increments based on commit messages.

To enable Auto-release, set the auto-release input to a space-separated list of the increments that are allowed. For example:

include:
  - project: 'procicd/lib'
    file: 'python/pypi-library-pipe.gitlab-ci.yml'
    inputs:
      auto-release: patch minor

(Note that the -pipe element must support the auto-release input.)

When enabled, include [patch], [minor], or [major] in commit messages to perform a release with that increment.

Changing files

For those files within the repo that require the version number in order to operate correctly, use the following methodology:

• Use a placeholder (typically 0.0.0) in configuration files in the code base to represent the version number
• Make sure that local build/test operations on developer workstations work as expected for version 0.0.0
• Perform a replace operation (typically using sed) in the pipeline after determining the version number but before build/deploy operations that require it

Examples of such files include pyproject.toml, terraform.tfvars, and pom.xml.

Example

The reference example of version number usage appears in the Steampunk Wizard Busy project and the PyPI library pipeline.

• The project CICD configuration references the full pipeline in the library.
• The pipeline configuration in the library includes the all-global and release-global elements which contain everything necessary to perform the release operations.
• The PyPI build job replaces the version number in the .toml file and performs the build operation.
• A Release object in GitLab corresponds with a deployed version of the application and a tagged commit in the code base.

About VerNum

• The PyPI page for VerNum describes it in detail.
• VerNum provides several schemes for version numbering; the most common and default is the simple major/minor/patch scheme (called just patch).
• If performing version increments frequently from the Run Pipeline page, consider including one of the scheme addons that will create a dropdown list for easy use.
• For even more sophisticated variants, include a .vernum.yml file in the project as per the documentation.
• VerNum doesn't always bootstrap. Add a tag with the name v0.0.0 before starting.

Phase skipping

STATUS Rough: Undergone lightweight review and/or testing - reference it to discern broad themes or extract examples

When developing and testing CICD elements, it's often necessary to run just certain jobs from a pipeline, and to turn jobs off and on, to get through testing quickly. We recommend two techniques, both of which work at the "phase" level (see yaml element types) so across a set of related/interdependent jobs. Combine both approaches to allow phase skipping with either an input (in the YAML itself) or a variable (which could be temporarily set in the test project or at runtime).

It's set up in the -pipe element. As an example, to allow skipping of a test phase, add skip-test to inputs: and support a variable called $CICD_SKIP_TEST.

Example of phase skipping:

  - local: mytype/mytype-test-phase.gitlab-ci.yml
    rules:
      - if: "'$[[inputs.skip-test]]'=='' && $CICD_SKIP_TEST==''"

Pipeline conditions

STATUS Sketchy: Author brainstormed ideas and recorded them - read with caution

The different pipeline types and conditions under which pipelines run can present challenges. Without attention, a project can end up with multiple pipelines running on the same push, pipelines triggering other pipelines in a loop, or simple confusion.

Guidelines and detailed pipeline types

In the base library, we define a more detailed set of pipeline types, then use a variable and the pipeline name in the UI to annotate them. The guidelines are:

• Never run tag pipelines; insead add release jobs to the same pipeline that creates the tag
• Only run branch pipelines on the default branch or specific branches (i.e. for hotfixes)
• Use merge request pipelines exclusively within merge request branches
• Try to avoid pipeline runs when the only activity is creating a branch with no changes
• If someone makes and pushes a branch outside of a merge request, do nothing
• Use global pipeline rules to control and annotate different types of operations

Referencing conditions

$CONDITION can be referenced in job-level rules. For example,

if: $CONDITION =~ /pipelinetest/

or

if: $CONDITION =~ /main|hotfix/

Defining conditions

The util/all-global element in the base library handles pipeline condition logic, sets the $CONDITION variable, and generates the correct title for the GitLab UI.

Static Docs Site

STATUS Sketchy: Author brainstormed ideas and recorded them - read with caution

• mdbook
• sphinx

Contributing to a CICD library

STATUS Rough: Undergone lightweight review and/or testing - reference it to discern broad themes or extract examples

General guidelines

• Treat the pipeline like a series of loosely coupled microservices, with each job performing only one function.
• If referencing one of the GitLab "templates", wrap it to provide a place to keep standard overrides and customizations - the wrappers live in the gitlab/ directory.
• Remember that arrays cannot be extended in an extends or override case - especially tricky with before_script - though the !reference form can be useful.
• See YAML comments for more notes; the maven directory contains the most up-to-date examples of best practices.

Contributing to the library

The library includes a mechanism for testing changes to the library itself

1. Pick a test project for testing out the pipeline changes (preferably with a small blast radius if things go wrong)
2. Create a branch and MR in this library project to work on the changes
3. Create a branch and MR in the test project you're using to work on the changes 1. In the test project, add a ref: to the include: pointing to the working branch of the library
4. Add a cicd-pipeline-test label to the MR in the test project
5. When developing, test for the $CONDITION=~/pipelinetest/ (see maven-base.gitlab-ci.yml for an example) as necessary to detect test runs (for example, to override branch name tests)
6. When complete, merge the MR in the library
7. Remove the ref: from the MR in the test project and merge it if needed
8. Generate a new numbered release of the library (using the VERSION_INCREMENT variable - to be implemented)

Upstreaming to the ProCICD base library

TODO