Base library

The ProCICD base library can be used as a starting point for custom CICD libraries.

Getting Started

To begin, copy the base library into a gropu within the organization's control. Some options:

  • Clone it - allows easy upstreaming of changes but exposes more of the history of ProCICD
    1. Use git clone to make a local copy of procicd/lib.
    2. Use git remote to add a new remote for the custom library project.
    3. Make initial adjustments (below)
    4. Push to the new remote
  • Copy it - makes upstreaming changes more difficult, but keeps the division clean.
    1. git clone git@gitlab.com:your/new/library.git (replace path to new GitLab project for library)
    2. cd library (replace with correct name)
    3. git archive --remote git@gitlab.com:procicd/lib.git v1.2.2 --output procicd.tar (replace version)
    4. tar xfv procicd.tar
    5. rm procicd.tar
    6. git add -a
    7. git commit -m 'ProCICD v1.2.2'
    8. git push

In either case, edit the README to replace procicd/lib with the path to the custom library project itself. Something like the following (on MacOS) will work:

find ./ -type f -exec sed -i '' -e 's@procicd/lib@your/new/library@g' {} +

Directories

Within a CICD library (including the base library), we suggest the following directories:

  • docker: Boilerplate job for docker builds. Used for custom job image prebuilds; can also be used in projects if needed.
  • gitlab: Placeholder/example directory for wrappers around the standard instance-level "templates" as needed for extension or customization.
  • images: Custom job images for prebuild (including Dockerfiles and .gitlab-ci.yml files for performing the builds).
  • java: Elements developed for Java and Maven projects. Maturity: low
  • python: Elements developed for Python projects. Maturity: high
  • release: Commonly used elements related to version numbering and release tagging, available for use as needed.
  • scratch: Mostly outdated reference material such as scripts from legacy CICD systems - don't use them in include:, and consider deleting them when no longer of use.
  • static: Elements for static site generation (GitLab Pages). Maturity: medium
  • util: Commonly used elements with broad utility, including:
    • save/load CICD state and generic registry packages.
    • all-global.gitlab-ci.yml - Include it in everything. Described below.
  • Name other directories in the library after the underlying technology (e.g. ruby).
  • The root of the library project contains a few special files:
    • .gitlab-ci.yml - The CICD pipeline definition for the library itself, which is capable of prebuilding custom job images (when they change) and handling release tagging of the library as well as pushing constants.yml to the generic package registry.
    • constants.yml - Values required by various jobs and prepackaged for download. (TODO)
    • VARIABLES.md - Index to global variables.

We left in some boilerplate elements not actually used today, in case they become useful in the future.

Elements (YAML files)

The library contains several specific types of elements (YAML files), distinguished by file name, indicating their intended use.

  • pipe: complete pipeline definition, ready to be referenced as an include: (with or without inputs:) in a project's CICD YAML. pipe elements may contain one or more phase or job elements. Example: python/pypi-library-pipe.gitlab-ci.yml.
  • phase: set of jobs that serve a common purpose within a pipeline, which might cross multiple stages, often pass values between each other in artifacts, and are typically referenced from a pipe element with include:local:. phase elements may contain one or more job elements.
  • job: exactly one job, which typically contains a job-label input that gets suffixed to job names to support repetition (see below). Example: python/poetry-build-job.yml.
  • base: only base ("hidden") jobs (starting with '.') to define YAML objects shared by jobs in multiple elements. Example: python/venv-base.gitlab-ci.yml.
  • global: elements that may be included only once in a pipeline, to include project-wide jobs or root variables: with options: and/or description:. Example: release/version-patch-scheme-global.gitlab-ci.yml.

Use the following format for element filenames: <parts>-<type>.gitlab-ci.yml where "parts" are all-lower hyphen-separated name parts and "type" is one of the types described above.

The full .gitlab-ci.yml filename extension (different from the GitLab "components" convention) allows Visual Studio Code (with the GitLab extension) to see the files as GitLab CICD configuration and handles linting appropriately.

Includes and inputs

Library elements are designed to be include:'d in project CICD YAML configurations, and elements might include: other elements, which might in turn include: yet more elements. So inclusion is nested.

Included elements use the inputs: mechanism (introduced as part of the "components" intiative at GitLab) to accept parameters, similar to function arguments in a traditional programming language.

The library uses the kabab-case naming convention for inputs: (hyphen-separated, all lower).

Try to avoid propagating inputs (i.e. having an input reference another input).

CICD Variables

In GitLab CICD, Variables are named values that may be set in group settings, project settings, on the Run Pipeline form, via API, via Git push options, globally in YAML, within a rules: item, within a job, or inside a script (among other ways). Variables can be used for if expressions and in scipts as shell environment variables, among other ways.

If this is your first time working with variables in CICD pipelines (or just for a refresher), consult the guide to variable precedence in the docs.

Because of the flexibility and power of CICD variables, the library applies some conventions to help make sense of them.

  • Use the ALL_CAPS_WITH_UNDERSCORES convention for all CICD variables and environment variables defined in a script item (clearly differentiates them from inputs to avoid confusion).
  • For variables set at project or group level and required by a pipeline, define them as a global variable (root level of YAML, either in the all-global element or in one of the pipe elements) so other engineers can see the variable is expected - typically the values may be an empty string, or contain a default value. Example: DEFAULT_JOB_IMAGE_REGISTRY in util/all-global.gitlab-ci.yml.
  • Note: Because we set default empty string values for variables, to test for a null value, test for '' rather than null in rules:.

See VARIABLES.md for details on global variables required by the library.

Jobs

A few thoughts on best practices for individual jobs:

  • Use base jobs and extends: to minimize duplication
  • Use numbers at the beginning of job names where necessary to force ordering in the web UI when viewing by stage.
  • Use the display function to highlight key values in job log outputs.
  • Job-level CICD variables make a handy place to keep error message text.

Library versioning

The library supports versioning in multiple ways:

  • CICD element (YAML) versioning uses Git tags and semantic versioning; refs can be used in the include: object in projects to specify a version of the library. To increment version: Set the VERSION_INCEREMENT variable in the Run Pipeline form and run a fresh pipeline.
  • Custom job image versions can be changed with the CUSTOM_JOB_IMAGE_VERSION variable, though usually not necessary - helpful for e.g. clearing Docker caches. Projects always get the latest version with the associated value, regardless of which version of the library they reference.
  • constants.yml goes into the generic package registry. Projects always get the very latest version, regardless of which version of the library they reference.