CI/CD Infrastructure

Introduction

Clea OS, as explained in the previous pages, is a Yocto project consisting of several layers and GitLab repositories. The main project that contains all references to the various components of Clea OS is the seco-manifest. This project is composed of several manifest files with the links and git references (branches, commit SHAs or tags) to the various repositories and keep tracks of the changes occurring to the above-mentioned references.

Basically, every new feature, every new bug fix and every new implementation to any of the projects that make up Clea OS must be reflected in an update on the manifest, which then allows Clea OS to be built. Not only that, every new change proposed through the Merge Request (MR) mechanism must be tested before it is introduced and the software artifacts of Clea OS must be accessible by everyone within a workflow as automated as possible.

These and other fundamental requirements that are detailed in this page are those that guided the development of the CI/CD (Continuous Integration and Continuous Deployment) infrastructure of Clea OS.

Gitlab CI/CD Repository

The gitlab-ci repository within the infrastructure group is the one that contains the CI/CD core within Clea OS. A detailed description of the content of the repository and the main CI/CD tasks it allows to perform can be found in the README file of the repository itself.

In summary, the repository contains several .yml format files, following GitLab syntax for CI/CD and describing the pipeline logic, together with python and shell scripts to perform automation tasks within pipeline jobs.

The repository is included as a versioned git submodule in the .gitlab-ci folder of each Clea OS repository requiring CI/CD automation tasks. The .gitlab-ci.yml file in the top level of each of these repositories then allows the automation steps defined in the submodule to be imported and extended according to the specific requirements of the project (see as an example this .gitlab-ci.yml file).

Main Automation Tasks

The main automation tasks that are implemented in the gitlab-ci repository are described in this section. For further details, please consult the dedicated documentation.

Manifest Integration

Whenever a developer opens a MR from a feature branch to the trunk (see Git Workflow page) of one of Clea OS layers, an integration branch is created in the seco-manifest containing the updated project revision. An example of a manifest file is the soc/rpi/seco-rpi one:

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

    <remote name="seco-layer-rpi"       alias="origin"   fetch="https://git.seco.com/clea-os/layers/seco" />
    <remote name="layer-raspberrypi"    alias="origin"   fetch="https://git.yoctoproject.org" />

    <project name="meta-seco-rpi"       revision="0fbac111470cda2d1b9f7d023c7b7f93b345edbe" path="layers/meta-seco/meta-seco-rpi"         remote="seco-layer-rpi" />
    <project name="meta-raspberrypi"    revision="80a12f7fddfeae28c43242765374e7ade3a2a59e" path="layers/meta-raspberrypi"      remote="layer-raspberrypi" />

</manifest>

Once the integration has occurred, before performing the merge, the change on the manifest needs to be tested. In order to do so, the build, deploy and test steps that are described in section Automatic Build/Deploy/Changelog/Test/Release are performed. The image below sums up the CI/CD workflow for MRs opened in Clea OS layers, and described by manifest-integration.yml configuration file.

Once the original MR in the layer is merged, the automation creates a MR on the manifest, which merges the project integration branch into the trunk, and merge it (see image below).

For further details about this CI/CD steps please consult the following page.

Layers Integration

Some of the recipes in the silicon vendors layers, more precisely those regarding the Board Support Packages (BSPs) integration, contains the references (URL, branch and commit SHA) to the Clea OS BSP repositories. In particular, the references to the version of U-Boot, Kernel, Firmware used is defined via the SHAs of the commits within the SRCREV.conf files.
For instance, some of the meta-seco-rk layer recipes contain the reference to the Rockhip BSP repositories and the corresponding conf/SRCREV.conf file looks something like this:

SRCREV:pn-u-boot-seco-rk = "394c839d8303424b01c9295c66d46e4292c8ad87"
SRCREV:pn-u-boot-seco-rk:u-boot-seco_2017_09_next = "00ce97d978a23baad747859a63f929aad2813b29"
SRCREV:pn-linux-seco-rk = "11ce97e283c773640450b8ec4f5abc0e5690a475"
SRCREV:pn-u-boot-seco-rk-fw-sysdata:u-boot-seco_2017_09_next = "00ce97d978a23baad747859a63f929aad2813b29"
SRCREV:pn-linux-seco-rk3399-rk = "14ea03d42144b8f5de4e4578a92d3c2dea93ebeb"

For this reason, whenever a developer opens a MR from a feature branch to the trunk (see Git Workflow page) of one of Clea OS BSP repositories, an integration branch is created by CI/CD in the corresponding layer, containing the updated BSP revision. Once the integration branch is created, also a MR is opened in the layer, thus triggering the same automation processes described in the Manifest Integration section.
The image below sums up the CI/CD workflow for MRs opened in Clea OS BSP repositories, and described by layers-integration.yml configuration file.

Once the original BSP repository MR is merged, the automation directly merges the corresponding layer repository MR, which in turn triggers the integration in the seco-manifest according to what described in the Manifest Integration section.

For further details about this CI/CD steps please consult the following page.

Automatic Build/Deploy/Changelog/Test/Release

The main automation pipeline is executed in the seco-manifest and allows software artifacts builds to be executed whenever a change occurs, i.e., integration on the manifest, or on a weekly basis on the trunk branches.
The image below provides an example of a CI/CD pipeline for most of the steps described in this section, and defined in manifest-pipeline-yocto.yml configuration file.

For further details about these CI/CD steps please consult the following page.

Yocto Build

The first and fundamental step in the pipeline is the build of Clea OS for all supported hardware platforms and for all available distros. Given the large number of supported boards and configurations, this stage is very demanding in terms of resources and time-consuming as well as being vital to the project.

Deploy on Azure

Once the builds have been executed, the products resulting from them, i.e. software artifacts, are packaged, renamed and uploaded to a dedicated Azure blob storage to make them available to the public or to specific customers.

Automatic Testing in LAVA

Another key step to ensure high quality of the software product released by SECO is to run automatic tests of new Clea OS versions on the supported hardware platforms. For this purpose, a LAVA lab dedicated to orchestrating Clea OS testing on SECO's hardware was implemented.

Changelog Creation

Every time a build is executed, the changelog is also generated. In particular, the job makes it possible to obtain a file in .md format containing the titles and URL references of all the MRs merged into the various projects composing Clea OS. This gives an idea of the various major changes between different versions of Clea OS.

Internal Build/Deploy Status Notification

Once all jobs associated to the build and deploy stages have finished, the notify stage allows to collect the results from all previous jobs (and the link to the artifacts) and to send them to a dedicated internal channel in Microsoft Teams.

GitLab Release

Only when the tag is executed will a final job also be executed that allows a GitLab release (see Release Procedure page for details) associated with that tag to be created. This job makes it possible to associate with the GitLab release all the links to the software artifacts produced by the build jobs in such a way that they are easily accessible, and allows the previously generated changelog file to be associated with the release description.

Integration of 'gitlab-ci' Modifications

As explained in the GitLab CI/CD repository section above, the gitlab-ci repository is integrated into Clea OS projects as a submodule and as reference in their .gitlab-ci.yml file. Both references need to be updated when a change in the repository shall be deployed to the projects using it. To achieve this, the CI pipeline is set up to update all these projects in one run every time a MR in the gitlab-ci repository is merged.

For further details about this CI/CD steps please consult the following page.