Staging Developer CI
Latest Tests
Table Of Contents
Architecture
The Architecture Goals are:
-
Use CodeLinaro Gitlab CI
- Runners
- Package Registry
-
Use TuxSuite
-
Per-Project non-volatile mount point
-
Linux kernel build
-
Yocto Bake
-
-
Use CDBA -To boot U-Boot and run pytest -To Boot Linux with an initramfs containing a test suite
Target Design
meta-qcom-ci
This layer will live on top of meta-qcom and be somehow similar to meta-rpb, but it's only goal would be to setup a generic test initramfs with tests, to be run by a systemd testsuite service at the end of the boot. The result initramfs should be generic to all qcom boards and built occasionally (weekly ?) or when the layer changes.
linux builder
This will build the pushed linux branch/tag via tuxsuite, retrieve the artifacts and push them to the projects' package registry to provide a stable URL set for download or usage from other CI pipelines.
Package Registry can be accessed with a project token, setup and usage is easier than CI jobs artifacts or tuxsuite URLs.
linux CI tester
This will take in argument a set of Linux objects (Image, DTBs, modules), will fetch the latest meta-qcom-ci initramfs, pack them as Android Boot images for all configured boards and them will boot each of them on the configured CDBA boards from QcomLT members or Qinaro. U-Boot builder This will build the pushed linux branch/tag with U-Boot buildman, retrieve the artifacts and push them to the projects' package registry to provide a stable URL set for download or usage from other CI pipelines.
U-Boot CI tester
This will take a board name in argument, build U-Boot, package it as Android Boot image and then will run the U-Boot pytest suite on the configured CDBA board from QcomLT members or Qinaro.
Design
Current design has been implemented in a “staging” codelinaro sub-group: https://git.codelinaro.org/linaro/qcomlt/ci/staging
Docker Image
In order to reduce dependency on docker.io, the common docker image is built in https://git.codelinaro.org/linaro/qcomlt/ci/staging/docker-image
A weekly pipeline rebuilds the image from ubuntu 22.04 and uploads it to CodeLinaro artifactory: https://artifacts.codelinaro.org/ui/repos/tree/General/linaro-260-ubuntu/ubuntu-22.04-qcomlt-ci
In order to push docker images to JFrog, the pipeline requires: DOCKER_USER: username to JFrog, today neil.armstrong DOCKER_PASS: login token than can be created at https://artifacts.codelinaro.org/ui/user_profile, with only the “Mask variable” setting enabled.
meta-qcom-ci
Repository URL: https://git.codelinaro.org/linaro/qcomlt/ci/staging/meta-qcom-ci
This layer is used on top of:
- Poky
- meta-openembedded
- meta-qcom It builds itself every week on top of master branches to use the current state-of-the-art Yocto/OE support.
It declares:
- A “qcom-ci” distro which is a simplified version of the meta-rpb wayland distro, it uses systemd by default and enables opengl and vulkan
- A “qcom-ci-arm64” machine which is a variant of qemuarm64.conf with freedreno, dummy kernel and adbd setup for non-android
- A “initramfs-test-image-ci” image recipe which is a variant of meta-qcom’s “initramfs-test-image” with a few test package and the “ci-testsuite”
It also adds the “ci-testsuite” recipe which is developed at https://git.codelinaro.org/linaro/qcomlt/ci/staging/ci-testsuite
The “ci-testsuite” is a very lightweight, simple and extensible test runner which:
- Emits LAVA signals
- Outputs on kernel log only
- Starts using systemd
- Adds a systemd shutdown hook in “/usr/lib/systemd/system-shutdown/” in order to ask CDBA to shut down with the "~~~~~~~~~~~~~~~~~~~~" signaling
- Runs simple script shells in order
- Supports “ci_testsuite_skip” and “ci_testsuite_allow” on the kernel command line to disable or enable specific tests
The Yocto/OE is built using TuxSuite, so it requires setting a “TUXSUITE_TOKEN” to the “CI/CD Settings” Variables settings, with only the “Mask variable” setting enabled.
After build, the pipeline will store the TuxMake artifacts and “tmp/deploy/images/qcom-ci-arm64/” files in the Package Registry.
linux builder
Repository URL: https://git.codelinaro.org/linaro/qcomlt/ci/staging/linux-build-pipeline
The Linux build pipeline is declared as an external Gitlab CI yaml to be added to the linux project with: “linux-build.yml@linaro/qcomlt/ci/staging/linux-build-pipeline” In the “CI/CD configuration file” entry of the Gitlab project’s “CI/CD Settings” panel.
As of now, it only builds the kernel using the TuxSuite API, which does all the kernel building by itself.
It required adding a “TUXSUITE_TOKEN” to the “CI/CD Settings” Variables settings, with only the “Mask variable” setting enabled.
The Pipeline will only build the following branch/tags names:
- next-*
- For linux-next
- qcomlt-*
- For Linaro engineers pushing their own branches/tags
- v5.1[0-9]+*
- For Linux stable v5.10+ It won’t build before because the current systemd won’t boot on kernels older than v5.10 due to missing functionalities
- v[6-9].[0-9]+*
- For Linux stable v6.0+
After build, the pipeline will store the following files into the Linux projects Package Registry: Image.gz modules.tar.xz dtbs.tar.xz And other TuxMake artifacts like json and reproducing scripts
Finally, the pipeline will trigger a CI test downstream pipeline with the URL of the kernel files as CI Variables.
linux CI tester
Repository URL: https://git.codelinaro.org/linaro/qcomlt/ci/staging/linux-build-pipeline
This is the key part of the CI system:
- Downloads the pre-build kernel files
- Generates a compressed cpio with modules
- Generates boot images scripts for each configured boards
- Generate the boot images with configured mkbootimg options
- Generates a child pipeline yaml with the:
- Configured boards
- Availability of DT on the currently built kernel
The child pipeline will then:
- Boot the generated boot image with cdba on the configured cdba server + board id
- Parse the boot log and generate a JUnit XML
- Pass the JUnit XML to Gitlab to be used as a test report
The Boards are configured in a YAML file: boards.yaml
The mkbootimg scripts are generated using the generate_bootimage_params.py script by parsing the boards.yaml file.
The generate_tests_yml.py will generate the child pipeline YAML by parsing the boards.yaml file and using the dtbs tarball content, using Jinja2 and the test-jobs-ci.yml.jinja2 template.
After the CDBA run, the boot log is parsed with the test_signal_parser.py and generate a JUnit XML using the Python junit_xml library.
In order to access the CDBA boards using SSH, a SSH private key is added with the corresponding SSH_KNOWN_HOSTS to validate the SSH server.
Manual/Prebuilt CI
In order to simplify life for engineers already having a local build routine, a separate repository has been added to trigger CI test runs by providing pre-built binaries:
Repository URL: https://git.codelinaro.org/linaro/qcomlt/ci/staging/manual-test
The idea is to clone this repo, create a branch named with the engineer login and simply add the files (Image.gz, dtbs.tar.xz & modules.tar.xz) the push to trigger the test.
The repository supports GIT LFS and any number of branches/tags/
How-To
Triggering Tests on a Custom Tree
You can push a branch/tag named “qcomlt-*”, a dedicated repository exists for this purpose: https://git.codelinaro.org/linaro/qcomlt/ci/staging/linux
Example: https://git.codelinaro.org/linaro/qcomlt/ci/staging/linux/-/pipelines/63198
Adding new Linux Tree
The cdba-test pipeline can be triggered by any pipeline, simply setup the following in a linux gitlab project:
- Enable CI
- Enable Package Registry
- Add TUXSUITE TOKEN variable
Now push branches/tag with the names:
- next-*
- qcomlt-*
- v5.1[0-9]+*
- v[6-9].[0-9]+*
And the pipeline will build with defconfig and trigger the tests.
Adding new boards
The boards are configured in : https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/blob/master/boards.yaml
Add the following public key to your CDBA allowed keys: https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/blob/master/staging-cdba-ci.pub
And edit boards.yaml with:
boardid:
dtb: "boardname.dtb"
base-addr: "0x80000000"
ramdisk-offset: "0x2000000"
cmdline: "root= clk_ignore_unused pd_ignore_unused audit=0 -- ci_testsuite_skip=0001-bootrr ci_testsuite_allow=0003-sysbench"
header-version: 0
pagesize: 2048
cdba:
- name: cdba-server-name
host: ssh://cdba@hostname:port
board: cdba-board-nameAllowed values for boardid are:
- dtb: dtb name in qcom/ subdirectory, mandatory
- cmdline: --cmdline option to mkbootimg, mandatory
- pagesize: --pagesize option to mkbootimg, mandatory
- header-version: --header_version option to mkbootimg, mandatory, with values 0 & 1, will cat dtb to Image.gz, with value >= 2 will pass DT with --dtb option
- base-addr: --base option to mkbootimg, optional
- ramdisk-offset: --ramdisk_offset option to mkbootimg, optional
- additional-args: addition args to pass to mkbootimg, optional
The cdba option will list all configured cdba server for this specific board, each entry must have:
- name: will be used to construct job name
- host: is passed to -h options of cdba
- board: is passed to -b option of cdba
It’s possible to skip/allow tests from the kernel cmdline by adding comma separated test names in ci_testsuite_skip and/or ci_testsuite_allow, example:
cmdline: "root= clk_ignore_unused pd_ignore_unused audit=0 -- ci_testsuite_skip=0001-bootrr ci_testsuite_allow=0003-sysbench"In order to test with a known to work files, run this “Validate, keep disabled” Scheduled Pipeline in Build / Pipeline Schedules.
Adding new tests
Tests are simple shells files in the tests sub-directory of https://git.codelinaro.org/linaro/qcomlt/ci/staging/ci-testsuite/-/tree/master/tests
A typical test would look like:
#!/bin/sh
source /usr/lib/ci-testsuite/utils/helpers.sh
testsuite_report_start_case $1
# run test code
testsuite_report_start_case "$1-do-test"
do_test
testsuite_report_end_case_status "$1-do-test"
testsuite_report_end_case_pass $1
exit 0Helpers are provided to report status, in https://git.codelinaro.org/linaro/qcomlt/ci/staging/ci-testsuite/-/blob/master/utils/helpers.sh
# testsuite_report_test_run_start <test-name>
Report test run starts
# testsuite_report_test_run_end <test-name>
Reports test run stops
# testsuite_report_test_pass <test-name>
Reports Successful test run
# testsuite_report_test_data <test-name> [measurement [unit]]
Reports Successful test case with measurement, and optional unit
# testsuite_report_start_case <test-name>
Report test case starts
# testsuite_report_end_case <test-name>
Report test case ends
# testsuite_report_end_case_status <test-name>
Uses return value of previous shell command to determine success, if return is 0 will call testsuite_report_end_case_pass or if !0 will call testsuite_report_end_case_failure
# testsuite_report_end_case_pass <test-name>
Reports successful test case
# testsuite_report_end_case_failure <test-name>
Reports failed test case
# testsuite_check_skip <test-name>
Checks of the test name is in the skipped kernel args
# testsuite_check_allow <test-name>
Checks of the test name is in the allow kernel args
The test name should be <4 digits number>-.sh, the test should only call a single top-level “testsuite_report_start_case $1” with the associated “testsuite_report_end_case_pass $1” and can also report multiple sub-test cases by appending the sub-test name to $1 like “$1-sub-test-1”.
In order to run test testsuite, the meta-qcom-ci must be updated in: https://git.codelinaro.org/linaro/qcomlt/ci/staging/meta-qcom-ci/-/blob/master/recipes-test/ci-testsuite/ci-testsuite.bb
With:
SRC_URI = "git://git.codelinaro.org/linaro/qcomlt/ci/staging/ci-testsuite.git;protocol=https;branch=master"And
SRCREV = "221326f7bd234f3c2fdb7492eb9276b250f1957b"Then build with:
DISTRO=qcom-ci MACHINE=qcom-ci-arm64 bitbake initramfs-test-image-ciThe resulting initramfs image can be passed to cbda-testes with the following variable:
INITRAMFS_URL:
description: "URL of cpio.xz initramfs to use for test (Optional)"Or locally by running the initramfs with your locally built kernel.
TODO
Non-TuxSuite Linux Build
It would be good to have a non-tuxsuite linux build pipeline building the kernel in the job, probably by using tuxmake locally:
Private Testing
CDBA Boards pool
Tests parsing and aggregation
All tests results should be parsed and aggregated to be presented in a board with statistics on results and detected regressions.