Setting up AOSP CI/CD Pipeline
Overview
Setting up an effective CI/CD pipeline for projects based on AOSP (Android Open Source Project) presents unique challenges due to the underlying AOSP build system, device customizations, and vendor adaptations. Traditional CI/CD systems often require significant customization or struggle to efficiently manage AOSP-based workflows, such as validating changes that span across multiple Git repositories simultaneously.
HexDroid offers a specialized CI/CD system designed specifically to address the intricacies of AOSP development. It streamlines the process by providing built-in support for common AOSP tasks and workflows, simplifying configuration and improving efficiency where generic solutions may fall short.
This guide will walk you through configuring a robust CI/CD pipeline for your AOSP project using HexDroid. We'll cover recommended practices and provide practical examples for creating essential pipelines:
- Pull Request Validation: Ensuring code quality and integration integrity before merging changes.
- Nightly Builds: Automating regular builds of the main development branch for internal testing and validation.
By following this guide, you'll learn how to leverage HexDroid to automate your AOSP build, test, and release processes, leading to faster development cycles and more reliable builds.
HexDroid recommends having a separate repository where all AOSP CI/CD workflows will live.
In this article we assume the repository name is aosp-ci.
Validating merge/pull requests
Pull request validation is a critical element of any AOSP development workflow. It ensures your team catches integration issues early, before they reach the main branch.
- Create the Workflow File
- Configure the Pipeline: Use the template below as a starting point and customize it to your project's needs.
- Commit and Push:
Commit the pipeline file to your repository and push it to the
mainbranch. - Verify Setup: Create a test pull request to verify that the validation pipeline is triggered correctly.
In your aosp-ci repository, create a new workflow file at .hexdroid/pr-build.yaml.
name: My AOSP Validation Build
version: 1.0
trigger:
- pull_request:
repo_regex: "MyOrg/aosp-manifest"
source_branch_regex: "(task|feature)/(.*)"
target_branch_regex: "aosp-15"
comment:
enabled: true
stages:
aosp_cf_x86_64_phone:
agent:
container:
image: my_aosp_builder:latest
steps:
- aosp:
manifest:
url: https://android.googlesource.com/platform/manifest
revision:
from: source_branch
name: default.xml
lunch:
product_name: aosp_cf_x86_64_phone
release_config:
- trunk_staging
- # this empty entry builds without any release config
build_variant:
- eng
- userdebug
- user
build:
commands:
- m installclean
- m dist
artifact:
paths:
- out/dist/otatools.zip
- out/dist/aosp_cf_x86_64_phone-target_files-root.zip
- out/dist/aosp_cf_x86_64_phone-ota-root.zip
Developer workflow
AOSP is a multi-repository project, which can make it challenging to validate changes that span across multiple repositories. Traditional CI systems often fall short in these scenarios. HexDroid CI is built to handle this complexity by supporting multi-repo workflows out of the box.
Imagine a developer needs to add a new HAL API. In this case:
- The API definition lives in the
vendor/my-org/hardware/repository. - The device-specific implementation is in
device/my-org/hardware/repository.
Since these changes are closely related, they need to be validated together as part of the same build. HexDroid CI allows this by using a temporary manifest branch.
To group the changes, the developer can create a temporary branch in the manifest repository that pins both affected projects to the appropriate feature branch. For example:
- <project path="vendor/my-org/hadrware" name="aosp-vendor-my-org-hardware" />
- <project path="device/my-org/hadrware" name="aosp-device-my-org-hardware" />
+ <project path="vendor/my-org/hadrware" name="aosp-vendor-my-org-hardware" revision="feature/add-hal-api" />
+ <project path="device/my-org/hadrware" name="aosp-device-my-org-hardware" revision="feature/add-hal-api" />
After creating a temp PR with this manifest branch, HexDroid CI will:
- Sync the code using the custom manifest.
- Build the combined changes.
This approach ensures that cross-repository changes can be tested in a single environment like they would be once merged. It reduces integration risk and improves developer confidence before merging the change.
Once the build is finished, and pull request(s) are ready to merge - then the developer can merge the
vendor/my-org/hardware/
and device/my-org/hardware/, and decline the manifest temp pull request.
If a job needs to be re-triggered, leaving a comment [CI CHECK] on the pull request will re-trigger the job.
Pipeline Options Explained
Let's break down the key parts of the pull request validation configuration:
Manifest Configuration
manifest:
url: https://android.googlesource.com/platform/manifest
+ revision:
+ from: source_branch
name: default.xml
HexDroid automatically selects the correct manifest revision based on the pipeline trigger.
In this example, revision.from: source_branch means the manifest will be checked out using the same branch
that triggered the job, for example: feature/add-hal-api.
Nightly builds
The goal of having nightly builds is to provide the latest integrated software to developers/testing teams, to achieve this the CI/CD pipeline must do the following:
- Build&Test the software
- Create an internal distribution build for developers/testing teams
In aosp-ci repository, create a new file at .hexdroid/nightly.yaml path, with the following starter template:
name: My AOSP Nightly Build
version: 1.0
trigger:
- cron:
ref: refs/heads/master
expression: "0 0 * * *"
stages:
aosp_cf_x86_64_phone:
agent:
container:
image: my_aosp_builder:latest
steps:
- aosp:
manifest:
url: https://android.googlesource.com/platform/manifest
revision: main
name: default.xml
lunch:
product_name: aosp_cf_x86_64_phone
release_config:
- trunk_staging
- # this empty entry builds without any release config
build_variant:
- eng
- userdebug
- user
build:
commands:
- m installclean
- m dist
artifact:
paths:
- out/dist/api/*
- out/host/linux-x86/cvd-host_package.tar.gz
- out/dist/otatools.zip
- out/dist/aosp_cf_x86_64_phone-target_files-root.zip
- out/dist/aosp_cf_x86_64_phone-ota-root.zip
release:
payloads:
- file:
path: out/dist/aosp_cf_x86_64_phone-ota-root.zip
type: ota
distributed: true
- file:
path: out/dist/aosp_cf_x86_64_phone-target_files-root.zip
type: target_files
distributed: false
- file:
path: out/dist/otatools.zip
type: otatools
distributed: false
meta:
target:
type: execs
separator: /
command:
- get_build_var PRODUCT_BRAND
- get_build_var TARGET_DEVICE
- get_build_var TARGET_PRODUCT
- get_build_var TARGET_BUILD_VARIANT
- get_build_var BUILD_VERSION_TAGS
version_code:
type: file
path: out/build_date.txt
version_name:
type: file
path: out/dist/build.prop
regex: "ro\\.system\\.build\\.id=(.+)"
Pipeline Options Explained
Let's break down the key parts of the night build pipeline configuration:
Trigger
Starter template sets up cron configuration so that your nightly AOSP pipeline will be triggered at midnight UTC every day. This can be expanded to have multiple cron triggers or even be triggered on push to the main branch of any AOSP repository. Refer to triggers documentation to find out all available triggers.
Stages
The template pipeline only defines one stage; however, in the production environment you will likely have more than one AOSP build target (e.g., multiple products, which can be based on different aosp versions).
HexDroid recommends having a stage per product, especially if these products do not share the same AOSP base, having multiple stages means stages can run in parallel so that the total time to build all products is not a sum of each product.
Example:
stages:
product_1:
# configuration for product 1
product_2:
# configuration for product 2
product_3:
# configuration for product 3
Different products can also be split into their own pipeline files so that they are concise.
Agent
This defines the environment of the stage; the most common configuration is to specify a container image, which provides a consistent and reproducible runtime for all commands in the stage.
Steps
In this example, we only have one aosp step defined. aosp step is HexDroid defined step, designed to make CI/CD for
AOSP understandable, and declarative.
Full documentation for the aosp step can be found here.
This step will be expanded into multiple steps, such as manifest sync and pinning, then building for each product
sub-configuration, such as user, userdebug builds.
This pipeline provides some opinionated options, such as what artifacts to upload, as well as how to set up OTA releases.
CI Artifacts
At minimum HexDroid recommends uploading the following build artifacts:
out/dist/otatools.zipout/dist/*-target_files-root.zipout/dist/*-ota-root.zip
When CI uploads build artifacts, they will be nicely grouped by aosp lunch target, which makes it easier to find a relevant file, especially when file names are the same:
OTA Release
aosp step comes with built-in release update(ota) management, which makes it easy to create release, distribute it for
your internal users, as well as later rolling out to production.
When uploading OTA releases, at minimum it's recommended to upload the following payloads:
release:
payloads:
- file:
path: out/dist/aosp_cf_x86_64_phone-ota-root.zip
type: ota
distributed: true
- file:
path: out/dist/aosp_cf_x86_64_phone-target_files-root.zip
type: target_files
distributed: false
- file:
path: out/dist/otatools.zip
type: otatools
distributed: false
aosp_cf_x86_64_phone-ota-root.zip is set distributed: true as this file is available to end target to download and
apply,
We also upload target_files and otatools as distributed: false - this means these files will be visible in
HexDroid
Release Management,
and these files later can be used to re-sign builds with release/production keys.
Files that are uploaded in release step will be automatically included as CI artifact - so there is no need to
duplicate paths, in artifact and release steps.