CI
On this page
.github/workflows/pr.yml checks every pull request, every push to main and, once a night, the whole
repo. It needs no secrets, so it runs the same on forks. This page says which jobs run for which change,
and why. The emulator tests run apart from it, on main and nightly only (Emulator tests).
Only what a change needs
The free runner pool is shared by every open PR. On 2026-11-18 a PR's run took 50 to 65 minutes, and most of that was jobs waiting for a runner (run 44246997798: 14 jobs queued at 05:50, started between 05:54 and 06:36, none longer than 10 minutes). A job that doesn't run frees a runner for another PR.
- The
changesjob lists the files the change touches: for a PR, its merge commit against the base it merges into; for a push tomain, the push against the commit before it. tools/ci/changes.pymaps those files to jobs with the table below.- Every other job
needs: changesand has a job-levelif:. A job skipped this way takes no runner, and GitHub reports a skipped job as a success, so the required checks (app,web,scripts,root,wake-word) still pass the merge gate.
It fails closed. Without a base commit, when git diff fails, or when changes itself fails, every job
runs. A change to a workflow, to this selector or to a file every Gradle module or npm workspace builds
with runs every job. The nightly run (and a manual run, workflow_dispatch) runs every job too, to catch
anything the map misses.
Don't put workflow-level paths: on pr.yml. A workflow that doesn't start reports no check at all, and a
missing required check blocks the merge for good.
The map
Generated by tools/ci/changes.py --table. tools/ci/test/test_changes.py fails when this copy differs,
when a pr.yml job isn't gated on its row, or when a file the Gradle tests declare as an input wouldn't
run app.
A pattern matches a path and everything under it. * stays within one folder, ** crosses folders,
{a,b} is either, and ! takes a path back out.
| Job or step | Runs when a PR changes |
|---|---|
| every job | .github/workflows, .github/actions, tools/ci, settings.gradle.kts, build.gradle.kts, gradle.properties, gradlew, gradlew.bat, gradle, frames/*/design/frame.json, package.json, package-lock.json |
| app: every module's unit tests | libs, core, pane-api, engines, panes, testing, build-logic, frames/*/android, frames/*/panes, THIRD-PARTY-NOTICES.md, LICENSE, LICENSES, CHANGELOG.md, README.md, docs/**/*.md, frames/*/README.md, frames/**/blueprints/*.yaml, tools/release/frame-abis.sh, tools/release/check-update-compat.sh, tools/voice/fetch-models.sh, kiln/packages/brand/src/tokens.css, kiln/packages/setup-channel/test/aqara-s1-plus-transcript.json, branding/glass-tile/bootanimation.zip, frames/aqara-s1-plus/tools/bracket/base_emulator.py, frames/aqara-s1-plus/tools/debloat/debloat.sh, frames/generic-android/installer/src/devices.ts, frames/aqara-s1-plus/tools/radio/s1kiosk_radio.sh, **/sh/edm/**, frames/aqara-s1-plus/tools/radio/hotspot-name |
| app: release build, update check | libs, core, pane-api, engines, panes, testing, build-logic, frames/*/android, frames/*/panes, THIRD-PARTY-NOTICES.md, LICENSE, LICENSES, tools/release/check-update-compat.sh, frames/aqara-s1-plus/tools/radio/hotspot-name |
| app: aioesphomeapi tests | libs, build-logic |
| libs-api23: lint at minSdk 23 | libs, build-logic |
| api23 | libs, core, pane-api, engines, panes, testing, build-logic, frames/*/android, frames/*/panes, tools/legacy |
| libs-publish | libs, build-logic, LICENSE, LICENSES, THIRD-PARTY-NOTICES.md, tools/release/check-lib-artifact.sh, tools/release/lib-consumer-check.sh |
| wake-word | libs, build-logic, tools/voice |
| web | anything but **/*.md, {libs,core,pane-api,engines,panes,testing,build-logic,frames/*/android,frames/*/panes}/**/{*.kt,*.java}, {libs,core,pane-api,engines,panes,testing,build-logic,frames/*/android,frames/*/panes}/**/src/{test,androidTest}/**, {libs,core,pane-api,engines,panes,testing,build-logic,frames/*/android,frames/*/panes}/**/src/*/res/**, !testing/kotlin/dev/glassworks/testing/RepoFiles.kt, !core/src/main/res/font/** |
| refract | kiln/workers |
| scripts: Clarity tests (ffmpeg, fast and slow sets) | tools/perf, frames/*/budgets.json, frames/*/clarity, docs/perf.md |
| perf-webrtc | tools/perf, frames/*/budgets.json, frames/*/clarity |
| root | frames/aqara-s1-plus/tools/root, frames/aqara-s1-plus/tools/radio |
| size | libs, core, pane-api, engines, panes, testing, build-logic, frames/*/android, frames/*/panes, THIRD-PARTY-NOTICES.md, LICENSE, LICENSES, frames/*/budgets.json, tools/size, tools/release/frame-abis.sh, frames/aqara-s1-plus/tools/radio, kiln/assets.lock.json |
| size-generic | frames/generic-android, core, pane-api, panes, libs, engines, testing, build-logic, LICENSE, LICENSES, THIRD-PARTY-NOTICES.md, tools/size, tools/legacy/apk-icons.py, tools/release/check-update-compat.sh |
| branding | branding/boot, branding/glass-tile, branding/lululemon-mirror, branding/bootlogos.json, frames/aqara-s1-plus/design/frame.json, frames/lululemon-mirror/design/frame.json |
| site | site, packages, frames, docs, tools/perf/grade.ts, tools/perf/page/rubric.json, **/*.md, libs/*/build.gradle.kts, libs/*/gradle.properties; or deletes or renames any file |
scripts always runs: its install script tests, tool tests, this selector's tests and the docs' link check take seconds. Its
Clarity tests (ffmpeg, the fast and slow sets) run only on their row.
Some rows need explaining.
- app reads more than the modules. The unit tests read
docs/**/*.md,CHANGELOG.md,README.mdand a few files from Kiln and the frames (their list isTest.readsRepoFilesinbuild-logic/src/main/kotlin/glassworks.android.gradle.kts). So a docs-only PR still runs the unit tests. It skips the release build, which reads none of them. - web works the other way round. The npm workspaces read across the repo (the frames, the panes'
pane.jsonand icons, the core's setup page and font,RepoFiles.kt). The row lists what can't reach them, andwebruns for anything else. - size-generic builds from no S1 module, so an S1-only change skips it.
Emulator tests
.github/workflows/core-emulators.yml holds every test that boots an Android emulator. The maintainer moved
them off pull requests on 2026-11-19 (#536): five jobs of about 10 minutes each took runners from every
Android PR in a shared pool (see above).
| Job | What | Runs |
|---|---|---|
conformance (API 23, 26, 29, 34) |
The core's conformance test (core/src/androidTest) |
push to main when Android code changes, nightly at 03:17 UTC (pr.yml's time), by hand |
libs-api23 |
esphome-api-android and improv androidTest on Android 6 |
the same |
legacy-verify |
The generic app's legacy APK (the Mirror's) installed on Android 6: fails when its verifier rejects a Glassworks class (tools/legacy/verify-api23.sh, #435). Before a Mirror flash, run the script on the build you flash |
the same |
report |
Opens, updates or closes the red-main issue | after those two, on main only |
A PR still gets the fast checks for the same code: the unit tests (app), lint's NewApi at minSdk 23
(pr.yml's libs-api23 and api23). None of the emulator jobs is a required check, so a PR never waits
for them.
A red emulator run on main (push, nightly or manual) opens one issue, Emulator tests fail on main,
labelled area: ci, with a link to the run. Each later red run adds a comment to that issue. The next green
run on main's newest commit closes it (runs overlap, and an older commit's green run doesn't speak for a newer
one; it can wait for the nightly). Reporters run side by side, with no concurrency group: GitHub keeps only one
waiting job per group and drops the others, which could lose a failure. When two red runs both open an issue,
they keep the older one and close the other. A failure ends with its issue open, reopening it if a green run
closed it meanwhile, and a green run reads main's newest commit again just before each close. A cancelled run, or a manual run on another branch, reports nothing. To check a branch
before you merge, run the workflow on it by hand (Actions, Emulator tests, Run workflow) or with
gh workflow run core-emulators.yml --ref <branch>.
tools/ci/test/test_changes.py pins this: the workflow's triggers, no emulator in any other workflow, and
the report job after every job.
Cancelled runs
A new push to a PR cancels that PR's older run (concurrency with the PR's number). A push to main,
the nightly run and a manual run each get a group of their own (github.run_id), so nothing cancels or
holds them. So does a rerun (attempt 2 on, by hand or by Reruns): rerunning an older commit's jobs
must not cancel the newer commit's run in the PR's group. A new push doesn't cancel a rerun either. core-emulators.yml runs on no PR, so it has no group: every run finishes. The test builds (test-build.yml,
generic-test-build.yml) cancel per branch. Releases and the libraries' release never cancel.
Caches
Every Gradle job restores Gradle's home through gradle/actions/setup-gradle. Runs on main (each push and
the nightly run) save it, and PRs only read it, so a PR rebuilds and retests only what it changed.
setup-gradle saves Gradle's home even when the job fails. On 2026-11-18 that kept api23 red for every PR.
From 18:32, main's api23 couldn't resolve groovy-3.0.22.jar (and app, once, tensorflow-lite-2.16.1.aar),
though Maven Central served both. Most likely the repository missed once and Gradle recorded the artifact as
missing in its dependency metadata (inferred, not seen). Each later api23 run restored that record, failed
without asking again, and saved it again. So every Gradle job ends with a step that runs only on
failure and deletes ~/.gradle/caches/modules-2/metadata-*. The next run resolves its dependencies afresh,
and the downloaded files stay. tools/ci/test/test_changes.py checks that each Gradle job ends with it.
To clear a bad entry by hand, delete that job's gradle-home-v2|Linux-X64|<job>[…] entries on main
(gh cache list --ref refs/heads/main, then gh cache delete <id>). The next run starts from another
job's entry and saves a clean one.
Two more caches follow the same rule: runs that aren't for a pull request (each push to main, the nightly
run, a manual run) save them, and a PR only reads them.
| Cache | What | Named by | Why |
|---|---|---|---|
apt-<image>-<arch>-<packages>-<version> |
The .deb files of Ubuntu packages a job installs (ffmpeg in scripts and perf-webrtc), through .github/actions/apt-install |
The runner image (ImageOS, then ImageVersion) and the packages. Several image versions serve at once, so without its own version's entry a run restores another's, and apt fetches only what differs |
On 2026-11-18 and 19 apt-get update && apt-get install ffmpeg took 0.3 to 26 minutes (median 10 in perf-webrtc, which then hit its 30-minute limit) and failed twice on a mirror's 503. From the cache there's no apt-get update and no download: the files are installed by path, whatever versions the image's lists name. When that fails (another image's files), it tries the image's lists, then updated lists. The cache keeps only the files of the versions installed. It also skips man-db's index rebuild |
robolectric-<version>-<jars> |
Robolectric's Android jars (~/.m2/repository/org/robolectric), in app |
Robolectric's version, then the jars it holds; a run restores the newest for its version | Robolectric fetches them from Maven Central while the tests run, outside Gradle's caches, and a failed fetch fails the whole test class: 6 of the 13 red app runs on 2026-11-18 and 19 |
Clarity's own cache (~/.cache/glassworks-perf) isn't kept between runs: its tests make the camera streams
there, and a restored copy would change what they check.
Faster jobs
What else cut a job's time (#536, measured on the runs of 2026-11-18 and 19):
| Job | Before | Now |
|---|---|---|
root |
The 32-bit ARM containers under QEMU on x86: 17 minutes (median), the longest job | On an Arm runner (ubuntu-24.04-arm), whose CPU runs them itself: 6 minutes on #630's first run. Its first step runs the image's uname -m, and QEMU is set up only when that fails |
perf-webrtc |
Four end-to-end files one after the other, 9.5 minutes | Two halves on two runners, perf-webrtc (relay) (the peer, the relay, Chrome and Firefox; 4.6 minutes) and perf-webrtc (dialogs) (HA's more-info dialogs; 5.7). tools/ci/test/test_changes.py fails when a WebRTC or dialog test file is in neither |
branding |
The S1's frames, then the Mirror's: 2.5 and 5.5 minutes | One frame per runner, branding (aqara-s1-plus) and branding (lululemon-mirror). Side by side on one runner they took 4.3 and 7.2 minutes: the generator already uses every core |
The site's prerender. /blueprints reads GitHub's API while the site builds, and a dropped connection
(fetch failed) failed about one site run in seven. The build runs once more after that error only (it did on #630's first run, and passed).
Reruns
A download that fails on one runner tends to keep failing there. On #631's runs (2026-11-19) the plugin portal and
Maven Central both answered "not found" (HTTP 404) for files they have, three times in a row, 45 seconds apart, so a
retry inside the job didn't help. A rerun on a fresh runner passed every time: 5 of 5 portal failures on 2026-11-18
and 19, which were 5 of the 13 red app attempts.
So .github/workflows/rerun-download-flakes.yml follows PR checks. When a run's first attempt fails, it reads each
failed job's log with tools/ci/rerun_flakes.py: only the output of the steps that failed (each up to GitHub's
##[error]Process completed with exit code …), and in a step that retries itself (the site's build, apt-install), only
its last try. Every failed step must show a download failure and nothing else, and when Gradle says what went wrong,
each of its * What went wrong: blocks must be a download, or a test task's failing tests that all failed on
Robolectric's fetch. If every failed job passes that, it reruns the failed jobs once (gh run rerun --failed). It writes one line in its job summary and posts nothing on the PR.
| A download failure | What the log says |
|---|---|
| Gradle, a dependency | Could not find org.jetbrains.kotlin:kotlin-stdlib:2.4.10. |
| Gradle, a file | Could not find okio-jvm-3.15.0.jar (com.squareup.okio:okio-jvm:3.15.0) |
| Gradle, a plugin | Plugin [id: 'org.gradle.kotlin.kotlin-dsl', version: '6.7.6'] was not found in any of the following sources |
| Gradle, a request | Could not GET 'https://…', Received status code 502 from server |
| Robolectric's jars | java.lang.AssertionError at MavenArtifactFetcher.java:130 |
| apt | Failed to fetch … 503 Service Unavailable, Unable to fetch some archives |
| A dropped connection | TypeError: fetch failed with other side closed, ECONNRESET, ETIMEDOUT, EAI_AGAIN |
It doesn't rerun:
- a run whose failed jobs include one that failed on anything else (a test, a compile error, Clarity), or a job whose log shows such a failure next to a download one. A Gradle test that failed counts as a download only when the line under it is Robolectric's fetch;
- a log with
Could not find method …, a build script error, even next to a download line; - a download line in an earlier step that passed, or in a try that the step itself retried;
- a job without a log or without a failed step, a second attempt, a fork's run, or a PR's run when the PR has a newer commit (it looks again just before the rerun, after reading the logs);
- a cancelled run.
Like size-comment.yml, it runs from the default branch after the PR's run, checks out the default branch only for the
script, and reads only the run's jobs and logs. Its token may only rerun runs and read (actions: write,
contents: read). tools/ci/test/test_rerun_flakes.py tests the patterns against lines from real failed jobs, and the
workflow's trigger, conditions and permissions.
To see what it would have decided on a past run: tools/ci/rerun_flakes.py --repo GlassworksHQ/glassworks --run <id> --dry-run --any-attempt. Replayed on 12 red runs from 2026-11-18 and 19, it would have rerun the 9 that failed on
downloads only, and none of the 3 with a real failure.
Adding a job or a path
- Add the job to
pr.ymlwithneeds: changesandif: ${{ !cancelled() && (needs.changes.result != 'success' || needs.changes.outputs.<key> != 'false') }}(the key is the job id with_for-): unlesschangessucceeded, the job runs. - Add its row to
RULESintools/ci/changes.py, and its output to thechangesjob. - When a job starts reading a new file, add the file to its row.
- Run
tools/ci/changes.py --table, paste the table above, and runpython3 -m unittest discover -s tools/ci/test.