ci(qualification): add Ubuntu 24.04 k3s scenario by purp · Pull Request #3651 · NVIDIA/OpenShell

purp · 2026-09-23T23:54:54Z

Summary

Add a disposable Ubuntu 24.04 k3s qualification lane that installs the candidate OpenShell binaries and images, then runs the shared conformance suite. The same environment, installer, and testsuite tuple works locally through tmachine and in GitHub Actions.

Related Issue

Closes #3455

Changes

Pin and checksum k3s v1.36.4+k3s1 for x86_64 and aarch64, wait for the API and node, and report the effective versions and node state.
Install the candidate gateway and runtime images against k3s with a guest-local kubeconfig and a pinned Agent Sandbox controller manifest.
Run k3s setup, installation, and conformance in one disposable VM; stop the VM on success, failure, or interruption.
Collect bounded, redacted guest diagnostics on failure and upload the diagnostic artifact from Actions.
Add the k3s tuple to branch and integration qualification and document the lane.

Testing

mise run pre-commit passes
mise run test passes
mise run ci passes
tmachine unit tests pass (cargo test --manifest-path tests/tmachine/Cargo.toml --locked)
Ansible syntax checks pass for the new playbooks
Unit tests added/updated
E2E tests added/updated (qualification matrix)

The k3s VM scenario needs Nix and QEMU, which are unavailable on this macOS host. The Branch E2E run passed, including the Ubuntu k3s conformance job. All PR checks passed after retrying transient Linux process-identity test and Darwin Nix cache failures.

Checklist

Follows Conventional Commits
Commits are signed off (DCO) and cryptographically signed
Architecture docs updated

github-actions · 2026-09-23T23:55:50Z

Label test:e2e applied for 46859f9. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

purp

gator-agent

PR Review Status

This qualification work matches linked issue #3455, but the new VM cleanup path can hide a QEMU crash and let a failed qualification run appear successful.

Action required: preserve nonzero QEMU exit failures while treating only Gator-initiated fallback termination as successful cleanup, and add the regression coverage described inline.

Blocking findings:

GATOR-c3b797ec-01: QemuVm::stop ignores independently nonzero QEMU exits.

Carried findings:

None

Non-blocking suggestions:

None

Gator metadata

Validation: Project-valid maintainer-authored implementation of linked issue #3455
Docs: Architecture documentation updated; no direct user-facing Fern docs change is required
Checks: OpenShell Branch Checks is failing and the current-head E2E run is still in progress
E2E: test:e2e is applied and current-head workflows are running
Head SHA: c3b797ec7dbfb29b07121b07ad1e88db4e8f118e
Base SHA: 679b1906776ce9315f9311002474b828b6323495
Merge base SHA: bfd126868c3a2385e7c13ebc22680282e796d50c
Patch ID: 87993302653dbfa658fa9407d35238c2306666b6
Gator payload: 9
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:in-review

purp

gator-agent

Re-check After Author Update

Thanks @purp. I checked your update preserving independently observed nonzero QEMU exits, accepting only tmachine-initiated fallback termination, and covering both status-42 paths plus fallback cleanup. The prior finding is resolved, and the independent follow-up review found no new blockers in the author-only delta.

Disposition: resolved. Required Branch Checks and E2E workflows are now running, so Gator is moving to pipeline monitoring.

Blocking findings:

No blocking findings remain

Carried findings:

None

Gator metadata

Validation: Project-valid maintainer-authored implementation of linked issue #3455
Docs: Architecture documentation updated; no direct user-facing Fern docs change is required
Checks: Required current-head Branch Checks and E2E workflows are queued or running
E2E: test:e2e is applied and the current-head workflow is active
Head SHA: 393c1bc6b5d707e07832daf475d0051aca155707
Base SHA: 679b1906776ce9315f9311002474b828b6323495
Merge base SHA: bfd126868c3a2385e7c13ebc22680282e796d50c
Patch ID: 4183c7a05720caa7ad49393d3e148562d7ef5848
Gator payload: 9
Review mode: follow_up
Previous reviewed SHA: c3b797ec7dbfb29b07121b07ad1e88db4e8f118e
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:watch-pipeline

purp · 2026-09-24T17:35:27Z

Reviewed commit 393c1bc6b5d707e07832daf475d0051aca155707. Two follow-up fixes are needed:

Require clean shutdown before publishing reusable VM caches. Previously, Docker/Podman cache creation waited for guest shutdown and a successful QEMU exit. The new QemuVm::stop() treats its own forced termination as success, including when shutdown exceeds 30 seconds. cached_layer() then promotes that disk into the reusable cache. This can preserve incomplete filesystem or service state and cause repeated failures in later runs. Preserve the distinction between clean shutdown and forced cleanup, and reject cache publication after forced termination. Forced cleanup remains appropriate for disposable guests. References: vm.rs, layer.rs.
Preserve failure diagnostics when Kubernetes requests stall. The diagnostics playbook issues Kubernetes requests without individual timeouts, while the Rust collector buffers output until the entire playbook exits. If the outer 90-second deadline expires, the buffer is discarded, no diagnostic artifact is written, and the disposable guest is removed. Even guest/systemd evidence collected earlier is lost. A local fixture against the unchanged collector confirmed that already-emitted output produces no artifact after timeout. Bound individual requests and preserve sanitized diagnostic sections as they complete. References: ansible.rs, diagnostics/k3s.yaml.

Verification: the current revision's k3s CI job passed, and all five existing tmachine unit tests passed locally. Those checks cover the successful qualification path and the prior QEMU exit-status fix; the two cases above need regression coverage. The full VM scenario was not rerun locally because Nix is unavailable.

Each finding will be addressed in a separate commit on this PR.

elezar · 2026-09-29T11:35:48Z

+# SPDX-License-Identifier: Apache-2.0
+
+---
+- name: Provision disposable k3s cluster


Question: Does Ansible galaxy offer a way to install this similar to Docker?

elezar · 2026-09-29T11:37:26Z

+    #[serde(default)]
+    pub variables: BTreeMap<String, String>,
+    #[serde(default)]
+    pub ephemeral: bool,


What is the purpose of ephemeral? Should this really be tied to a specific environment?

elezar · 2026-09-29T11:38:55Z

+        name = "kubernetes-binaries";
+        use_galaxy = false;
+        playbooks = [
+          "ansible/playbooks/openshell.yaml"
+          "ansible/playbooks/gateway-kubernetes.yaml"
+        ];
+        inputs = {
+          openshell_cli_binary = "../artifacts/binaries/${muslTarget}/openshell";
+          openshell_gateway_binary = "../artifacts/binaries/${gnuTarget}/openshell-gateway";
+          openshell_supervisor_image = "../artifacts/images/openshell-supervisor-tmachine.tar";
+          openshell_sandbox_image = "../artifacts/images/openshell-sandbox-tmachine.tar";
+        };
+      }


Why don't we use helm? This seems like we're re-implementing logic that would already be present?

elezar · 2026-09-29T11:39:31Z

      ansible.builtin.command: "{{ item }}"
      loop:
-        - k3s version
+        - k3s --version


Should probably just be a fixup commit.

elezar · 2026-09-29T11:40:42Z


    pub(super) async fn stop(mut self) -> Result<()> {
-        if self.child.try_wait().context("check QEMU guest state")?.is_some() {
+        if let Some(status) = self.child.try_wait().context("check QEMU guest state")? {


It may make sense to separate these changes into a different PR so that we're not tiying them to adding k3s support.

copy-pr-bot · 2026-09-29T12:34:20Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

purp · 2026-09-29T22:46:59Z

Closing this draft in favor of #3848. Its smaller tmachine and Helm approach is the preferred direction; remaining requirements from #3455 can be addressed there.

purp requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners September 23, 2026 23:54

purp added the test:e2e Requires end-to-end coverage label Sep 23, 2026

purp mentioned this pull request Sep 23, 2026

ci(qualification): add Ubuntu 24.04 k3s scenario #3455

Open

7 tasks

purp added this to the OpenShell 0.1.0 milestone Sep 24, 2026

purp commented Sep 24, 2026

View reviewed changes

Comment thread tests/tmachine/src/qemu/vm.rs Outdated

purp added the gator:in-review Gator is reviewing or awaiting PR review feedback label Sep 24, 2026

purp commented Sep 24, 2026

View reviewed changes

purp added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 24, 2026

purp self-assigned this Sep 24, 2026

purp mentioned this pull request Sep 24, 2026

bug: managed-workspace E2E tests intermittently fail with SQLite lock contention #3679

Open

4 tasks

drew modified the milestones: OpenShell 0.1.0, OpenShell 0.1.1, OpenShell 0.1.4 Sep 25, 2026

elezar reviewed Sep 29, 2026

View reviewed changes

purp marked this pull request as draft September 29, 2026 12:34

SDAChess mentioned this pull request Sep 29, 2026

test(tmachine): add K3s conformance scenario #3848

Open

9 tasks

purp added 5 commits September 29, 2026 08:57

test(tmachine): add local Ubuntu k3s conformance scenario

0bd4984

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

fix(tmachine): enforce clean shutdown for reusable disks

ae3234f

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

feat(tmachine): run k3s qualification in a disposable guest

1e1ec70

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

feat(tmachine): retain sanitized k3s failure diagnostics

2925d6a

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

ci(qualification): run Ubuntu k3s conformance in Actions

5e13dff

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

purp force-pushed the feat/3455-ubuntu-k3s-qualification/purp branch from c8e53de to 5e13dff Compare September 29, 2026 16:13

purp closed this Sep 29, 2026

Navigation Menu

ci(qualification): add Ubuntu 24.04 k3s scenario - #3651

ci(qualification): add Ubuntu 24.04 k3s scenario#3651
purp wants to merge 5 commits into
NVIDIA:mainfrom
purp:feat/3455-ubuntu-k3s-qualification/purp

purp commented Sep 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

purp left a comment

Uh oh!

Uh oh!

purp left a comment

Uh oh!

purp commented Sep 24, 2026

Uh oh!

elezar Sep 29, 2026

Uh oh!

elezar Sep 29, 2026

Uh oh!

elezar Sep 29, 2026

Uh oh!

elezar Sep 29, 2026

Uh oh!

elezar Sep 29, 2026

Uh oh!

copy-pr-bot Bot commented Sep 29, 2026

Uh oh!

purp commented Sep 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

purp commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

purp left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

purp left a comment

Choose a reason for hiding this comment

Re-check After Author Update

Uh oh!

purp commented Sep 24, 2026

Uh oh!

elezar Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

elezar Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

elezar Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

elezar Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

elezar Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

copy-pr-bot Bot commented Sep 29, 2026

Uh oh!

purp commented Sep 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

purp commented Sep 23, 2026 •

edited

Loading