Repository navigation
Create the e2e GKE cluster with private nodes and no public control-plane IP - #845
Merged
Merged
Conversation
wallrj-cyberark
force-pushed
the
e2e-gke-private-cluster
branch
from
October 6, 2026 13:05
7319f13 to
427d55a
Compare
…lane IP - The nightly e2e started failing on 2026-10-06 because a new org policy on the CI project rejects GKE clusters with public nodes or a public control-plane endpoint. - Create the cluster with private nodes and a private IP endpoint, and use the DNS-based endpoint so the GitHub runner can still reach the API server. - The private nodes depend on a Cloud NAT in the project's default network. The CI service account cannot create one, so it must be set up separately. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Richard Wall <richard.wall@cyberark.com>
wallrj-cyberark
force-pushed
the
e2e-gke-private-cluster
branch
from
October 6, 2026 13:10
427d55a to
98eaa85
Compare
wallrj-cyberark
marked this pull request as ready for review
October 6, 2026 14:53
Collaborator
|
I've created the Cloud NAT in gcloud compute routers create jetstack-secure-e2e \
--project machineidentitysecurity-jsci-e --region europe-west1 --network default
gcloud compute routers nats create jetstack-secure-e2e \
--project machineidentitysecurity-jsci-e --region europe-west1 --router jetstack-secure-e2e \
--auto-allocate-nat-external-ips --nat-all-subnet-ip-rangesThese match the commands in the PR description. The region is europe-west1 because the e2e workflow uses zone
The NAT covers every subnet in the The |
mladen-rusev-cyberark
approved these changes
Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nightly GKE e2e test fails because the CI project now rejects the cluster we create. This PR creates a cluster that the new policy allows. It cannot pass until someone creates a Cloud NAT in the project — see below.
What is broken
Since 2026-10-06,
gcloud container clusters createinhack/e2e/test.shfails with:The nightly run on 2026-10-05 passed with no code changes in between, so the policy is new. Failing run: https://github.com/jetstack/jetstack-secure/actions/runs/37402574029/job/112072803252
What this changes
--enable-private-nodes, which needs--enable-ip-alias) and no public control-plane IP (--enable-private-endpoint).--enable-dns-access, thenget-credentials --dns-endpoint). Access to that endpoint is authorised with IAM.Before you merge: the project needs a Cloud NAT
Private nodes have no internet access, so they cannot pull cert-manager from the Venafi registry or reach the Venafi API. Someone with network admin rights on
machineidentitysecurity-jsci-eneeds to run this once:The CI service account (
gke-cluster-creation) cannot do this itself: it lackscompute.routers.create. I tried creating the NAT from the script first and it failed with a 403.Test evidence: the cluster now passes the policy and the runner can reach it
Run 37468760730 on this PR:
Created [.../clusters/test-secretless-261006-131128]), so the org policy accepts it.get-credentials --dns-endpointsucceeded andkubectl create ns venafiworked, so the runner can reach the API server through the DNS endpoint.venctl components kubernetes applythen timed out after 10 minutes withresource Deployment/venafi/cert-manager-cainjector not ready ... Available: 0/1. This is the missing NAT: the images come from the external Venafi registry. The agent images in Artifact Registry are reachable without NAT through Private Google Access.Once the NAT exists, re-run the
test-e2ejob on this PR to confirm.[with Claude]