Skip to content

Create the e2e GKE cluster with private nodes and no public control-plane IP - #845

Merged
wallrj-cyberark merged 1 commit into
masterfrom
e2e-gke-private-cluster
Oct 6, 2026
Merged

wallrj-cyberark merged 1 commit into
masterfrom
e2e-gke-private-cluster

Conversation

@wallrj-cyberark

@wallrj-cyberark wallrj-cyberark commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

The nightly GKE e2e test fails because the CI project now rejects the cluster we create. This PR creates a cluster that the new policy allows. It cannot pass until someone creates a Cloud NAT in the project — see below.

What is broken

Since 2026-10-06, gcloud container clusters create in hack/e2e/test.sh fails with:

Operation denied by org policy: ["customConstraints/custom.disableGkePublicNodes": "Disables the creation of GKE clusters with public nodes." "customConstraints/custom.disableGkePublicControlPlane": "Disables the creation of GKE clusters with public control plane endpoints."]

The nightly run on 2026-10-05 passed with no code changes in between, so the policy is new. Failing run: https://github.com/jetstack/jetstack-secure/actions/runs/37402574029/job/112072803252

What this changes

  • The cluster now has private nodes (--enable-private-nodes, which needs --enable-ip-alias) and no public control-plane IP (--enable-private-endpoint).
  • The GitHub runner reaches the API server through the cluster's DNS-based endpoint (--enable-dns-access, then get-credentials --dns-endpoint). Access to that endpoint is authorised with IAM.

Before you merge: the project needs a Cloud NAT

Private nodes have no internet access, so they cannot pull cert-manager from the Venafi registry or reach the Venafi API. Someone with network admin rights on machineidentitysecurity-jsci-e needs to run this once:

gcloud compute routers create jetstack-secure-e2e \
  --project machineidentitysecurity-jsci-e --region europe-west1 --network default
gcloud compute routers nats create jetstack-secure-e2e \
  --project machineidentitysecurity-jsci-e --region europe-west1 --router jetstack-secure-e2e \
  --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges

The CI service account (gke-cluster-creation) cannot do this itself: it lacks compute.routers.create. I tried creating the NAT from the script first and it failed with a 403.

Test evidence: the cluster now passes the policy and the runner can reach it

Run 37468760730 on this PR:

  • The cluster was created (Created [.../clusters/test-secretless-261006-131128]), so the org policy accepts it.
  • get-credentials --dns-endpoint succeeded and kubectl create ns venafi worked, so the runner can reach the API server through the DNS endpoint.
  • venctl components kubernetes apply then timed out after 10 minutes with resource Deployment/venafi/cert-manager-cainjector not ready ... Available: 0/1. This is the missing NAT: the images come from the external Venafi registry. The agent images in Artifact Registry are reachable without NAT through Private Google Access.

Once the NAT exists, re-run the test-e2e job on this PR to confirm.

[with Claude]

@wallrj-cyberark wallrj-cyberark added the test-e2e To signal e2e test job to be run label Oct 6, 2026
@wallrj-cyberark
wallrj-cyberark force-pushed the e2e-gke-private-cluster branch from 7319f13 to 427d55a Compare October 6, 2026 13:05
…lane IP

- The nightly e2e started failing on 2026-10-06 because a new org policy on
  the CI project rejects GKE clusters with public nodes or a public
  control-plane endpoint.
- Create the cluster with private nodes and a private IP endpoint, and use
  the DNS-based endpoint so the GitHub runner can still reach the API server.
- The private nodes depend on a Cloud NAT in the project's default network.
  The CI service account cannot create one, so it must be set up separately.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Richard Wall <richard.wall@cyberark.com>
@wallrj-cyberark
wallrj-cyberark force-pushed the e2e-gke-private-cluster branch from 427d55a to 98eaa85 Compare October 6, 2026 13:10
@wallrj-cyberark
wallrj-cyberark marked this pull request as ready for review October 6, 2026 14:53
@mladen-rusev-cyberark

Copy link
Copy Markdown
Collaborator

I've created the Cloud NAT in machineidentitysecurity-jsci-e. I used my mladen.rusev@cyberark.com account, which has owner rights on the project through gcp_1034149387603_owner@cyberark.com.

gcloud compute routers create jetstack-secure-e2e \
  --project machineidentitysecurity-jsci-e --region europe-west1 --network default

gcloud compute routers nats create jetstack-secure-e2e \
  --project machineidentitysecurity-jsci-e --region europe-west1 --router jetstack-secure-e2e \
  --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges

These match the commands in the PR description. The region is europe-west1 because the e2e workflow uses zone europe-west1-b.

gcloud compute routers nats describe shows:

Field Value
natIpAllocateOption AUTO_ONLY
sourceSubnetworkIpRangesToNat ALL_SUBNETWORKS_ALL_IP_RANGES

The NAT covers every subnet in the default network in europe-west1, not only the e2e cluster. It also has a small ongoing cost.

The test-e2e job can be re-run now.
[generated by Claude]

@wallrj-cyberark
wallrj-cyberark merged commit 6408677 into master Oct 6, 2026
8 of 10 checks passed
@wallrj-cyberark
wallrj-cyberark deleted the e2e-gke-private-cluster branch October 6, 2026 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test-e2e To signal e2e test job to be run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants