Jubin Soni - Portfolio & Blog

A deployment pipeline that runs kubectl apply from CI has a quiet flaw: the cluster's real state is whatever the last pipeline run happened to push, plus whatever someone changed by hand since. Nobody can say for sure what is running or why. GitOps fixes that by reversing the direction. The cluster pulls its desired state from git and continuously reconciles toward it.

Two things landed recently that make a modern version of this setup worth writing down. The Kubernetes Gateway API reached v1.0 in late 2023, with Gateway, GatewayClass and HTTPRoute now stable. And Argo CD and Argo Rollouts have matured into a solid pair for delivery and progressive rollout. This post wires them together: CI builds the image, a git commit changes the desired state, Argo CD syncs it, and Argo Rollouts shifts traffic gradually through a Gateway API HTTPRoute while checking metrics.

The shape of the system

The split between two repositories is deliberate:

Repo	Contains	Changed by
App repo	Source, Dockerfile, unit tests, CI workflow	Developers
GitOps repo	Manifests, Kustomize overlays, Argo CD Applications	CI bot (image tags), platform team (structure)

Keeping them apart means a deploy is a small, reviewable commit, a rollback is a git revert, and the audit trail is the git log.

Repository layout

gitops/
├── apps/
│   └── shop-api/
│       ├── base/
│       │   ├── rollout.yaml
│       │   ├── services.yaml
│       │   ├── httproute.yaml
│       │   ├── analysis-template.yaml
│       │   └── kustomization.yaml
│       └── overlays/
│           ├── staging/kustomization.yaml
│           └── production/kustomization.yaml
├── platform/
│   └── gateway.yaml
└── argocd/
    └── applicationset.yaml

Gateway API in two minutes

Ingress was always underspecified, and every controller filled the gaps with annotations. Gateway API splits the roles properly:

Resource	Owned by	Purpose
`GatewayClass`	Infrastructure provider	Which implementation (Envoy Gateway, Istio, GKE, Cilium, NGINX Gateway Fabric)
`Gateway`	Platform team	Listeners, ports, TLS, which namespaces may attach routes
`HTTPRoute`	Application team	Hostnames, paths, headers, backends and weights

Weighted backends in an HTTPRoute are part of the standard, which is exactly what canary releases need. No vendor annotations required.

yaml

# platform/gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public
  namespace: gateway-system
spec:
  gatewayClassName: envoy            # use the class your controller provides
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-example-com
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: "true"

yaml

# apps/shop-api/base/httproute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop-api
spec:
  parentRefs:
    - name: public
      namespace: gateway-system
  hostnames: ["shop.example.com"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /api }
      backendRefs:
        - name: shop-api-stable
          port: 80
          weight: 100
        - name: shop-api-canary
          port: 80
          weight: 0

The Rollout

Argo Rollouts replaces the Deployment with a Rollout resource that understands steps and analysis. It talks to the Gateway API through a traffic router plugin, which edits the weights in your HTTPRoute for you.

First enable the plugin in the Rollouts controller configuration. Use the release that matches your Argo Rollouts version.

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-rollouts-config
  namespace: argo-rollouts
data:
  trafficRouterPlugins: |-
    - name: "argoproj-labs/gatewayAPI"
      location: "https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/releases/download/<version>/gatewayapi-plugin-linux-amd64"

Then the Rollout itself:

yaml

# apps/shop-api/base/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: shop-api
spec:
  replicas: 6
  revisionHistoryLimit: 3
  selector:
    matchLabels: { app: shop-api }
  template:
    metadata:
      labels: { app: shop-api }
    spec:
      containers:
        - name: api
          image: registry.example.com/shop-api:1.8.1   # bumped by CI
          ports: [{ containerPort: 8080 }]
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 5
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits:   { memory: 512Mi }
  strategy:
    canary:
      stableService: shop-api-stable
      canaryService: shop-api-canary
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: shop-api
            namespace: shop
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis:
            templates: [{ templateName: error-rate }]
        - setWeight: 50
        - pause: { duration: 5m }
        - analysis:
            templates: [{ templateName: error-rate }]
        - setWeight: 100

The two Services select the same pods by label, and Rollouts adds a rollouts-pod-template-hash selector to each so that stable points at the old ReplicaSet and canary at the new one:

yaml

# apps/shop-api/base/services.yaml
apiVersion: v1
kind: Service
metadata: { name: shop-api-stable }
spec:
  selector: { app: shop-api }
  ports: [{ port: 80, targetPort: 8080 }]
---
apiVersion: v1
kind: Service
metadata: { name: shop-api-canary }
spec:
  selector: { app: shop-api }
  ports: [{ port: 80, targetPort: 8080 }]

Automated analysis

The step that makes it progressive delivery instead of just slow delivery is analysis. If the canary's error rate is bad, the rollout aborts and traffic returns to stable, with no human awake.

yaml

# apps/shop-api/base/analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  metrics:
    - name: http-error-rate
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            sum(rate(http_requests_total{service="shop-api-canary",code=~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="shop-api-canary"}[2m]))

Check what happens when the canary receives almost no traffic. A ratio of two near-zero numbers is noisy, and NaN results from an empty query can fail or pass depending on your configuration. I add a minimum-traffic condition, or make the 10 percent step long enough to collect meaningful samples.

Argo CD: one Application per environment

Argo CD watches the GitOps repo. An ApplicationSet generates Applications from directories, so adding a service is a new folder, not a new hand-written Application.

yaml

# argocd/applicationset.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: apps
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - git:
              repoURL: https://github.com/acme/gitops.git
              revision: main
              directories:
                - path: apps/*
          - list:
              elements:
                - env: staging
                  cluster: https://staging.k8s.example.com
                - env: production
                  cluster: https://prod.k8s.example.com
  template:
    metadata:
      name: '{{path.basename}}-{{env}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/acme/gitops.git
        targetRevision: main
        path: 'apps/{{path.basename}}/overlays/{{env}}'
      destination:
        server: '{{cluster}}'
        namespace: '{{path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
      ignoreDifferences:
        - group: gateway.networking.k8s.io
          kind: HTTPRoute
          jsonPointers:
            - /spec/rules/0/backendRefs/0/weight
            - /spec/rules/0/backendRefs/1/weight

That ignoreDifferences block is the one people forget. Argo Rollouts changes the HTTPRoute weights at runtime. Without it, Argo CD sees drift and selfHeal fights the rollout by resetting the weights, and you get a very confusing deploy.

The CI side: a commit, not a kubectl

CI builds and pushes the image, then opens a change in the GitOps repo. For production I prefer a pull request that auto-merges after checks, so that policy checks run and there is a visible record.

yaml

# .github/workflows/release.yml (app repo)
  promote:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          repository: acme/gitops
          token: ${{ secrets.GITOPS_TOKEN }}
          path: gitops
      - name: Bump image tag
        working-directory: gitops/apps/shop-api/overlays/staging
        run: |
          kustomize edit set image registry.example.com/shop-api=registry.example.com/shop-api@${{ needs.build.outputs.digest }}
      - name: Commit
        working-directory: gitops
        run: |
          git config user.name  "release-bot"
          git config user.email "release-bot@example.com"
          git commit -am "shop-api: deploy ${GITHUB_SHA::7} to staging"
          git push

Pin by digest. Tags move, and a GitOps repo that says 1.8.1 does not tell you which bits 1.8.1 was on Tuesday.

When it goes wrong

Situation	What happens	What you do
Analysis fails	Rollout aborts, traffic returns to stable	Look at the failing metric, fix forward or revert the commit
Bad config, pods never become ready	Rollout stays paused at the first step	`kubectl argo rollouts abort shop-api`, then revert
Someone edits the live Rollout by hand	`selfHeal` reverts it within minutes	Make the change in git
Argo CD cannot reach the repo	Cluster keeps running the last synced state	Fix access, nothing is lost
Need to roll back fast	`git revert` the deploy commit	Argo CD syncs the previous digest

Useful commands while a rollout is in flight:

bash

kubectl argo rollouts get rollout shop-api -n shop --watch
kubectl argo rollouts promote shop-api -n shop      # skip a pause
kubectl argo rollouts abort shop-api -n shop
argocd app get shop-api-production
argocd app history shop-api-production

A few opinions

Use staging as a real gate. Auto-sync staging on every merge, and promote to production through a second commit after checks. The point of a canary is to reduce blast radius, not to be your only test.
Keep secrets out of git. Use External Secrets Operator or Sealed Secrets. A GitOps repo full of base64 is a leak waiting for a screenshot.
Do not template everything. Kustomize overlays are enough for most teams. Reach for Helm when you consume third-party charts, not for your own services by default.
Alert on sync state. An Application stuck OutOfSync or Degraded for an hour is a problem someone should hear about.

Wrap-up

GitOps gives you a trustworthy answer to "what is running", and Gateway API with Argo Rollouts gives you a standard, vendor-neutral way to roll out gradually and back out automatically. Start with one service in staging, get the ignoreDifferences and analysis queries right there, then repeat for the rest through the ApplicationSet.

Kubernetes GitOps with Argo CD and Gateway API: Progressive Delivery for CI/CD

The shape of the system

Repository layout

Gateway API in two minutes

The Rollout

Automated analysis

Argo CD: one Application per environment

The CI side: a commit, not a kubectl

When it goes wrong

A few opinions

Wrap-up

Resources