GitOps: The Modern Way to Manage Kubernetes Deployments

Who this is for

You deploy to Kubernetes by running kubectl apply from CI, and you cannot reliably answer what is running in the cluster right now.

What actually changes

Kubernetes already reconciles: you declare three replicas, a controller keeps three alive. GitOps moves the declaration out of whoever last ran a command and into a repository, then puts a controller in the cluster that reconciles against it. See Why Kubernetes uses desired state for the underlying idea.

Three things follow:

The cluster pulls rather than CI pushing. Your pipeline no longer needs cluster credentials — it writes a commit. That takes cluster credentials out of the pipeline entirely.

Git is the audit log. "Who changed this and when" is git log, with a review attached.

Drift is corrected, not just detected — once you turn it on. Argo CD only reports drift until selfHeal is set; with it, anyone who edits a live resource by hand watches it revert. That is the feature, and it surprises people.

ArgoCD, concretely

kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Then an Application tells it which repository to watch and where to apply it:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/example/infra-config.git
    targetRevision: main
    path: apps/web/overlays/production

  destination:
    server: https://kubernetes.default.svc
    namespace: web

  syncPolicy:
    automated:
      prune: true      # delete what was removed from Git
      selfHeal: true   # revert changes made in the cluster
    syncOptions:
      - CreateNamespace=true

Those two booleans are the whole behaviour, and both deserve a decision rather than a copy-paste.

prune: true means deleting a manifest from the repository deletes the resource. Without it, removals accumulate as resources nobody owns. With it, a bad merge deletes production.

selfHeal: true means a manual kubectl scale is undone within minutes. That is correct, and it is also the thing to disable deliberately during an incident if you need to hold a manual change — argocd app set web --self-heal=false, and remember to put it back.

Repository layout

Keep application code and cluster configuration in separate repositories. They have different review requirements, different access, and different change rates — and a config repo that triggers on every application commit is a config repo nobody can read the history of.

infra-config/
  apps/
    web/
      base/                 # Deployment, Service, Ingress
      overlays/
        staging/            # kustomize patches: replicas, hostnames
        production/

Promotion is then a pull request that changes one image tag in one overlay, which is a diff a human can actually review.

The moving-tag problem

This is the mistake that quietly defeats GitOps.

image: registry.example.com/web:latest     # the manifest never changes
image: registry.example.com/web:1.8.3      # a commit per release
image: registry.example.com/web@sha256:9f2a…   # exactly one artefact, forever

With :latest, pushing a new image changes nothing in Git, so the controller sees no drift and does nothing — and when a pod does restart for an unrelated reason it quietly picks up different code. Git no longer describes the cluster, which was the entire point. Worse, the rollback you reach for does not work, because the previous commit points at the same moving tag.

Pin to a tag at minimum and a digest where you can. The reasoning is the same as Why artifacts should be immutable.

If updating the tag by hand is the friction, Argo CD Image Updater can watch a registry and, with its git write-back method configured, write the commit for you — the commit still happens, which keeps the model intact. Its default method edits the Application in the cluster instead, which does not.

Secrets

Plain Secret manifests cannot go in Git; they are base64, not encryption. Three workable approaches:

• Sealed Secrets — encrypt with a cluster-held public key; only that cluster can decrypt. The ciphertext is safe to commit.
• External Secrets Operator — keep the value in a real secret manager and commit only a reference to it.
• SOPS — encrypt specific YAML values, with keys from a KMS.

All three preserve the property that the repository is complete. Picking none of them and applying secrets by hand reintroduces exactly the drift you adopted GitOps to remove.

Common mistakes

1. :latest, or any moving tag. See above. It is the one that makes the whole thing decorative.
2. prune: true with everything in one Application. One bad merge, one large deletion. Split by application so blast radius matches review.
3. Fighting self-heal. Repeatedly re-applying a manual change that keeps reverting means the repository is wrong. Fix it there.
4. Application code and config in one repository. Every app commit triggers a sync, and the config history becomes unreadable.
5. No health checks on the resources. Argo reports Synced — manifests applied — which is not Healthy. A Deployment can be synced and failing to pull its image.

If this did not work

• Synced but nothing changed. Check the image reference. A moving tag plus an unchanged manifest is the usual answer.
• Stuck Progressing, then degraded. The manifests applied and the workload did not start. Describe the pod — a bad image reference produces ImagePullBackOff, which Argo will report as unhealthy without explaining it.
• OutOfSync on a field nobody set. A mutating webhook or a controller is adding it. Use ignoreDifferences for that specific path rather than turning self-heal off.
• It reverted my fix during an incident. It will. Disable self-heal on that Application explicitly, and commit the fix.
• Not sure what a manifest renders to. Run it through the manifest analyzer before committing.

Alternatives

Flux is the other mature choice and encodes the same ideas with a different surface — more CRDs, no bundled UI, and a tighter fit with Kustomize and Helm as first-class sources. If a bundled dashboard matters to you, that points to Argo.

Next steps

Put one non-critical application under a GitOps Application with selfHeal: true and prune: false. Live with the self-healing for a week before turning pruning on — that order surfaces the drift you did not know you had, without the deletion risk.

Related on this site

• Why Kubernetes uses desired state — GitOps is this idea with Git as the store.
• Why artifacts should be immutable — Why a moving tag breaks reconciliation quietly.
• ImagePullBackOff — The failure a bad image reference produces after Argo syncs it.
• Manifest analyzer — Check what you are about to commit.