Leader Election in Kubernetes Controllers

You run two replicas of your controller for availability. Without leader election, both try to reconcile the same resources at the same time.

Estimated Reading Time : 6m

The problem

Controllers modify cluster state — creating Deployments, updating status, managing external resources. If two instances reconcile the same resource simultaneously, you get race conditions: duplicate resources, conflicting updates, and unpredictable behavior.

Leader election ensures that only one instance is actively reconciling at any time. The others stand by and take over if the leader fails.

Enabling leader election

controller-runtime handles leader election with a single option:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    LeaderElection:          true,
    LeaderElectionID:        "backup-controller-leader",
    LeaderElectionNamespace: "backup-system",
})

That’s it. controller-runtime acquires a lease, and only the instance holding the lease runs the reconcilers. The others block on mgr.Start() until they become the leader.

How it works

controller-runtime uses a Kubernetes Lease object (or ConfigMap in older versions) as a distributed lock:

1. On startup, each instance tries to acquire the Lease named backup-controller-leader
2. One instance wins and becomes the leader
3. The leader periodically renews the lease (default: every 10 seconds)
4. Non-leaders watch the lease and attempt to acquire it if the leader stops renewing
5. When the leader crashes or shuts down, another instance acquires the lease within seconds

kubectl get leases -n backup-system

NAME                        HOLDER                     AGE
backup-controller-leader    controller-pod-abc-123     5m

Configuration

The default timings work for most cases, but you can tune them:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    LeaderElection:          true,
    LeaderElectionID:        "backup-controller-leader",
    LeaderElectionNamespace: "backup-system",
    LeaseDuration:           15 * time.Second,  // how long the lease is valid
    RenewDeadline:           10 * time.Second,  // how long the leader has to renew
    RetryPeriod:              2 * time.Second,  // how often non-leaders check
})

LeaseDuration — how long the lease is valid without renewal. After this period, other instances consider the leader dead.

RenewDeadline — how long the leader has to renew before it voluntarily steps down. Must be less than LeaseDuration.

RetryPeriod — how often non-leaders attempt to acquire the lease. Must be less than RenewDeadline.

Shorter durations mean faster failover but more API server traffic.

RBAC

The controller’s service account needs permission to manage Leases:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: backup-controller-leader-election
  namespace: backup-system
rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "create", "update"]

Without this, leader election fails silently and the controller never starts reconciling.

Graceful shutdown

When a leader receives a shutdown signal (SIGTERM), controller-runtime releases the lease before exiting. This lets another instance take over immediately instead of waiting for the lease to expire.

// ctrl.SetupSignalHandler() handles SIGTERM/SIGINT
if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
    log.Error(err, "manager exited")
}

In Kubernetes, this happens naturally during pod rollouts — the old pod receives SIGTERM, releases the lease, and the new pod acquires it.

Gotchas

Leader election ID must be unique per controller. If you run multiple different controllers in the same namespace, each needs its own LeaderElectionID. Two controllers sharing the same ID will fight for the same lease.

The leader can’t do anything if it can’t reach the API server. If the leader loses network connectivity, it can’t renew the lease. It steps down after RenewDeadline, and another instance takes over. When connectivity returns, the old leader sees it’s no longer the leader and stops reconciling.

Don’t run leader-elected controllers with replicas: 1. The point is high availability. With one replica, a pod restart means downtime while the new pod starts and acquires the lease. Run at least 2 replicas.

Development mode. When running locally against a dev cluster, you can disable leader election to simplify debugging:

leaderElect := os.Getenv("ENABLE_LEADER_ELECTION") == "true"

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    LeaderElection:   leaderElect,
    LeaderElectionID: "backup-controller-leader",
})