Troubleshooting Kubernetes Webhook Timeout: No Endpoints Available for AWS LB Controller and External Secrets during ArgoCD Sync

Symptoms

During the ArgoCD sync process, a webhook timeout occurs, indicating that there are no available endpoints for the AWS Load Balancer Controller and External Secrets.

Environment

Kubernetes cluster running on AWS with ArgoCD, AWS Load Balancer Controller, and External Secrets configured.

Most Likely Causes

The webhook timeout typically occurs when the AWS Load Balancer Controller or External Secrets is unable to communicate with the Kubernetes API server due to misconfiguration, network issues, or unavailability of the service endpoints.

What to Check First

1. Confirm kubeconfig context, namespace, cluster version, and whether the workload is managed by Helm, GitOps, or another controller.
2. Inspect events, pod status, endpoints, and admission or webhook failures before changing manifests.
3. Check whether the problem is control plane admission, scheduling, networking, storage, image pull, or application readiness.

Fix Steps

1. Check the status of the AWS Load Balancer Controller

   Verify that the AWS Load Balancer Controller is running and has the necessary permissions to interact with the Kubernetes API.

   Safe to run: read-only

   kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller
   kubectl describe deployment aws-load-balancer-controller -n kube-system

2. Inspect the External Secrets configuration

   Ensure that the External Secrets are correctly configured and that the service is running without errors.

   Changes system state: review before running

   kubectl get externalsecrets -n <namespace>
   kubectl describe externalsecret <externalsecret-name> -n <namespace>

3. Check the webhook configuration

   Review the webhook configuration for both the AWS Load Balancer Controller and External Secrets to ensure they are correctly set up.

   Safe to run: read-only

   kubectl get validatingwebhookconfiguration
   kubectl describe validatingwebhookconfiguration <webhook-name>

4. Verify network connectivity

   Ensure that there are no network policies or security groups blocking traffic between the AWS Load Balancer Controller, External Secrets, and the Kubernetes API server.

   Safe to run: read-only

   kubectl get networkpolicies -n kube-system
   aws ec2 describe-security-groups --group-ids <your-security-group-id>

5. Check service endpoints

   Confirm that the services for the AWS Load Balancer Controller and External Secrets have available endpoints.

   Changes system state: review before running

   kubectl get svc -n kube-system
   kubectl get endpoints <service-name> -n kube-system

6. Review ArgoCD application logs

   Check the logs of the ArgoCD application to identify any specific errors related to the webhook timeout.

   Safe to run: read-only

   kubectl logs deployment/argocd-server -n argocd
   kubectl logs deployment/argocd-application-controller -n argocd

Validation

• kubectl get events no longer shows the same failure after the affected object is reconciled.
• The workload reaches the expected ready state and endpoints are populated for the service.
• A rollout restart or controller reconcile completes without reintroducing the failure.

Logs to Check

• kubectl describe output for the affected object.
• Pod, controller, admission webhook, and ingress controller logs.
• Cluster events and audit logs when authorization or admission is involved.

Rollback and Escalation

• Use Helm rollback, Git revert, or the previous manifest revision if the change breaks reconciliation.
• Avoid deleting persistent volumes or secrets during diagnosis unless a backup and restore path is confirmed.
• Remove temporary debug pods, broad RBAC grants, or network policy exceptions after validation.

Escalate When

• Escalate if the same error persists after rollback and a clean retry from the original failing path.
• Escalate if logs show authorization, data loss, certificate, replication, or production availability risk outside the local service owner scope.

Edge Cases

• If the AWS Load Balancer Controller is not installed, follow the installation guide in the AWS documentation.
• If the External Secrets controller is not responding, consider restarting the pod or checking for resource limits.

Notes from the Field

• When a controller owns the object, direct kubectl edits can be overwritten. Find the source of truth before patching.
• Admission webhook and endpoint timeouts often look like application failures but are usually control plane reachability problems.