Copy the command definition below into:
~/.claude/commands/k8s-troubleshoot.md---
name: k8s-troubleshoot
description: Debug Kubernetes pod failures and issues
shortcut: kt
category: devops
difficulty: advanced
estimated_time: 2 minutes
---
<!-- markdownlint-disable MD024 -->
<!-- DESIGN DECISION: Streamlines K8s debugging -->
<!-- Kubernetes failures are cryptic (CrashLoopBackOff, ImagePullBackOff, OOMKilled).
Developers waste hours running kubectl describe/logs/events manually.
This command provides systematic debugging workflow with specific fix recommendations. -->
<!-- VALIDATION: Handles common failures -->
<!-- CrashLoopBackOff (app crashes) -->
<!-- ImagePullBackOff (registry issues) -->
<!-- Pending (resource constraints) -->
<!-- OOMKilled (memory limits) -->
# Kubernetes Troubleshooter
Systematically debugs Kubernetes pod failures (CrashLoopBackOff, ImagePullBackOff, Pending, OOMKilled) with root cause analysis and specific fixes.
## When to Use This
- Pod stuck in CrashLoopBackOff
- Pod stuck in ImagePullBackOff
- Pod stuck in Pending
- Pod terminated with OOMKilled
- Service not accessible
- Ingress not routing traffic
- Cluster-level issues (use cluster admin tools)
## How It Works
You are a Kubernetes troubleshooting expert. When user runs `/k8s-troubleshoot` or `/kt`:
1. **Identify the issue:**
Ask user:
- Pod name or deployment?
- Namespace?
- What symptom? (CrashLoopBackOff, Pending, etc.)
2. **Gather diagnostic data:**
```bash
kubectl get pod <pod> -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp
```
3. **Analyze root cause:**
- **CrashLoopBackOff** → App crashes on startup
- **ImagePullBackOff** → Can't pull container image
- **Pending** → Can't schedule (resources, node selector)
- **OOMKilled** → Out of memory
- **Error** → Misconfigured manifest
- **Evicted** → Resource pressure on node
4. **Provide specific fix:**
- Show exact error from logs/events
- Explain root cause
- Provide fix (code, manifest, or command)
- Include verification steps
## Output Format
```markdown
## Diagnosis
**Pod:** [pod-name]
**Status:** [Current status]
**Root Cause:** [Specific issue]
## Evidence
```
[Relevant logs/events/describe output]
```
## Fix
[Step-by-step fix with commands/manifest changes]
## Verification
```bash
[Commands to verify fix worked]
```
```
## Examples
### Example 1: CrashLoopBackOff
**User:** `/kt`
- Pod: `webapp-7d6f9c8b4-x9k2l`
- Namespace: `production`
- Symptom: CrashLoopBackOff
**Troubleshooting workflow:**
```bash
# Check pod status
kubectl get pod webapp-7d6f9c8b4-x9k2l -n production
# Output: CrashLoopBackOff
# Get detailed info
kubectl describe pod webapp-7d6f9c8b4-x9k2l -n production
# Last State: Terminated, Exit Code: 1
# Check current logs (if pod is running)
kubectl logs webapp-7d6f9c8b4-x9k2l -n production
# Error: Cannot connect to database
# Check previous crash logs
kubectl logs webapp-7d6f9c8b4-x9k2l -n production --previous
# Error: getaddrinfo ENOTFOUND postgres
# Error: Failed to connect to database at postgres:5432
```
**Output:**
```markdown
## Diagnosis
**Pod:** webapp-7d6f9c8b4-x9k2l
**Namespace:** production
**Status:** CrashLoopBackOff (Exit Code 1)
**Root Cause:** Application can't connect to database - hostname "postgres" not found
## Evidence
```
Error from container logs:
Error: getaddrinfo ENOTFOUND postgres
Error: Failed to connect to database at postgres:5432
at TCPConnectWrap.afterConnect [as oncomplete]
```
## Root Cause Analysis
The application is trying to connect to database hostname "postgres" but:
1. No service named "postgres" exists in the namespace
2. OR the DATABASE_URL environment variable is misconfigured
3. OR the database service is in a different namespace
## Fix
### Option 1: Database service doesn't exist
Check if database service exists:
```bash
kubectl get svc -n production | grep postgres
```
If missing, deploy database or create service pointing to external database.
### Option 2: Wrong hostname in environment variable
Check current DATABASE_URL:
```bash
kubectl get deploy webapp -n production -o yaml | grep DATABASE_URL
```
Fix the environment variable in deployment:
```yaml
env:
- name: DATABASE_URL
value: "postgresql://user:pass@postgres-service:5432/dbname"
# NOT: postgresql://user:pass@postgres:5432/dbname
```
Apply fix:
```bash
kubectl set env deployment/webapp \
DATABASE_URL="postgresql://user:pass@postgres-service:5432/dbname" \
-n production
```
### Option 3: Database in different namespace
If database is in namespace "database", use FQDN:
```bash
kubectl set env deployment/webapp \
DATABASE_URL="postgresql://user:pass@postgres.database.svc.cluster.local:5432/dbname" \
-n production
```
## Verification
```bash
# Watch pod restart with fix
kubectl get pods -n production -w
# Check new pod logs
kubectl logs -f deployment/webapp -n production
# Verify app is healthy
kubectl get pods -n production
# Should show: Running (not CrashLoopBackOff)
```
```
### Example 2: ImagePullBackOff
**User:** `/kt`
- Pod: `api-deployment-5f6d8c9b7-h4k8m`
- Namespace: `staging`
- Symptom: ImagePullBackOff
**Output:**
```markdown
## Diagnosis
**Pod:** api-deployment-5f6d8c9b7-h4k8m
**Namespace:** staging
**Status:** ImagePullBackOff
**Root Cause:** Cannot pull container image from registry
## Evidence
```
Events:
Warning Failed 3m (x4 over 5m) kubelet Failed to pull image "myregistry.io/api:v2.0.0": rpc error: code = Unknown desc = Error response from daemon: pull access denied for myregistry.io/api, repository does not exist or may require 'docker login'
Warning Failed 3m (x4 over 5m) kubelet Error: ErrImagePull
Normal BackOff 2m (x6 over 5m) kubelet Back-off pulling image "myregistry.io/api:v2.0.0"
Warning Failed 2m (x6 over 5m) kubelet Error: ImagePullBackOff
```
## Root Cause Analysis
Kubernetes cannot pull the image because:
1. Image doesn't exist (wrong tag, deleted image)
2. Registry requires authentication (missing imagePullSecrets)
3. Network issue (can't reach registry)
## Fix
### Check 1: Verify image exists
```bash
# Try pulling image manually
docker pull myregistry.io/api:v2.0.0
# If it fails with authentication error, need imagePullSecret
```
### Check 2: Create registry secret (if private registry)
```bash
# Create Docker registry secret
kubectl create secret docker-registry regcred \
--docker-server=myregistry.io \
--docker-username=your-username \
--docker-password=your-password \
--docker-email=[email protected] \
-n staging
```
### Check 3: Add imagePullSecret to deployment
```yaml
spec:
template:
spec:
imagePullSecrets:
- name: regcred # Add this!
containers:
- name: api
image: myregistry.io/api:v2.0.0
```
Apply fix:
```bash
kubectl patch deployment api-deployment -n staging -p '
spec:
template:
spec:
imagePullSecrets:
- name: regcred
'
```
### Check 4: Verify image tag exists
```bash
# List available tags
curl -u username:password https://myregistry.io/v2/api/tags/list
# If v2.0.0 doesn't exist, use correct tag
kubectl set image deployment/api-deployment api=myregistry.io/api:v1.5.0 -n staging
```
## Verification
```bash
# Watch pod creation
kubectl get pods -n staging -w
# Check events (should show "Pulled" successfully)
kubectl describe pod <new-pod-name> -n staging
# Verify pod is running
kubectl get pods -n staging
```
```
### Example 3: Pod Pending (Resource Constraints)
**User:** `/kt`
- Pod: `db-0`
- Namespace: `database`
- Symptom: Pending for 5 minutes
**Output:**
```markdown
## Diagnosis
**Pod:** db-0
**Namespace:** database
**Status:** Pending
**Root Cause:** Insufficient CPU resources on nodes
## Evidence
```
Events:
Warning FailedScheduling 2m (x10 over 5m) default-scheduler 0/3 nodes are available: 3 Insufficient cpu.
Pod resource requests:
Requests:
cpu: 4000m
memory: 8Gi
```
## Root Cause Analysis
Pod cannot be scheduled because:
- Pod requests 4 CPUs (4000m)
- No node has 4 CPUs available
- All 3 nodes are at capacity
## Fix
### Option 1: Reduce resource requests (if over-provisioned)
Check actual usage of similar pods:
```bash
kubectl top pods -n database
```
If actual usage is much lower (e.g., 1 CPU), reduce requests:
```yaml
resources:
requests:
cpu: 2000m # Reduced from 4000m
memory: 4Gi # Reduced from 8Gi
limits:
cpu: 4000m
memory: 8Gi
```
### Option 2: Add more nodes to cluster
```bash
# For managed Kubernetes (GKE example)
gcloud container clusters resize my-cluster --num-nodes=5
# For EKS, update node group size in AWS Console or CLI
```
### Option 3: Scale down other workloads
```bash
# Identify resource-hungry pods
kubectl top pods --all-namespaces --sort-by=cpu
# Scale down non-critical workloads
kubectl scale deployment non-critical-app --replicas=1 -n other-namespace
```
## Verification
```bash
# Watch pod scheduling
kubectl get pods -n database -w
# Verify pod is running
kubectl get pod db-0 -n database
# Check node resources
kubectl top nodes
```
```
### Example 4: OOMKilled (Out of Memory)
**User:** `/kt`
- Pod: `worker-6d8f7c9b4-p2m5n`
- Namespace: `jobs`
- Symptom: Pod keeps restarting, status shows OOMKilled
**Output:**
```markdown
## Diagnosis
**Pod:** worker-6d8f7c9b4-p2m5n
**Namespace:** jobs
**Status:** Terminated (OOMKilled)
**Root Cause:** Container exceeded memory limit
## Evidence
```
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Mon, 01 Jan 2024 10:00:00 +0000
Finished: Mon, 01 Jan 2024 10:05:23 +0000
Current memory limit: 512Mi
```
## Root Cause Analysis
Pod was killed because:
- Container used more than 512Mi memory
- Kubernetes OOMKiller terminated it (Exit Code 137)
- Likely: Memory leak, or limit too low for workload
## Fix
### Option 1: Increase memory limit (if workload legitimately needs more)
```yaml
resources:
requests:
memory: 1Gi # Increased from 512Mi
limits:
memory: 2Gi # Increased from 512Mi
```
Apply:
```bash
kubectl set resources deployment worker \
--requests=memory=1Gi \
--limits=memory=2Gi \
-n jobs
```
### Option 2: Fix memory leak (if app has leak)
Check memory growth over time:
```bash
kubectl top pod worker-6d8f7c9b4-p2m5n -n jobs --containers
```
If memory constantly grows, investigate app code for leaks.
### Option 3: Add resource monitoring
Add Prometheus metrics to track memory:
```yaml
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
```
## Verification
```bash
# Watch memory usage
kubectl top pod -n jobs -l app=worker --containers
# Check pod no longer OOMKilled
kubectl get pods -n jobs
# Should show: Running (not CrashLoopBackOff)
# Monitor over time
watch kubectl top pod -n jobs
```
```
## Pro Tips
**Use kubectl describe pod first (shows events)**
**Check --previous logs for crash analysis**
**Look at Events sorted by time: --sort-by=.metadata.creationTimestamp**
**CrashLoopBackOff = app issue, ImagePullBackOff = registry issue, Pending = resources**
**Exit Code 137 = OOMKilled, Exit Code 1 = app error**
## Common Exit Codes
- **0**: Success
- **1**: Application error
- **137**: OOMKilled (128 + 9)
- **139**: Segmentation fault
- **143**: SIGTERM (graceful shutdown)