arista-device-health

---
name: arista-device-health
description: >-
  Arista EOS device health check and triage procedure. Use when
  troubleshooting Arista 7000, 7500, or 720X series switches — assessing
  CPU, memory, interfaces, environment, and agent/daemon health. Covers
  MLAG state validation and VXLAN/EVPN health as data center extension
  steps. EOS is Linux-native — standard Linux diagnostics (bash top,
  df, dmesg) are valid troubleshooting tools alongside EOS show commands.
  Includes agent health monitoring via show agent for EOS-specific
  daemon failure detection.
license: Apache-2.0
metadata:
  safety: read-only
  author: network-security-skills-suite
  version: "1.0.0"
  openclaw: '{"emoji":"🔍","safetyTier":"read-only","requires":{"bins":["ssh"],"env":[]},"tags":["arista","eos","health"],"mcpDependencies":[],"egressEndpoints":[]}'
---

# Arista EOS Device Health Check

Structured triage procedure for assessing Arista EOS device health. Produces a
prioritized findings report with severity classifications and recommended actions.

EOS runs on a Linux kernel with individual processes (agents) managing each
protocol and subsystem. This architecture means Linux-native tools (`bash top`,
`bash df`, `bash dmesg`) are valid supplements to EOS show commands. Agent
health is a first-class concern — a crashed or stuck agent can silently affect
a subsystem even when aggregate device metrics look normal.

The procedure covers core device health first, then extends to MLAG and
VXLAN/EVPN for data center deployments.

## When to Use

- Device reported as slow, dropping traffic, or unresponsive
- Scheduled health audit of Arista switches (leaf, spine, or border)
- Post-change verification after upgrades, patches, or configuration changes
- Capacity planning data collection for CPU, memory, and link utilization
- Incident response when an Arista switch is suspected as the fault domain
- MLAG inconsistency or split-brain detection in a data center pair
- VXLAN/EVPN overlay health assessment — VTEP reachability, BGP EVPN peering
- Agent crash or restart detected in syslog

## Prerequisites

- SSH or console access to the device (privilege 1 minimum)
- EOS 4.28 or later (commands validated against EOS 4.30+)
- Network reachability to management interface confirmed
- Awareness of the device's normal baseline (CPU, memory, traffic patterns)
- For MLAG configurations: access to both MLAG peers for cross-validation
- For VXLAN/EVPN: knowledge of expected VTEP count and BGP EVPN peer list

## Procedure

Follow this sequence. Core device health (Steps 1–5) applies to all deployments.
Steps 6–7 are data center extensions for MLAG and VXLAN/EVPN — skip if not
configured on this device.

### Step 1: Establish Baseline Context

```
show version
show uptime
show clock
show inventory
```

Record: hostname, EOS version, uptime, hardware model, total memory, current time.
Short uptime indicates a recent reload or crash — check `show reload cause` for
the trigger. Note the hardware platform — Arista's 7050X, 7280R, 7500R, and
720X series have different memory and CPU profiles.

### Step 2: CPU and Memory Assessment

EOS reports CPU and memory through both EOS commands and Linux-native tools.

```
show processes top once
show version | include Free memory
```

**Linux-native alternative** (provides more granularity):
```
bash timeout 5 top -bn1 | head 20
bash free -m
```

`show processes top once` shows per-process CPU and memory similar to Linux `top`.
EOS CPU is typically consumed by agents — each protocol has its own process.

Key agents to watch:
- **Stp** — Spanning Tree: high CPU indicates topology instability
- **Rib** — Routing Information Base: route churn or large table operations
- **Ebra** — EOS Bridge Agent: L2 forwarding, MAC learning storms
- **IpRib** — IP RIB agent: routing table processing
- **Bgp** — BGP agent: peer negotiations, route processing
- **Fap-sobek** / **Sand** — ASIC agents: forwarding plane, hardware programming

EOS memory: `show version` includes total and free memory. Free memory below
20% of total is warning. Linux `free -m` provides buffer/cache detail — EOS
uses Linux page cache aggressively, so "used" memory includes reclaimable cache.

### Step 3: Agent and Daemon Health

EOS runs each subsystem as an independent agent. A crashed or stuck agent
affects its subsystem silently — device-level metrics may look normal while
one protocol is completely inoperative.

```
show agent
show agent [name] logs | tail 20
show logging last 50 | include AGENT|agent|crash|restart
```

`show agent` lists all EOS agents with their PID, state, and uptime.

Healthy state: all agents show `running` with uptimes matching device uptime.
Red flags:
- Agent with shorter uptime than device → it crashed and restarted
- Agent in `crashed` or `not running` state → subsystem is down
- Multiple agent restarts in logs → recurring instability

If an agent shows recent restart, check its logs: `show agent [name] logs`.
Common restart causes: memory exhaustion, uncaught exception, watchdog timeout.

### Step 4: Interface Health

```
show interfaces status
show interfaces counters errors
show interfaces counters discards
```

For interfaces with errors:
```
show interfaces [name]
show interfaces [name] transceiver
```

Error classification:
- CRC / FCS errors → Layer 1 (cabling, optics, SFP seating)
- Input errors without CRC → buffer overruns, MTU mismatch
- Alignment errors → duplex mismatch or L1 issue
- Output discards → egress congestion; check QoS or link capacity
- Late collisions → duplex mismatch (should not occur on modern links)

Optics DOM: `show interfaces transceiver` provides Tx/Rx power, temperature,
voltage. Low Rx power (below -10 dBm for most 10G SFP+) indicates fiber or
optic degradation.

### Step 5: Environment and Platform

```
show environment all
show environment cooling
show environment power
show environment temperature
show reload cause
show logging last 30 | include ERR|WARN|traceback|panic
```

Check: all temperature sensors within limits, all power supplies OK, all
fans operational. `show environment all` is a comprehensive single command.

`show reload cause` — if the device recently reloaded, this explains why.
Expected causes: user-initiated, software upgrade. Unexpected causes:
kernel panic, watchdog timeout, power loss.

Review syslog for tracebacks, panics, or agent crashes. EOS logs are also
accessible via `bash cat /var/log/messages | tail 50`.

### Step 6: MLAG Health (Data Center Extension)

Skip this step if MLAG is not configured. MLAG state is often the single most
important health indicator in an Arista data center deployment — a degraded
MLAG pair can cause traffic black-holes or asymmetric forwarding.

```
show mlag
show mlag detail
show mlag interfaces
show mlag config-sanity
```

**MLAG state assessment:**

| Field | Healthy Value | Problem Indicator |
|-------|--------------|-------------------|
| state | active | disabled, inactive |
| negotiation status | connected | connecting, disconnected |
| peer-link status | up | down, errdisabled |
| local-interface status | up | down |
| config-sanity | consistent | inconsistent |

`show mlag detail` provides:
- **Peer address** and **peer-link** interface status
- **Heartbeat** interval and status — lost heartbeats indicate control plane
  reachability issues between peers
- **Reload delay** timers — critical for preventing traffic loss during boot

`show mlag config-sanity` checks for configuration mismatches between peers.
Inconsistencies can cause traffic loops or black-holes. Address any `inconsistent`
result before continuing other triage.

`show mlag interfaces` — verify all MLAG interfaces show `active-full` on both
sides. `active-partial` means one side is down — reduced redundancy.
`disabled` means the MLAG interface is administratively or operationally down.

### Step 7: VXLAN/EVPN Health (Data Center Extension)

Skip this step if VXLAN/EVPN is not configured. This assesses overlay fabric
health for VXLAN deployments using BGP EVPN as the control plane.

```
show vxlan vtep
show vxlan address-table
show interfaces vxlan 1
show bgp evpn summary
```

**VXLAN health checks:**
- `show vxlan vtep` — verify expected VTEP count. Missing VTEPs indicate
  underlay reachability or BGP EVPN peering issues.
- `show interfaces vxlan 1` — VTEP source interface must be up. Flood list
  should match expected remote VTEPs (for flood-and-learn) or be empty
  (for BGP EVPN with ingress replication).
- `show vxlan address-table` — MAC learning across the overlay. Stale or
  missing entries indicate control plane issues.

**BGP EVPN peering:**
- `show bgp evpn summary` — all EVPN peers must be in Established state.
  Peers in Active/Connect state have transport or configuration issues.
- Compare EVPN route count to baseline — significant deviation indicates
  route withdrawal or redistribution problems.

For L3 VXLAN (symmetric IRB):
```
show ip route vrf [name] summary
show bgp evpn route-type ip-prefix
```

Verify: VRF route counts match expectations, no unexpected route withdrawals,
EVPN type-5 routes present for inter-VRF routing.

## Threshold Tables

Reference: `references/threshold-tables.md` for detailed per-parameter thresholds.

| Parameter | Normal | Warning | Critical | Notes |
|-----------|--------|---------|----------|-------|
| CPU 5-min avg | < 40% | 40–70% | > 70% | Per-agent breakdown in `show processes top` |
| Memory free | > 30% | 20–30% | < 20% | Linux page cache is reclaimable |
| Agent state | All running | Agent restarted | Agent crashed/not running | Check `show agent` |
| Interface error rate | < 0.01% | 0.01–0.1% | > 0.1% | |
| Output discards/hr | < 100 | 100–1000 | > 1000 | |
| MLAG state | active/connected | config inconsistency | peer-link down | |
| VXLAN VTEP count | Matches expected | Missing VTEPs | Vxlan1 interface down | |
| Temperature | Within spec | 5°C of max | At/above max | Per-sensor |

## Decision Trees

### Primary Triage

```
Is the device reachable?
├── No → Check console, power, environment. Check reload cause after recovery.
└── Yes
    ├── Agent health issue?
    │   ├── Agent crashed / not running → Subsystem is down
    │   │   ├── Identify which agent → Determines affected protocol/feature
    │   │   ├── Bgp agent → BGP sessions will be down
    │   │   ├── Stp agent → STP not running, L2 loops possible
    │   │   ├── Ebra agent → L2 forwarding affected
    │   │   └── Action: check agent logs, attempt restart, escalate to TAC
    │   ├── Agent restarted recently → Subsystem had a disruption
    │   │   └── Check logs: `show agent [name] logs` for crash reason
    │   └── All agents running, uptimes match → Agent layer healthy
    │
    ├── CPU issue?
    │   ├── Identify top agent by CPU
    │   │   ├── Stp high → STP topology change, check link flaps
    │   │   ├── Rib/IpRib high → Route churn, check BGP/OSPF peers
    │   │   ├── Ebra high → MAC learning storm, check for loops
    │   │   ├── Bgp high → Large table, peer negotiation, route policy
    │   │   └── Fap/Sand high → ASIC programming backlog
    │   └── No single agent dominant → General overload, check traffic rates
    │
    ├── Memory issue?
    │   ├── Free < 20% → Check top consumers via `show processes top once`
    │   ├── Linux cache: `bash free -m` → Cache is reclaimable, not a real leak
    │   └── Steady growth → Memory leak in agent, collect data, plan reload
    │
    ├── Interface errors? → Classify error type
    │   ├── CRC/FCS errors → Layer 1 (cable, optic, SFP)
    │   ├── Output discards → QoS or congestion
    │   └── Alignment errors → Duplex mismatch
    │
    ├── MLAG issue? (if configured)
    │   ├── See MLAG State Triage tree below
    │   └── MLAG healthy → Continue
    │
    ├── VXLAN/EVPN issue? (if configured)
    │   ├── Missing VTEPs → Check underlay reachability, BGP EVPN peers
    │   ├── EVPN peer not Established → Check transport, route-map, ASN
    │   └── VTEP and peers OK → Overlay healthy
    │
    └── All within thresholds → Document clean health
```

### MLAG State Triage

```
MLAG configured?
├── No → Skip MLAG checks
└── Yes
    ├── MLAG state: active?
    │   ├── No → MLAG disabled or inactive
    │   │   └── Check: `show mlag` for reason, peer reachability
    │   └── Yes → Continue
    │
    ├── Peer-link status: up?
    │   ├── No → CRITICAL — peer-link down
    │   │   ├── Traffic orphaned on MLAG interfaces
    │   │   ├── Check: physical link, SFP, port-channel members
    │   │   └── Verify: reload-delay timers configured to prevent storms
    │   └── Yes → Continue
    │
    ├── Negotiation: connected?
    │   ├── No → Control plane issue between peers
    │   │   └── Check: heartbeat, peer IP, VLAN configuration
    │   └── Yes → Continue
    │
    ├── Config sanity: consistent?
    │   ├── No → Configuration mismatch
    │   │   ├── `show mlag config-sanity` for details
    │   │   ├── Common: VLAN mismatch, STP priority, trunk allowed VLANs
    │   │   └── Fix mismatches before trusting MLAG state
    │   └── Yes → Continue
    │
    ├── MLAG interfaces: all active-full?
    │   ├── active-partial → One side down, reduced redundancy
    │   │   └── Check: interface status on both peers
    │   ├── disabled → Interface operationally down
    │   │   └── Check: physical link, member port status
    │   └── active-full → Healthy
    │
    └── All checks pass → MLAG healthy
```

### Escalation Criteria

Escalate to senior engineer or Arista TAC when:
- CPU sustained > 90% for 15+ minutes with no identifiable cause
- Memory below 15% free with no recent change to explain consumption
- Agent in crashed state that does not recover after restart
- Multiple agents crashing or restarting within a short period
- Traceback or kernel panic in logs (`show logging`, `bash dmesg`)
- Any environmental alarm (power, fan, temperature)
- More than 3 routing neighbor state changes in the last hour
- MLAG peer-link failure with no clear physical cause
- MLAG config-sanity inconsistencies that cannot be resolved
- VXLAN overlay losing more than 10% of expected VTEPs

## Report Template

```
DEVICE HEALTH REPORT
====================
Device: [hostname]
Platform: Arista EOS
Model: [from show inventory]
Software: [EOS version]
Uptime: [uptime string]
Check Time: [timestamp]
Performed By: [operator/agent]

SUMMARY: [HEALTHY | WARNING | CRITICAL | EMERGENCY]

AGENT HEALTH:
  Total agents: [count]
  Running: [count] | Crashed: [count] | Not running: [count]
  Recent restarts: [list any agents with uptime < device uptime]

FINDINGS:
1. [Severity] [Component] — [Description]
   Domain: [System | Interface | MLAG | VXLAN | Environment]
   Observed: [metric value]
   Threshold: [normal/warning/critical range]
   Action: [recommended action]

2. ...

DC EXTENSIONS:
  MLAG: [healthy | degraded | critical | not configured]
    State: [active/inactive], Peer-link: [up/down], Config-sanity: [consistent/inconsistent]
  VXLAN/EVPN: [healthy | degraded | critical | not configured]
    VTEPs: [observed/expected], EVPN peers: [established/total]

RECOMMENDATIONS:
- [Prioritized action list]

NEXT CHECK: [date based on severity — CRITICAL: 24hr, WARNING: 7d, HEALTHY: 30d]
```

## Troubleshooting

### Device Unresponsive to SSH

Try console access. If console is also unresponsive, check power and environment
via out-of-band management. After recovery: `show reload cause` for the trigger,
`bash dmesg | tail 50` for kernel messages, `show logging last 50` for pre-crash
syslog. EOS stores crash data in `/var/log/` — accessible via `bash ls -la /var/log/`.

### Agent Crash and Recovery

If `show agent` shows an agent as `crashed` or with uptime shorter than the device:
1. Check agent logs: `show agent [name] logs | tail 30`
2. Check syslog: `show logging last 50 | include [agent-name]`
3. Linux-level: `bash journalctl -u [agent-name] --no-pager | tail 30`
4. If the agent is not running, attempt: `agent [name] shutdown` then
   `no agent [name] shutdown` in configuration mode (note: this is a config change)

Agent crashes are the most common EOS-specific failure mode. Collect logs before
restarting — the crash data is needed for TAC investigation.

### CPU Spikes During Health Check

EOS show commands can briefly spike CPU on the management plane. Wait 30 seconds
after connecting before collecting CPU data. Use `show processes top once` rather
than interactive `top` (which holds a session) for scripted checks.

### MLAG Split-Brain Detection

If both peers report MLAG state `active` but negotiation shows `disconnected`,
this is a split-brain condition. Both peers believe they are the primary. Causes:
peer-link physical failure, peer-link VLAN issue, or heartbeat network partition.
Immediate action: verify physical peer-link connectivity. If peer-link is
physically up but MLAG reports disconnected, check VLAN trunking on the
peer-link and the heartbeat network (usually the management VRF).

### Linux-Native Diagnostics

EOS's Linux foundation means standard Linux tools work from `bash`:
- `bash top -bn1` — process-level CPU and memory (more granular than EOS `top`)
- `bash free -m` — memory with buffer/cache detail
- `bash df -h` — filesystem usage (important for /mnt/flash and /var/log)
- `bash dmesg | tail` — kernel messages (hardware errors, driver issues)
- `bash uptime` — load averages
- `bash cat /proc/meminfo` — detailed memory breakdown

These supplement EOS show commands when deeper investigation is needed.
Prefix all commands with `bash` from the EOS CLI, or enter `bash` for a shell.

### Inconsistent Memory Readings

EOS's Linux page cache uses available memory for file caching. `show version`
may report low free memory while actual memory pressure is low. Use
`bash free -m` and check the `available` column (not `free`) for the real
available memory. The `available` column accounts for reclaimable cache and
buffers.