Diagnose Receive Drops Across the NIC, Softirq and Network Stack

Intermittent application timeouts and rising receive-drop counters do not necessarily mean the NIC is faulty. On a dedicated server, the receive path spans NIC queues, the driver, software interrupts and the network stack. The same symptom can originate at different layers. Troubleshooting should correlate application errors with counter increases over the same time window, rather than expand every buffer as soon as dropped appears.

The examples below use Debian 12 and require administrator privileges plus iproute2, ethtool, sysstat and python3. After paying for an AmdMetal dedicated server, you can find its connection credentials in the control panel; use the first IPv4 address for SSH. We begin with read-only sampling, without changing addresses or routes.

1. Identify the Receive Interface and Failure Window

Start with ip -br address to identify the interface holding the application's service address, then trace through any VLAN, bridge or bond to the physical interface carrying the traffic. Do not simply select the first NIC alphabetically. The following commands prompt for the interface and automatically create a sampling directory:

read -r -p "Physical interface: " IFACE
ip link show dev "$IFACE" || exit 1
D=$(mktemp -d /var/tmp/rx-audit.XXXXXX)
printf '%s\n' "$IFACE" > "$D/interface"
uname -r > "$D/kernel"
ethtool -i "$IFACE" > "$D/driver"
ip -d link show dev "$IFACE" > "$D/link"
printf '%s\n' "$D"

Record when application errors occur, the request rate, packet sizes and whether a bulk import is running. Interface counters may cover a long uptime, so their cumulative values cannot establish a causal link to a timeout that has just occurred. Reloading the driver, rebooting or recreating an interface may also reset counters.

NIC, interface and network-stack statistics are not independent accounts that can simply be added together. Some fields may describe the same event. Check the driver's definitions before interpreting the increases.

If the application runs in a network namespace, the physical interface shows the host's receive path, while network-stack statistics must also be collected in the application's namespace. An absence of errors on the host does not mean everything is healthy inside the container.

2. Capture Interface and Driver Statistics Over the Same Window

Continue in the same shell. The example uses a 30-second observation window; this is a sampling parameter, not a service commitment. Capture a workload phase in which the problem actually occurs, and collect a baseline over an equally long healthy window.

snapshot() {
    label=$1
    date -Is > "$D/$label.time"
    ip -s -s link show dev "$IFACE" > "$D/$label.link"
    ethtool -S "$IFACE" > "$D/$label.nic" 2>&1
    cat /proc/net/softnet_stat > "$D/$label.softnet"
    nstat -az > "$D/$label.protocol"
}
snapshot before
mpstat -P ALL 1 30 > "$D/cpu"
snapshot after
diff -u "$D/before.nic" "$D/after.nic"

These reads run sequentially; they are not an atomic snapshot. For short bursts, interpret the results alongside the timestamps rather than expecting packet-for-packet agreement between files. If ethtool -S reports that the operation is unsupported, keep that output. “Statistics unavailable” does not mean “no packet loss.”

Check What the Counters Mean First

If error counters containing CRC or frame increase, investigate link-layer evidence. For fields containing missed, no_buffer or discard, consult the definitions for the current driver. They may concern receive descriptors, device buffers or filtering behavior; the English field name alone does not establish the cause.

Also save the output of ethtool -g "$IFACE" and ethtool -l "$IFACE" to inspect ring and channel information, respectively. Unsupported queries do not, by themselves, indicate a fault. Maximum values describe device limits, not recommended production settings.

3. Calculate Per-CPU softnet Deltas

/proc/net/softnet_stat uses hexadecimal values. Its first three columns are processed, dropped and time_squeeze, in that order. Later columns can vary with the kernel and are not parsed here. The script below compares rows and reports the row number in the sampled data. If CPUs are hot-plugged during sampling, collect a new sample rather than continuing to match rows mechanically.

python3 - "$D/before.softnet" "$D/after.softnet" <<'PY'
import sys
from pathlib import Path

def load(path):
    return [
        [int(v, 16) for v in line.split()[:3]]
        for line in Path(path).read_text().splitlines()
    ]

before, after = map(load, sys.argv[1:])
if len(before) != len(after):
    raise SystemExit("CPU layout changed; sample again")

print("row processed_delta dropped_delta time_squeeze_delta")
for row, (old, new) in enumerate(zip(before, after)):
    delta = [b - a for a, b in zip(old, new)]
    if any(v < 0 for v in delta):
        raise SystemExit("Counter reset or wrap; sample again")
    print(row, *delta)
PY

An increase in dropped here indicates drops from the software receive backlog. An increase in time_squeeze means a processing round reached its budget limit; it does not mean packets have already been dropped. softnet statistics are aggregated per CPU, not dedicated to a single NIC, so you must rule out traffic from other interfaces during the same interval.

Compare the deltas with %soft in the mpstat output. A server can have idle CPU capacity overall while one core is under receive-side pressure, so average utilization is not enough. If needed, save /proc/interrupts and check whether NIC queue interrupts are heavily concentrated on particular CPUs. Interrupt distribution is a clue, not an acceptance criterion requiring an even spread.

4. Choose the Tuning Layer Based on Evidence

If NIC missed-type counters increase while softnet dropped stays unchanged, inspect the receive ring and driver queues first. Conversely, if softnet dropped rises alongside pressure on one core, investigate the software backlog and traffic distribution. If both remain stable, compare protocol-error deltas in nstat with application logs instead of continuing to tune the NIC blindly.

Save the current configuration before making changes:

ethtool -g "$IFACE" > "$D/rings" 2>&1
ethtool -l "$IFACE" > "$D/channels" 2>&1
sysctl net.core.netdev_max_backlog \
       net.core.netdev_budget \
       net.core.netdev_budget_usecs > "$D/budgets"

Consider changing ring capacity only when the driver supports it and the evidence points there. “Set it to the maximum” is not a general solution. Expanding the backlog may trade fewer drops for longer queueing delays; increasing the processing budget may take CPU time away from the application. Changes to multiqueue distribution must also account for IRQs, RSS and the number of traffic flows. A single flow will not automatically spread across all cores as a result.

Do not immediately try channel reconfiguration, a driver reload or a link reset on a remote system. These operations may disrupt SSH. First prepare a controlled procedure, record the original settings and establish a recovery path. Change only one setting per iteration, and restore its original value first if problems occur.

5. Validate the Result with the Application Workload

Keep request types, test duration and concurrency patterns comparable during retesting, and record the actual traffic. Comparing only the absolute number of drops in two windows can mistake a reduction in load for a successful optimization. Even at the same throughput, a larger number of small packets can place different demands on receive processing.

Validation should include at least three sets of results:

• Receive-path evidence: the target counter deltas decrease, with no new link errors. An unchanged cumulative total does not constitute failure.
• Processing cost: per-core softirq load and application CPU usage show no new significant contention, and queueing latency has not worsened.
• Application behavior: timeouts, failed requests and tail latency improve together, rather than only the low-level counters looking better.

If low-level counters remain stable but application errors continue, limit the conclusion to “No evidence of corresponding receive drops was found during this window.” Continue investigating the network stack and application processing rather than declaring the network completely healthy. Retain the interface, driver and kernel details, sample files and original settings so that you can reassess the causal chain when the same symptoms recur.

Frequently asked questions