Server clock synchronization: validate chrony sources and recovery

Payment callbacks rejected as expired, intermittent login-token failures, and out-of-order logs across processes can all stem from the same problem: a system clock that is not reliably synchronized. Changing the time zone only changes how time is displayed; it does not correct clock error. Start troubleshooting with time sources, synchronization status, and application tolerances—not a command that forces a clock adjustment.

Using chrony on Debian 12, this article follows a workflow of checking sources, interpreting offsets, adding a readiness gate, and verifying recovery. After paying for an AmdMetal dedicated server, retrieve the credentials from the console and connect over SSH; if multiple IPv4 addresses are assigned, use the first one. The steps below do not change network addresses or require management ports to be publicly exposed.

1. Establish a baseline: distinguish display settings from clock correction

First, record the system time, time zone, kernel version, and current time-synchronization service. Application logs should use a consistent timestamp format that includes the time zone. Keep UTC timestamps in troubleshooting records as well, so display differences are not mistaken for clock drift.

date --iso-8601=seconds
date -u --iso-8601=seconds
timedatectl status
uname -r
systemctl is-active chrony systemd-timesyncd

The synchronization fields in timedatectl provide useful clues, but they do not replace checking the time sources. Also confirm that no other time-synchronization process or configuration-management task is adjusting the clock concurrently. Check virtual machines on the host individually: a correctly synchronized host does not establish that its guests are synchronized.

If chrony is not installed, it can be installed from the system repositories. However, installation may start the service with the distribution's default configuration. On a machine already running applications, review the configuration that will take effect and the package's installation behavior first, rather than installing it and watching what happens.

sudo apt-get update
sudo apt-get install chrony
systemctl status chrony --no-pager

Wall-clock jumps can affect running databases, authentication services, and task schedulers. Until application behavior has been assessed, do not run date -s or chronyc makestep, and do not repeatedly restart the synchronization service in an attempt to “fix the time.”

2. Check time sources: reachability is not trustworthiness

After installation, inspect the main configuration and the files it references. Use time sources approved by your organization. Do not copy public addresses of unknown provenance or assume that any particular source is infrastructure provided by AmdMetal.

sudo grep -R -E \
'^[[:space:]]*(server|pool|peer|sourcedir|confdir|include|makestep|local|offline)' \
/etc/chrony
chronyc activity
chronyc sources -v
chronyc sourcestats -v

server and pool define sources, while included directories may introduce additional settings. Deduplicate sources based on the actual servers involved. Multiple names may ultimately point to the same infrastructure, so the number of configuration lines is not a measure of source independence.

In sources -v, * marks the currently selected synchronization source, + marks a source whose measurements are combined with others, and ? marks a source that is currently unusable for selection, possibly because it is unreachable or has too few samples. Reach is an octal reachability history. A value of 377 means responses were received for the last eight polls; it does not establish that the time is accurate.

Troubleshoot source problems in order

Check name resolution first, then routing to the source addresses and outbound traffic rules, and finally whether samples continue to update. Receiving an NTP response proves only that communication works. Incorrect source time, asymmetric delays, or disagreement between samples can still affect source selection. Do not loosen the firewall for the entire machine simply to remove a ? marker.

3. Interpret offsets: do not judge synchronization by one field

chronyc tracking
journalctl -u chrony -b --no-pager -n 100

Reference time should keep updating. A Leap status of Not synchronised must fail acceptance checks. System time is chrony's estimate of the remaining correction; a value close to zero is not independent proof that the error relative to true UTC is small.

Last offset and RMS offset describe measurements and their statistics, while Root delay and Root dispersion provide information about the path to the reference clock and the associated uncertainty. Read these fields alongside source status instead of treating a single decimal value as proof of a healthy clock.

Record several consecutive sets of results spanning multiple polling intervals. Assess convergence after a cold start and the rebuilding of samples after network recovery separately from steady-state operation. Continually changing the configuration to reduce a single observed value makes it difficult to identify which change had an effect.

4. Add a synchronization gate with a bounded wait

For time-dependent applications, add a bounded wait to startup or deployment checks. The values below are engineering examples only. Actual thresholds should be determined by token lifetimes, audit requirements, and application testing; they are not service commitments.

chronyc waitsync 12 0.05 0 5

This command performs at most twelve checks, five seconds apart. It requires the clock to enter a synchronized state and the remaining correction to fall below 0.05 seconds. The third parameter is zero, so no additional skew limit is imposed. A successful exit means these conditions were met; it does not guarantee an error of less than fifty milliseconds relative to true UTC.

Automation must check the exit status rather than swallow failure output:

if chronyc waitsync 12 0.05 0 5; then
    printf '%s\n' 'Clock gate passed'
else
    printf '%s\n' 'Clock gate failed' >&2
    exit 1
fi

A failed gate should block the relevant startup step and notify operations, not automatically step the clock as a repair. network-online.target concerns network readiness only; it does not prove that clock synchronization is complete.

Test the failure path

In an isolated test environment, verify that the gate exits with a nonzero status when no usable sources are available. Then restore the sources and verify the success path. Do not change the global system time on a production dedicated server to introduce an offset: doing so affects every service and contaminates the logs used to establish the sequence of an incident.

5. Validate recovery and retain evidence that can be interpreted

If sources were manually marked offline, run chronyc online only after confirming that the network has recovered. If additional samples are needed, run chronyc burst 4/4 once against the configured sources, then inspect source selection and tracking again. Do not call it in a loop and generate pointless requests.

When reviewing makestep, pay particular attention to configurations such as makestep 1.0 3: this allows offsets above the threshold to be corrected by stepping the clock during the first few clock updates after startup. Restarting the service may re-enter that phase, so “just restart it” is not risk-free. If a large clock step is genuinely necessary, first isolate the affected applications and prepare a recovery plan.

Final acceptance checks should confirm that:

• A suitable source is selected, the reference time continues to advance, and the system does not remain on stale samples for an extended period.
• The remaining correction and uncertainty meet application thresholds, with no clear deterioration across consecutive observations.
• Both the gate's success path and failure exit have been tested, and alerts can be received.
• Authentication, callback handling, and log ordering have been retested at the application level, rather than merely confirming that processes are alive.
• Configuration changes, command output, and the observation interval have been retained; access to source addresses in logs is controlled through permissions.

Prefer a monotonic clock for measuring elapsed time and enforcing timeouts, and use the wall clock for application timestamps. Keep application timekeeping separate from host synchronization acceptance checks. This prevents a clock correction from turning a normal duration negative and keeps subsequent incident records interpretable.

Frequently asked questions