Technical Guide
Comparing Healthy and Failing Linux Hosts Before Broad Changes
Compare concrete host, service, package, resolver, route, and authentication state between a healthy and failing Linux system before restarting or rewriting configuration broadly.
Quick Read
- Symptom: Compare concrete host, service, package, resolver, route, and authentication state between a healthy and failing Linux system before restarting or rewriting configuration broadly.
- Check first: Capture the exact failing command, client, user, hostname, service, timestamp, and error before restarting anything.
- Risk: Read-only checks
Symptoms
A Linux host can fail because of authentication, service state, resolver configuration, routes, package/repository state, permissions, or an upstream dependency. Restarting services or rewriting configuration too early removes the evidence that distinguishes those cases.
Environment
Linux servers and services where a comparable healthy host, user path, service instance, or network path is available for side-by-side troubleshooting.
Most Likely Causes
Linux incidents often present as a generic connection or service failure even though the useful differences are concrete: unit state, recent journal entries, package version, repository candidate, resolver configuration, routes, firewall state, SSH authentication, environment, or file permissions.
What to Check First
Capture the exact failing command, client, user, hostname, service, timestamp, and error before restarting anything.
Choose the closest healthy comparison host or service instance and record meaningful differences in OS/package level, role, network path, and identity context.
Collect service, journal, resolver, route, repository/package, and authentication evidence from both systems using read-only commands first.
Related Guides
Use these when the problem moves into a neighboring part of the same workflow.
Operational Steps
- Establish host and package context
Compare distribution/release, kernel, relevant package version, architecture, enabled repositories, and the package candidate that would be installed today. A host that looks identical at the application layer may differ because one has a newer package, a disabled repository, a held package, or a different configuration package revision.
- Compare service state and recent journal evidence
Use systemd state and the journal to compare ActiveState/SubState, unit file state, recent restarts, exit status, dependency failures, and configuration errors. Capture the journal window around the failed request or start attempt. Do not restart the unit merely to see whether it helps; first preserve why it is currently stopped, degraded, or rejecting requests.
- Compare resolver and route state
Record configured nameservers/search domains, the resolver implementation in use, DNS answers, interface addresses, route table, default gateway, and policy routes relevant to the destination. Run the same name-resolution and reachability test on the healthy host. If DNS differs, stay with resolver/suffix/upstream DNS evidence; if DNS agrees but the route differs, move to network pathing.
- Compare SSH or application authentication context
For access failures, compare the user identity, SSH client options, key selected, server-side sshd configuration that applies to that user/source, PAM/account state, and authentication log entries. A successful shell as another user does not prove the failing identity follows the same auth path.
- Compare configuration and permissions only where the symptom points
Diff the relevant service configuration, environment file, unit override, ownership, mode, ACL, or SELinux/AppArmor context instead of recursively comparing the entire host. Focus on files the failing service actually reads and paths it actually needs.
- Correct the first verified difference and rerun the same transaction
Once a specific difference explains the symptom, make the smallest approved correction and repeat the original client command or service request. If the result does not change as predicted, restore the previous setting before moving to another layer.
Validation
The incident record contains a side-by-side comparison of the Linux state relevant to the symptom rather than only a list of commands run on the failing host.
The first meaningful difference is classified as service/runtime, package/repository, DNS/resolver, route/network, authentication, or configuration/permission state.
After correction, the same user/client transaction that originally failed now succeeds or produces a narrower, explainable error.
Temporary diagnostic changes are removed or documented after validation.
Logs to Check
systemd unit state and journal entries for the affected service and its dependencies.
Authentication logs such as journal/sshd, auth.log, or secure for SSH and PAM failures, depending on distribution.
Package-manager and repository output for dependency, candidate-version, signature, or metadata problems.
Resolver/network evidence including resolvectl or resolver config, DNS query results, routes, neighbor state, and host firewall evidence where relevant.
Rollback and Escalation
Record the original service configuration, package/repository state, route/resolver setting, or permission before changing it.
Revert targeted diagnostic changes that do not produce the expected result instead of leaving several speculative changes in place.
Escalate When
Escalate when the comparison points to a shared identity, DNS, repository, overlay-network, storage, or security-policy dependency owned elsewhere.
Escalate before package downgrade, filesystem repair, security-policy relaxation, or a change that can affect multiple services without a tested recovery path.
Notes from the Field
A shell prompt is not proof that the failing service path is healthy. Validate the same identity, destination, and transaction that failed.
The journal window immediately before a restart is often more useful than the clean startup messages that appear afterward.