Chapter 10 — Troubleshooting recipes

Each recipe follows the universal sequence — observe and scope → identify the exact layer and target → preserve evidence → make one reversible change → verify service and data → document outcome — which follows from the read-only inventory tools, filesystem-checker constraints, firewall runtime/permanent separation, and the mutation risk of repair/restore operations (lsblk(8); e2fsck(8); firewall-cmd; restic restore).

Recipe A — a service is unreachable from a client

Run the layered client-side test of Chapter 4 (resolution → route → TCP → listener → TLS), then the fast triage matrix to pick the next discriminator. On the server, verify the listener and its bind address (sudo ss -lntup) and recent logs (sudo journalctl -u SERVICE --since '-15 min' --no-pager). A listener is necessary but not sufficient — check host firewall, upstream ACLs, and return path (ss(8); journalctl(1)).

Recipe B — disk full, but du disagrees

Check both capacity axes (df -hT, df -ih), then hunt the hidden consumers: deleted-but-open files (/proc/PID/fd; lsof +L1 where installed) and data shadowed under a mount point. Distinguish bytes from inodes from quotas from reserved space before deleting anything (df(1); proc(5)). If the answer is log growth, remember that vacuuming deletes archived journals — export evidence first (Chapter 3).

Recipe C — permission denied despite open-looking modes

Inspect the whole chain: namei -l /path/to/file for every parent component (directory execute = search), getfacl -p for ACL masks (watch the effective: annotations), and the MAC layer (SELinux labels/AVCs on Fedora/RHEL-family; AppArmor deny logs on Ubuntu). Weakening controls is the last resort, after identifying the real layer (getfacl(1); Fedora SELinux; Ubuntu AppArmor).

Recipe D — SSH won't let me in

Establish which failure it is: connection refused/timeout (Chapter 4 layered test), authentication rejected (inspect effective policy with sudo sshd -T, including -C context for Match rules; unit and config-include discovery per distribution), or post-auth failure (journal). Any change to the SSH control plane follows the lockout-safe preflight and sequence of Chapter 4 — including keeping the original session open until a fresh login succeeds (sshd(8); sshd_config(5); Ubuntu OpenSSH).

Recipe E — boot hangs or a mount is wrong

Inspect fstab as systemd sees it: findmnt --fstab --verify --verbose for parsability/usability, findmnt for what actually mounted, then the generated mount unit status. Remember nofail/noauto/_netdev/x-systemd.* change boot dependencies; test the intended mount before rebooting (systemd.mount(5); findmnt(8)).

Recipe F — service fails after a config edit

Collect evidence before restarting: systemctl status, journalctl -u NAME -b, the unit's systemctl cat (drop-ins included), and the application's own config checker. Distinguish daemon-reload (unit definitions) from reload (application config) — neither performs the other. Restart only when the cause is understood, and keep monitoring logs during recovery (systemctl(1)).

Recipe G — package transaction fails or offers to remove things

Never resolve with force flags. Simulate (apt-get -s, DNF's transaction review, dnf history info ID after), inspect the proposed removals/replacements, and fix the underlying repository/dependency/file-ownership problem. On Arch, remember partial upgrades are unsupported — resolve with a full pacman -Syu, not selective upgrades (DNF Command Reference; ArchWiki: pacman; apt-get(8)).

Recipe H — suspected compromise, first hour

Follow Chapter 9: preserve logs and volatile state before any cleanup, write the timeline with times/time zone, scope before containing, and do all credential work from a trusted clean device (NIST SP 800-61r3). The recipes in this edition deliberately stop short of malware-forensics workflows — see the source index's verification note.