Chapter 3 — Packages, services, processes, logs
Package management: never paste across families
Verify the distribution first (cat /etc/os-release, os-release(5)), then use the native manager. Package ownership queries are also manager-specific: dpkg -S (Debian/Ubuntu), rpm -qf (Fedora/RHEL), pacman -Qo (Arch).
Debian / Ubuntu — APT. Inspect with apt list --upgradable, apt show PACKAGE, apt policy PACKAGE; refresh indexes with sudo apt update — this refreshes indexes only, not installed packages. Install/remove with sudo apt install/remove PACKAGE; simulate with apt-get -s install PACKAGE. apt is the interactive user-facing CLI; for stable scripting interfaces prefer apt-get/apt-cache (Debian Reference: package management; Ubuntu Server: Package management; apt(8)). Review the proposed removals/upgrades before confirming.
Fedora / RHEL / AlmaLinux — DNF. Typical patterns: dnf info PACKAGE, dnf repoquery --whatprovides /path, sudo dnf install/remove PACKAGE, sudo dnf upgrade, then dnf history list / dnf history info ID to audit what happened (DNF Command Reference). One automation trap: dnf check-update exits 100 when updates are available, 0 when none, 1 on error — do not write automation that treats every nonzero result as the same failure. Review the transaction summary, especially removals and repository changes; avoid --assumeyes for exploratory/admin work. (DNF vs DNF5 availability and exact command behavior depend on the release — verify locally.)
Arch — pacman. Inspect/search with pacman -Ss, pacman -Qi, pacman -Ql, pacman -Qo; install with sudo pacman -S PACKAGE; update with the full system upgrade sudo pacman -Syu. Arch explicitly does not support partial upgrades: do not run pacman -Sy PACKAGE as an install shortcut, because refreshed repository metadata against un-upgraded installed packages can create incompatible library states (ArchWiki: pacman; ArchWiki: System maintenance — partial upgrades are unsupported).
HIGH-RISK — force/assume flags and provenance. Avoid as casual troubleshooting:
-y/--assumeyes, DNF--allowerasing(explicitly permits removals — DNF5 upgrade),--nogpgcheck, DNF--skip-broken, pacman-Rdd/--overwrite, and automatic autoremove. These can bypass review/signature checks, force removals, or leave dependency/database state inconsistent; resolve the underlying repository/dependency/ownership problem instead (DNF Command Reference; ArchWiki: pacman; apt-get(8)). Never disable GPG/signature verification to make an install proceed; never pipe an unaudited remote script directly into a root shell — download, inspect, verify publisher/checksum/signature from a trusted source, then run the documented installer with least privilege.
systemd services: runtime state is not boot state
First confirm systemd (ps -p 1 -o comm=, systemctl --version), then inspect: systemctl status NAME, systemctl cat NAME, systemctl show NAME, systemctl list-units --type=service --all (systemctl(1)). Two states are commonly confused: systemctl start NAME starts now; enable NAME arranges boot activation; enable --now / disable --now do both. An active unit need not be enabled, and an enabled one need not be running (systemd.service(5)).
restartstops and starts;reloadasks the service to reload its own configuration and only works if the unit/service supports it.- After editing or adding a unit or drop-in:
sudo systemctl daemon-reload(rereads unit definitions; does not restart the application or re-read the application's own config), then restart/reload the affected unit, and verify withsystemctl statusplusjournalctl -u NAME -b(systemd.unit(5)). - Prefer drop-ins under
/etc/systemd/system/NAME.service.d/created withsystemctl edit NAME; inspect the merged result withsystemctl cat NAME(systemctl(1)).
Timers [systemd]. Inspect systemctl list-timers --all and systemctl status NAME.timer; validate calendar expressions with systemd-analyze calendar 'Mon..Fri 09:00'. A timer commonly activates a same-stem .service; Persistent=true on a calendar timer can trigger a missed run after the machine returns, subject to unit/timer semantics (systemd.timer(5); systemd.time(7); systemd-analyze(1)).
Processes and resources
Inventory: ps -eo pid,ppid,user,stat,%cpu,%mem,comm --sort=-%cpu (procps/GNU-style; verify local ps syntax), pgrep -a pattern, pstree -ap where installed, and interactive top (ps(1)). Get a specific process's state/parent/limits from ps, /proc/PID/status, /proc/PID/limits, and systemctl status if service-managed (Linux /proc documentation).
Memory and CPU: free -h, vmstat 1, uptime, nproc; mpstat/iostat when sysstat is installed (sysstat). Load average is runnable/uninterruptible task pressure averaged over time, not a direct CPU-percent metric — correlate it with per-core CPU utilization, process state, and memory/swap/IO pressure (kernel loadavg documentation; free(1)).
HIGH-RISK — killing processes. Collect evidence first (
systemctl status,journalctl -u NAME -b,ps,free,df, socket checks), then sendTERMfirst —systemctl stop NAMEfor a service orkill -TERM PIDfor an unmanaged process — and reserveKILLfor a process confirmed not to have exited, because SIGKILL cannot be caught and prevents orderly cleanup (kill(1); systemctl(1)). A PID can be reused: re-check identity before signaling an old PID.
Sockets: ss -lntup (some process attribution needs root); ip address, ip route, getent hosts NAME, and resolvectl status where systemd-resolved is used (ss(8); ip(8)). Do not assume netstat, ifconfig, or nslookup are installed on a minimal host.
Logs: the journal, with the retention reality
Query patterns [systemd]: journalctl -b (current boot), journalctl -u NAME -b, journalctl -u NAME --since '1 hour ago', journalctl -f -u NAME (follow), journalctl -p warning..alert (severity), journalctl -k -b (kernel), and journalctl -k -b -1 for the previous boot when retained (journalctl(1)).
Logging differs by distro and configuration: systemd systems commonly query journald while traditional syslog files may also exist under /var/log; journald storage can be volatile (runtime) or persistent according to configuration — never assume logs survive reboot or host loss (journald.conf(5)). Export or forward important logs off-host, restrict access, synchronize clocks, and define retention appropriate to operations and privacy requirements. journalctl --disk-usage inspects usage.
HIGH-RISK — journal vacuuming.
journalctl --vacuum-time=/--vacuum-size=delete archived journal files; they are not display filters. Check--disk-usage, capture any required incident evidence, and prune only under an explicit retention policy (journalctl(1)).