Chapter 3 — Packages, services, processes, logs

Package management: never paste across families

Verify the distribution first (cat /etc/os-release, os-release(5)), then use the native manager. Package ownership queries are also manager-specific: dpkg -S (Debian/Ubuntu), rpm -qf (Fedora/RHEL), pacman -Qo (Arch).

Debian / Ubuntu — APT. Inspect with apt list --upgradable, apt show PACKAGE, apt policy PACKAGE; refresh indexes with sudo apt update — this refreshes indexes only, not installed packages. Install/remove with sudo apt install/remove PACKAGE; simulate with apt-get -s install PACKAGE. apt is the interactive user-facing CLI; for stable scripting interfaces prefer apt-get/apt-cache (Debian Reference: package management; Ubuntu Server: Package management; apt(8)). Review the proposed removals/upgrades before confirming.

Fedora / RHEL / AlmaLinux — DNF. Typical patterns: dnf info PACKAGE, dnf repoquery --whatprovides /path, sudo dnf install/remove PACKAGE, sudo dnf upgrade, then dnf history list / dnf history info ID to audit what happened (DNF Command Reference). One automation trap: dnf check-update exits 100 when updates are available, 0 when none, 1 on error — do not write automation that treats every nonzero result as the same failure. Review the transaction summary, especially removals and repository changes; avoid --assumeyes for exploratory/admin work. (DNF vs DNF5 availability and exact command behavior depend on the release — verify locally.)

Arch — pacman. Inspect/search with pacman -Ss, pacman -Qi, pacman -Ql, pacman -Qo; install with sudo pacman -S PACKAGE; update with the full system upgrade sudo pacman -Syu. Arch explicitly does not support partial upgrades: do not run pacman -Sy PACKAGE as an install shortcut, because refreshed repository metadata against un-upgraded installed packages can create incompatible library states (ArchWiki: pacman; ArchWiki: System maintenance — partial upgrades are unsupported).

HIGH-RISK — force/assume flags and provenance. Avoid as casual troubleshooting: -y/--assumeyes, DNF --allowerasing (explicitly permits removals — DNF5 upgrade), --nogpgcheck, DNF --skip-broken, pacman -Rdd/--overwrite, and automatic autoremove. These can bypass review/signature checks, force removals, or leave dependency/database state inconsistent; resolve the underlying repository/dependency/ownership problem instead (DNF Command Reference; ArchWiki: pacman; apt-get(8)). Never disable GPG/signature verification to make an install proceed; never pipe an unaudited remote script directly into a root shell — download, inspect, verify publisher/checksum/signature from a trusted source, then run the documented installer with least privilege.

systemd services: runtime state is not boot state

First confirm systemd (ps -p 1 -o comm=, systemctl --version), then inspect: systemctl status NAME, systemctl cat NAME, systemctl show NAME, systemctl list-units --type=service --all (systemctl(1)). Two states are commonly confused: systemctl start NAME starts now; enable NAME arranges boot activation; enable --now / disable --now do both. An active unit need not be enabled, and an enabled one need not be running (systemd.service(5)).

  • restart stops and starts; reload asks the service to reload its own configuration and only works if the unit/service supports it.
  • After editing or adding a unit or drop-in: sudo systemctl daemon-reload (rereads unit definitions; does not restart the application or re-read the application's own config), then restart/reload the affected unit, and verify with systemctl status plus journalctl -u NAME -b (systemd.unit(5)).
  • Prefer drop-ins under /etc/systemd/system/NAME.service.d/ created with systemctl edit NAME; inspect the merged result with systemctl cat NAME (systemctl(1)).

Timers [systemd]. Inspect systemctl list-timers --all and systemctl status NAME.timer; validate calendar expressions with systemd-analyze calendar 'Mon..Fri 09:00'. A timer commonly activates a same-stem .service; Persistent=true on a calendar timer can trigger a missed run after the machine returns, subject to unit/timer semantics (systemd.timer(5); systemd.time(7); systemd-analyze(1)).

Processes and resources

Inventory: ps -eo pid,ppid,user,stat,%cpu,%mem,comm --sort=-%cpu (procps/GNU-style; verify local ps syntax), pgrep -a pattern, pstree -ap where installed, and interactive top (ps(1)). Get a specific process's state/parent/limits from ps, /proc/PID/status, /proc/PID/limits, and systemctl status if service-managed (Linux /proc documentation).

Memory and CPU: free -h, vmstat 1, uptime, nproc; mpstat/iostat when sysstat is installed (sysstat). Load average is runnable/uninterruptible task pressure averaged over time, not a direct CPU-percent metric — correlate it with per-core CPU utilization, process state, and memory/swap/IO pressure (kernel loadavg documentation; free(1)).

HIGH-RISK — killing processes. Collect evidence first (systemctl status, journalctl -u NAME -b, ps, free, df, socket checks), then send TERM first — systemctl stop NAME for a service or kill -TERM PID for an unmanaged process — and reserve KILL for a process confirmed not to have exited, because SIGKILL cannot be caught and prevents orderly cleanup (kill(1); systemctl(1)). A PID can be reused: re-check identity before signaling an old PID.

Sockets: ss -lntup (some process attribution needs root); ip address, ip route, getent hosts NAME, and resolvectl status where systemd-resolved is used (ss(8); ip(8)). Do not assume netstat, ifconfig, or nslookup are installed on a minimal host.

Logs: the journal, with the retention reality

Query patterns [systemd]: journalctl -b (current boot), journalctl -u NAME -b, journalctl -u NAME --since '1 hour ago', journalctl -f -u NAME (follow), journalctl -p warning..alert (severity), journalctl -k -b (kernel), and journalctl -k -b -1 for the previous boot when retained (journalctl(1)).

Logging differs by distro and configuration: systemd systems commonly query journald while traditional syslog files may also exist under /var/log; journald storage can be volatile (runtime) or persistent according to configuration — never assume logs survive reboot or host loss (journald.conf(5)). Export or forward important logs off-host, restrict access, synchronize clocks, and define retention appropriate to operations and privacy requirements. journalctl --disk-usage inspects usage.

HIGH-RISK — journal vacuuming. journalctl --vacuum-time= / --vacuum-size= delete archived journal files; they are not display filters. Check --disk-usage, capture any required incident evidence, and prune only under an explicit retention policy (journalctl(1)).