Chapter 4 — Networking: DNS, routing, firewall, SSH, TLS

Who owns the interface and the resolver?

Before editing, identify the configuration owner — this is the step that prevents editing a file that something else generates or overwrites (Ubuntu: About Netplan; Debian Wiki: NetworkConfiguration):

# [READ-ONLY]
systemctl is-active NetworkManager systemd-networkd systemd-resolved networking
nmcli general status; nmcli device status      # NetworkManager present?
networkctl list; networkctl status             # systemd-networkd present?
readlink -f /etc/resolv.conf                   # stub symlink or real file?
resolvectl status                              # only meaningful if systemd-resolved runs

Unavailable commands/services are normal. Record which daemon is active, which interface it manages, which profile/file is active, and whether resolver configuration is a symlink before changing anything. systemctl is-active can return non-zero if one queried unit is inactive — read each printed status.

• Ubuntu: Netplan is a YAML abstraction with renderers for NetworkManager and systemd-networkd; Netplan configuration and renderer may be the persistent source of truth, so do not edit a generated backend file if Netplan owns it (Ubuntu: About Netplan; Netplan YAML).
• Fedora/RHEL/AlmaLinux: NetworkManager connection profiles are the standard configuration/control mechanism; inspect with nmcli connection show --active and nmcli connection show PROFILE (RHEL 9: Configuring and managing networking — note: this portal denied automated retrieval on 2026-10-10; verify against the installed release docs).
• systemd-networkd: configured by .network/.netdev files, inspected with networkctl; networkctl reconfigure IFACE changes a live interface, so have rollback and console access first (systemd.network; networkctl).
• Debian/ifupdown: /etc/network/interfaces applies only if ifupdown is actually in use; NetworkManager or networkd may own the link instead. Avoid configuring the same interface in two managers (Debian Reference ch.5).
• DNS: resolvectl applies only with systemd-resolved (resolvectl). A symlinked /etc/resolv.conf pointing at a local stub is not itself an upstream resolver; follow the stub's status/configuration to the real per-link upstream (systemd-resolved).

The layered connectivity test (client side)

Test from the affected client and, where possible, from the server: resolution → route choice → TCP reachability → local listener → protocol/TLS. No single failed ping, port probe, or DNS query proves a firewall fault; compare results by layer, address family, hostname/SNI, and vantage point. The ip utility reports/manages addresses, routes, and policy rules, and ip route get DESTINATION asks the kernel to resolve a route while ip rule exposes policy-routing rules (ip(8); ip-route(8)). getent follows the host's NSS path used by ordinary applications, while dig sends DNS queries — a dig answer does not prove NSS/application resolution succeeds (getent(1); dig(1); RFC 1034). DNS is cached per TTL, so a correct authoritative change need not appear immediately in cached answers, and DNS supports both UDP and TCP — a UDP-only test does not test TCP/53 (RFC 1034; RFC 7766).

# [READ-ONLY] set HOST/PORT/IP for the actual service; needs dig/ip/getent/curl/openssl
HOST=service.example.net
PORT=443
IP=203.0.113.25

getent ahosts "$HOST"                                   # NSS path apps actually use
dig "$HOST" A +time=2 +tries=1                          # default resolver
dig "$HOST" AAAA +time=2 +tries=1
dig @192.0.2.53 "$HOST" A +time=2 +tries=1              # specific resolver, UDP
dig +tcp @192.0.2.53 "$HOST" A +time=2 +tries=1         # same, TCP/53

ip -br address
ip route get "$IP"                                      # kernel's route choice, IPv4
ip -6 route get 2001:db8::25                            # ...and IPv6

curl -v --connect-timeout 5 --resolve "$HOST:$PORT:$IP" \
  --output /dev/null "https://$HOST:$PORT/"             # bypasses DNS, keeps SNI

openssl s_client -connect "$IP:$PORT" -servername "$HOST" \
  -verify_hostname "$HOST" -verify_return_error -showcerts </dev/null

Interpretation, layer by layer:

1. Name-resolution/NSS failure: getent fails or returns an unexpected address while dig succeeds — points toward NSS, /etc/hosts, or resolver integration rather than an authoritative DNS fault; inspect /etc/nsswitch.conf, /etc/hosts, /etc/resolv.conf, and resolvectl status.
2. DNS response distinction: NXDOMAIN = the responding system says the name does not exist; SERVFAIL = resolution failure at the queried resolver; timeout = no usable response (routing, filtering, server, or transport). Compare resolver IPs, A vs AAAA, UDP vs TCP, and authoritative answers before changing records. dig +trace NAME helps when recursion/delegation is suspect but can be blocked by network policy.
3. Route/address-family problem: ip route get shows selected interface, next hop, and source. A route lookup proves the local kernel's choice, not that gateways or return routes work; compare curl -4/curl -6 when both families exist.
4. TCP path/service problem: the curl --resolve test bypasses ordinary DNS for that request while preserving hostname and TLS SNI. Connection refusal commonly indicates an active reject or no listener; a timeout is ambiguous among filtering, routing, outage, and silent drops. Confirm at both ends rather than declaring "the firewall" from a timeout.
5. Listener/bind problem (server side, root):

# [READ-ONLY · root]
sudo ss -lntup
sudo systemctl status SERVICE
sudo journalctl -u SERVICE --since '-15 min' --no-pager

A service bound only to 127.0.0.1/::1 is not listening externally; a listener on one address family may not cover the other. A listener is necessary, not sufficient — host firewall, upstream firewall, routing, and application-level access controls can still block traffic.

1. Firewall evidence: identify the active manager first — sudo firewall-cmd --state, --get-active-zones, --zone=ZONE --list-all for firewalld; sudo nft list ruleset for nftables; sudo ufw status verbose for UFW. A local ruleset cannot show an upstream ACL or the remote host's return-path policy (firewall-cmd; nft(8)).
2. Packet-path evidence (optional, root): capture on both endpoints while repeating one controlled connection attempt:

# [READ-ONLY · root]
sudo tcpdump -ni any "host $IP and tcp port $PORT"

A client SYN leaving with no reply narrows the problem beyond that capture point or to the return path; a server-side SYN arriving with no SYN-ACK points to server-side filtering, no matching listener, or local TCP policy. RST/ICMP responses are clues, not infallible attribution. Captures expose metadata — avoid storing or sharing unnecessary packet contents (tcpdump(8)).

1. TLS/application layer: if TCP connects but curl/s_client fails, inspect the exact TLS alert, certificate chain, SAN, validity dates, trusted issuer, and SNI-selected certificate. curl -k disables verification and is not a valid success test — use it only as a narrowly scoped diagnostic to isolate whether verification is the failing layer. openssl s_client without -verify_return_error normally continues after verification errors, so require verification explicitly (OpenSSL s_client). TLS service identity is checked against identifiers in the certificate's Subject Alternative Name: a DNS name must match a DNS-ID and an IP reference must match an IP-ID (RFC 9525).

Fast triage matrix (ss(8); getent(1)):

Known limits of casual tools: ping tests ICMP echo, not TCP port reachability — no reply can mean ICMP is filtered, and a reply does not prove the service port is open (ping(8)). tracepath -n DEST reveals some hops and path-MTU behavior, but routers suppress or rate-limit probes, so missing hops do not prove a broken route (tracepath(8)). For multi-homed, policy-routed, VPN, or NAT systems, inspect both ip rule and all tables (ip route show table all, ip -6 route show table all), and test from an independent vantage — NAT, split DNS, source-based ACLs, asymmetric return routes, and reverse proxies make "works from the server itself" an incomplete external test.

Lockout-safe change procedures

The mandatory preflight (before SSH, firewall, or network changes). (1) Establish recovery: confirm hypervisor/cloud serial console, physical console, remote management, second independent network path, or another verified out-of-band method, and that someone can use it now — a second SSH session over the same route is a check, not out-of-band access. (2) Record state and back up: save the relevant SSH/network/firewall files and outputs (ip address, ip route show table all, ip rule, ss -lntup, active firewall status/rules, active profiles) — protect these backups, as configs can contain Wi-Fi, VPN, or private-key references. (3) Preserve access: keep the original privileged session open; open and test a second session first. (4) Prepare a timed rollback: schedule a detached rollback (systemd timer) or use the manager's checkpoint/try feature; verify the rollback command in advance. (5) Validate candidates before activation (below). (6) Make one narrow change at a time. (7) Prove recovery before commit: from a genuinely new connection, verify SSH on the intended address/port, both address families, and the application from an external vantage; check logs and counters. (8) Commit or roll back deliberately: cancel the rollback only after everything works (sshd(8); firewall-cmd; Netplan try; nmcli; nft(8)).

Validation tools by target:

• SSH: sudo sshd -t (syntax), sudo sshd -T (effective configuration), with -C to test Match-specific settings for the intended user/client context (sshd(8); sshd_config(5)).
• nftables: sudo nft -c -f /path/to/candidate.nft validates without applying (nft(8)).
• firewalld: sudo firewall-cmd --check-config for permanent config; stage narrow runtime rules and inspect before persisting (firewall-cmd).
• Netplan: validate with the installed Netplan tooling; prefer sudo netplan try remotely — it applies a candidate and reverts unless confirmed within the timeout (Netplan try).
• NetworkManager: prefer a supported nmcli device checkpoint around the actual activation; it automatically restores previous configuration if not confirmed before the timeout (nmcli) — checkpoint and netplan try are not interchangeable and protect neither firewall nor cloud config.

SSH change sequence. Determine the active unit first (systemctl status ssh.service sshd.service) and whether configuration is included from sshd_config.d — distribution packaging differs, and unit names differ (Ubuntu: OpenSSH server). Then:

# [MUTATING · root] — keep the original session open throughout
sudo cp -a /etc/ssh/sshd_config /root/sshd_config.prechange
sudo sshd -t          # validate the edit
sudo sshd -T          # inspect effective settings (use -C for Match context)
sudo systemctl reload <the-actual-unit>   # reload, not an assumed name

Then open a fresh SSH session and do not close the original until the new one authenticates with the intended privilege. Changing PermitRootLogin, PasswordAuthentication, AllowUsers, ListenAddress, port, host keys, or firewall source allowlists carries direct lockout risk (sshd_config(5)).

Firewall change sequences. First determine the owner and the full packet path — cloud/provider security group, router ACL, host manager, container/namespace, IPv4/IPv6 — and check ingress and return-path rules (firewall-cmd; nftables quick reference).

• firewalld: firewalld separates runtime from permanent configuration; --reload replaces runtime with permanent state and discards runtime-only changes; --complete-reload may terminate active connections; runtime additions can carry an expiring --timeout (not combinable with --permanent). Workflow: add the narrow rule at runtime (optionally --timeout), test from the target client, then add the permanent equivalent and verify with --permanent --list-all (firewall-cmd; runtime vs permanent).
• nftables: back up the active ruleset and its persistent source; validate with nft -c -f; arm detached rollback; apply; verify counters and a fresh inbound session from a non-allowlisted vantage. Base-chain policy drop discards packets not accepted by rules. nft -c proves syntax/semantics, not reachability or correct policy (nft(8); nftables wiki).
• UFW: inspect status and numbered rules; use UFW's own commands when it owns policy — do not edit its underlying nftables/iptables rules directly. Ensure the SSH allow rule is in effect before enabling a restrictive policy; changing default incoming policy over the only SSH path can immediately sever access (Ubuntu: Firewall).

Network change sequence. Prefer the native rollback facilities (Netplan try, NM checkpoint). Change one item at a time — default gateway, address/prefix, DNS, VLAN, bond, bridge, MTU, routing policy, VPN — and confirm the manager being edited actually owns the link. After activation, check ip -br address, ip route get ADMIN_CLIENT_IP, DNS status, and a fresh connection from the administrator's actual source; a locally correct route does not guarantee upstream return routing. Root is generally required to persist or activate host network changes, and DHCP/profile changes, resolver restarts, and interface reconfiguration can interrupt all active sessions even when syntax is valid (Netplan try; nmcli).