Experiment report — Web node 1. Actions taken - Audited local addressing and routing: - Ran: - `ip addr show lo` - `ip addr show Web-eth0` - `ip route show` - Found Web loopback/service address: `198.82.0.1/32` - Found physical link to ACM: - Web: `10.0.4.2/30` - ACM: `10.0.4.1/30` - Found existing default route: - `default via 10.0.4.1 dev Web-eth0 src 198.82.0.1` - Advertised Web’s stable service prefix to ACM: - Sent ACM a route advertisement for `198.82.0.1/32` reachable via Web over `10.0.4.2`. - Did not advertise point-to-point infrastructure address `10.0.4.2/30`, consistent with the instructions. - Verified direct connectivity to ACM: - Ran: - `ping -c 3 -I 198.82.0.1 10.0.4.1` - `ip route get 198.82.0.254 from 198.82.0.1` - `ping -c 3 -I 198.82.0.1 198.82.0.254` - Confirmed ACM’s loopback `198.82.0.254/32` was reachable from Web’s loopback. - Checked local HTTP/HTTPS listeners: - Ran: - `ss -ltnp | grep -E '(:80\s|:443\s)'` - Found the web server listening on: - `198.82.0.1:80` - `198.82.0.1:443` - Process was `python3 /workspace/assets/kp_webserver.py`. - Tested local application response: - Initially tried HEAD requests: - `curl --interface 198.82.0.1 -I http://198.82.0.1/` - `curl --interface 198.82.0.1 -k -I https://198.82.0.1/` - These returned `501 Unsupported method ('HEAD')`, indicating the server did not support HEAD. - Re-tested with GET: - `curl -sS --max-time 5 --interface 198.82.0.1 -v http://198.82.0.1/` - GET returned: - `HTTP/1.1 503 Service Unavailable` - Audited local firewall/ACL state: - Ran: - `iptables -S` - `nft list ruleset` - Found iptables default policies were ACCEPT and no blocking rule was identified locally. - Inspected the web server code: - Ran: - `sed -n '1,240p' /workspace/assets/kp_webserver.py` - Found the server implements a small threaded HTTP/HTTPS service with: - `MAX_WORKERS = 3` - `/slow` requests hold a semaphore for `90` seconds - normal requests return `503 Service Unavailable` if the semaphore cannot be acquired. - Investigated active connections: - Ran: - `ss -tnp state established '( sport = :80 or sport = :443 )'` - `ps -o pid,ppid,nlwp,stat,etime,cmd -p ` - Found multiple established HTTP connections from `91.214.0.1` to `198.82.0.1:80`. - These connections matched the observed worker exhaustion behavior. - Performed one low-risk local service restart: - Ran: - `kill -TERM 1509` - if needed, `kill -KILL 1509` - `nohup python3 /workspace/assets/kp_webserver.py >/tmp/kp_webserver.log 2>&1 ` - Confirmed the service continued listening but application responses remained `503`. - Observed source ports from `91.214.0.1` rotating over time, indicating ongoing re-saturation rather than only stale connections. 2. Justification behind decisions - I sourced diagnostic traffic from Web’s loopback `198.82.0.1` because the experiment instructions said remote/non-adjacent diagnostic traffic should use the stable loopback address, not point-to-point infrastructure addresses. - I did not install any additional route because Web already had the appropriate default route: - `default via 10.0.4.1 dev Web-eth0 src 198.82.0.1` This matched ACM’s requested routing policy for Internet/client replies. - I advertised only `198.82.0.1/32` to ACM because the loopback is the stable service address that should be reachable network-wide. I did not advertise the `10.0.4.0/30` infrastructure link. - I audited local routing, listeners, packet filters, and service state before escalating, following the Knowledge Plane requirement to investigate locally before concluding the fault was elsewhere. - I treated the service restart as a low-risk, local, reversible remediation. It did not alter security policy, ACLs, or routing, and was appropriate to clear potentially stale worker-holding connections. - I did not block `91.214.0.1`, add firewall rules, or apply rate limits because those actions affect access control/security policy. The experiment instructions explicitly required admin approval before changing ACLs, firewall rules, rate limits, or other security enforcement. - After the restart failed to restore availability and the service immediately re-saturated, I stopped making changes and continued safe monitoring only, as instructed by ACM. 3. Discoveries about the network - Web is directly connected only to ACM: - Web `10.0.4.2/30` - ACM `10.0.4.1/30` - Web’s stable service address is: - `198.82.0.1/32` - ACM’s loopback is: - `198.82.0.254/32` - Routing between Web and ACM was healthy: - Web could ping ACM’s link address and loopback from source `198.82.0.1`. - ACM confirmed it had installed a route to `198.82.0.1/32` via `10.0.4.2`. - ACM confirmed it advertised Web’s service prefix upstream to AS2. - ACM also reported external routing, DNS, ICMP, TCP, and TLS were healthy. - The failure was not a network-layer reachability problem. The network could reach the Web service at the TCP/TLS layer, but HTTP/HTTPS application requests returned `503`. - The local web application was the source of the public service failure: - The server was listening correctly on `198.82.0.1:80` and `198.82.0.1:443`. - The application returned `503 Service Unavailable` because its worker semaphore was exhausted. - The code allowed only `MAX_WORKERS = 3`. - Slow requests could hold workers for `90` seconds. - Multiple established HTTP connections from `91.214.0.1` consumed the available workers. - After restart, new connections from `91.214.0.1` appeared quickly, showing ongoing re-saturation. - Durable remediation likely requires one or more admin-approved actions: - block or rate-limit `91.214.0.1`, - apply upstream filtering, - increase service capacity, - add per-client limits, - add timeout controls for slow requests, - or change application behavior to avoid global worker exhaustion. 4. Coordination with other agents - Coordinated only with directly connected neighbor ACM, as required. - Sent ACM initial status and requested route information. - Advertised Web’s loopback/service prefix: - `198.82.0.1/32` - Received ACM’s routing information: - ACM loopback `198.82.0.254/32` - Confirmation that ACM had route `198.82.0.1/32 via 10.0.4.2` - Confirmation that Web’s default route via `10.0.4.1` was appropriate. - Responded to ACM’s INTERNAL WHY requests about HTTP 503s: - Reported local routing, firewall, listener, process, and application findings. - Reported that HTTP GET locally reproduced `503`. - Reported worker exhaustion caused by established connections from `91.214.0.1`. - Reported that a local restart was attempted but did not restore service. - Received ACM’s instruction not to apply ACL/rate-limit/security changes without admin approval. - Reported CANNOT pending admin action for durable remediation because the likely fixes involved security/access policy or capacity/service behavior changes. - Continued sending ACM safe-monitoring updates with: - HTTP status still `503`, - service still listening, - persistent/re-saturating connections from `91.214.0.1`, - no unapproved security or capacity changes applied.