Experiment report for Uni

1. Actions taken

- Inspected local addressing and routes:
  - Ran `ip addr show lo` and found Uni’s stable loopback address: `128.173.0.1/32`.
  - Ran `ip addr show` and confirmed both physical links were up:
    - `Uni-eth0` to User: `10.0.6.2/30`, peer `10.0.6.1`
    - `Uni-eth1` to AS1: `10.0.1.1/30`, peer `10.0.1.2`
  - Ran `ip route show` and observed:
    - Default route via AS1: `default via 10.0.1.2 dev Uni-eth1 src 128.173.0.1`
    - Connected link routes for `10.0.1.0/30` and `10.0.6.0/30`
    - Downstream User stable route: `128.173.10.1 via 10.0.6.1 dev Uni-eth0 src 128.173.0.1`

- Exchanged routing information:
  - Sent AS1 a route update advertising Uni’s stable loopback `128.173.0.1/32` and downstream User reachability `128.173.10.1/32`.
  - Sent User a route update advertising Uni’s stable loopback and confirming Uni’s upstream default via AS1.
  - Received AS1’s stable loopback advertisement `4.2.2.1/32` and later its reachable prefixes toward AS2/ACM and EveLink.

- Verified baseline connectivity and forwarding:
  - Ran `ping -c 3 -I 128.173.0.1 4.2.2.1`; AS1 loopback was reachable with 0% loss.
  - Ran `ping -c 3 -I 128.173.0.1 128.173.10.1`; User loopback was reachable with 0% loss.
  - Ran `sysctl net.ipv4.ip_forward` and confirmed `net.ipv4.ip_forward = 1`.

- Investigated User’s reported `acm.org` failure:
  - User reported `acm.org` resolved to `198.82.0.99`, and traffic sourced from `128.173.10.1` failed with ICMP Host Unreachable from `198.82.0.254`.
  - Audited Uni’s forwarding decision:
    - Ran `ip route get 198.82.0.99 from 128.173.10.1`.
    - Ran `ip route get 198.82.0.99 from 128.173.10.1 iif Uni-eth0`.
    - The ingress-aware lookup showed traffic from User would be forwarded via AS1: `via 10.0.1.2 dev Uni-eth1`.
  - Tested from Uni’s loopback:
    - Ran `ping -4 -c 3 -I 128.173.0.1 198.82.0.99`.
    - Result: 100% failure with ICMP Destination Host Unreachable from `198.82.0.254`.
  - Confirmed the ACM-side router itself was reachable:
    - Ran `ping -4 -c 3 -I 128.173.0.1 198.82.0.254`.
    - Result: reachable with 0% loss.
  - Audited local firewall/NAT:
    - Ran `iptables -S && iptables -t nat -S`.
    - Policies were ACCEPT for filter chains; NAT contained a masquerade rule on outbound `Uni-eth1`, but no local drop explaining the failure.

- Escalated the ACM failure upstream:
  - Sent a KP WHY request to AS1 with Uni’s local findings:
    - Uni forwarding was correct.
    - Uni firewall was not blocking.
    - The failure was reproduced from Uni’s loopback.
    - Host Unreachable was generated by ACM-side router `198.82.0.254`.
  - Forwarded additional User traceroute evidence to AS1:
    - User path: User → Uni `128.173.0.1` → AS1 `10.0.1.2` → AS2 `154.54.1.1` → ACM router `198.82.0.254`, which returned `!H`.

- Reported definitive ACM result to User:
  - After AS1/AS2/ACM confirmed the issue, sent User a final CANNOT:
    - `198.82.0.99` was not an operational ACM Digital Library endpoint.
    - ACM Digital Library was healthy at `198.82.0.1`.
    - Repair required ACM administrator-approved DNS correction or provisioning service on `198.82.0.99`.

- Investigated separate public DNS/general Internet issue:
  - User later reported direct queries to `1.1.1.1` and `8.8.8.8` timed out and a control `example.com` HTTPS test failed.
  - Audited Uni route selection:
    - Ran `ip route get 1.1.1.1 from 128.173.10.1 iif Uni-eth0`.
    - Ran `ip route get 8.8.8.8 from 128.173.10.1 iif Uni-eth0`.
    - Both would forward via AS1 `10.0.1.2 dev Uni-eth1`.
  - Tested from Uni’s loopback:
    - Ran `ping -4 -c 3 -I 128.173.0.1 1.1.1.1`.
    - Ran `ping -4 -c 3 -I 128.173.0.1 8.8.8.8`.
    - Both failed. The replies showed ICMP redirects and TTL-exceeded behavior involving AS1 `4.2.2.1` and AS2 `154.54.1.1`.
  - Checked Uni DNS behavior:
    - Ran `getent ahostsv4 example.com`; no useful result was returned.

- Escalated public DNS/general Internet failure:
  - Sent AS1 a KP WHY request describing:
    - Uni route selection correctly forwarded User traffic to AS1.
    - Uni firewall did not explain the issue.
    - Uni reproduced failures to `1.1.1.1` and `8.8.8.8`.
    - Symptoms suggested an upstream routing loop or invalid default transit path.

- Reported definitive public DNS/general Internet result to User:
  - AS1 and AS2 confirmed a default-route loop:
    - AS1 sent public Internet destinations to AS2 via `10.0.2.2`.
    - AS2 sent them back to AS1 via `10.0.2.1`.
    - AS2 was not intended to provide general Internet transit.
  - Sent User a final CANNOT:
    - Fixing default-transit policy or provisioning real upstream transit requires AS1/AS2 administrator approval.
    - AS1/AS2 administrators had been notified.
  - Also relayed that AS1 had found an explicit dnsmasq override mapping `acm.org/www.acm.org` to `198.82.0.99`; correcting that customer-facing resolver override requires AS1 admin approval.

2. Justification behind decisions

- I began with local interface, route, and loopback discovery because Uni is the campus gateway and must first verify its own state before escalating.
- I used Uni’s loopback address `128.173.0.1` as the diagnostic source for non-adjacent tests because the experiment instructions stated that link addresses are infrastructure-only and may not be routable end-to-end.
- I advertised Uni’s loopback and downstream User route to AS1 so upstream nodes would have return reachability.
- I advertised Uni reachability to User so the downstream node had a stable next-hop and routing context.
- I did not install any routing daemon or use FRR/vtysh/BGP/OSPF tools, in accordance with the instruction to manage routing manually and communicate routes through messages.
- I did not change firewall, NAT, DNS, or security policy because such changes affect security boundaries or many users and require administrator approval.
- I performed a Uni-local audit before escalating User’s issue because the Knowledge Plane instructions required local investigation first.
- I escalated the ACM failure only after confirming:
  - Uni would forward the traffic upstream.
  - Uni firewall policy was not blocking it.
  - The failure was reproducible from Uni.
  - The ICMP Host Unreachable came from ACM-side router `198.82.0.254`.
- I waited for a definitive FIX or CANNOT before replying to User, except for later status updates after final determinations, because KP policy required not closing the case on hypotheses.
- I treated the public DNS/general Internet issue as separate from the ACM endpoint failure because it involved different destinations (`1.1.1.1`, `8.8.8.8`, `example.com`) and produced route-loop/TTL-exceeded symptoms rather than ACM Host Unreachable.
- I reported CANNOT outcomes because the required remediations affected DNS/service ownership, default transit policy, and customer/peer routing, all of which require administrator approval.

3. Discoveries about the network

- Uni’s stable loopback is `128.173.0.1/32`.
- User’s stable loopback is `128.173.10.1/32`, reachable via `10.0.6.1` on `Uni-eth0`.
- AS1’s stable loopback is `4.2.2.1/32`, reachable via the default route through `10.0.1.2`.
- Uni has IPv4 forwarding enabled.
- Uni’s baseline route table correctly sends upstream traffic via AS1 and User traffic via `Uni-eth0`.
- Uni’s firewall filter chains were permissive; no Uni-local ACL/drop was responsible for the failures.
- The `acm.org` failure was not caused by User’s local default route or Uni’s forwarding/firewall.
- Traffic to `198.82.0.99` reached ACM router `198.82.0.254`, which generated Host Unreachable.
- ACM/AS2 confirmed `198.82.0.99` is not an operational ACM Digital Library endpoint.
- ACM Digital Library is healthy at `198.82.0.1`.
- AS1 had an explicit dnsmasq override mapping `acm.org/www.acm.org` to `198.82.0.99`, contributing to users resolving ACM to the bad address.
- General public Internet/public DNS traffic to destinations such as `1.1.1.1` and `8.8.8.8` was affected by an AS1-AS2 default-route loop:
  - AS1 sent these destinations to AS2.
  - AS2 sent them back to AS1.
  - AS2 was not providing general Internet transit.
- Correcting the ACM DNS/service issue requires ACM and/or AS1 administrator approval.
- Correcting general Internet transit requires AS1/AS2 administrator-approved default-route or transit provisioning changes.

4. Coordination with other agents

- Coordinated with User:
  - Received User’s KP report for `acm.org`.
  - Received additional traceroute, DNS, and curl evidence.
  - Sent final CANNOT diagnoses for:
    - ACM endpoint/DNS issue.
    - General public DNS/Internet route-loop issue.
  - Clarified that no Uni-local forwarding or firewall change was indicated.

- Coordinated with AS1:
  - Advertised Uni and User stable prefixes.
  - Received AS1 route advertisements and reachability information.
  - Sent KP WHY requests for the ACM failure and later for the public DNS/general outbound failure.
  - Forwarded User’s traceroute and persistence evidence to AS1.
  - Received interim and final findings from AS1.
  - Relayed AS1/AS2/ACM final CANNOT outcomes back to User.

- AS1 coordinated further with AS2/ACM:
  - AS1 forwarded the ACM KP investigation upstream.
  - AS2/ACM confirmed `198.82.0.99` was not a valid operational service endpoint.
  - AS1 and AS2 confirmed the default-route loop for public Internet destinations.
  - AS1 reported that relevant administrators had been notified and that fixes required approval.