How the tutorial was built

The organizers' half: preparing cards, writing solutions, generating the participant material. To just do the exercises, see Run it yourself.

Two rules shape everything: participants never touch a machine they own, and organizers never hand-edit anything a participant sees.

The material is generated

Nothing participants receive is written by hand — it is derived from the solutions, so the two cannot drift:

doca_flow_solution.c     the working program (answer key)
        |  regen_templates.py
        v
doca_flow_template.c     same file, three function bodies hollowed to TODOs
        |  update_participants_repo_on_github.py   (ships it as doca_flow_ecn.c)
        v
participants repo        no solutions anywhere in it

PCC works identically: make_pcc_exercise.py strips the two rate reactions from the finished controller. Both generators have --check, which fails if a scaffold has drifted — run it before handing anything out.

Script	Derives
regen_templates.py	DOCA Flow scaffold, from the solution
make_pcc_exercise.py	DOCA PCC scaffold, from the controller
update_participants_repo_on_github.py	The participant repo, and pushes it

The fleet

Twelve cards, five sites, none identical. admin/fleet.py drives them over SSH by Tailscale name. It is transport and aggregation only — each verb ships a script from admin/local_scripts/ over stdin and tabulates the results, so any unreachable card can be driven by hand with the same script.

./admin/fleet.py doca | firmware | links   # what is each card running
./admin/fleet.py pcc-ready                 # are the PCC knobs live, staged, or unset
./admin/fleet.py deps --install            # install what is missing
./admin/fleet.py test-tutorial HOST        # clean build + full exercise, pass/fail

Three things worth knowing:

• setup_roce_loopback.sh imposes a layout rather than adapting — it wipes every bridge and SF, then rebuilds three SFs, three namespaces, bridges and hugepages, asserting each before exit. RDMA device indices are assigned in probe order, so starting from zero is the only way to reliably land on mlx5_2/mlx5_3, which the guides hardcode. None of it survives a reboot.
• USER_PROGRAMMABLE_CC=1 needs a genuine power-cycle to go live; the BlueField-3 has its own power rail, so a reboot will not do it.
• test-tutorial is the readiness check: clean build, loopback, traffic, ECN marking confirmed in a capture, PCC, rate collapse.

Getting participants onto the cards

The cards live in racks at five sites and are not reachable from the internet, so participants joined a Tailscale tailnet — laptop and SmartNIC on one small private network, no account needed, no sign-in, just a pre-approved key. Keys were issued reusable, ephemeral and tagged, so laptops removed themselves from the tailnet when they went offline instead of leaving forty stale nodes behind. SSH opened at the start of the session and closed at the end of the conference.

The guide sent out ahead of time is kept as Giving participants access. fleet.py addresses cards by their Tailscale names too, which is why there is no ssh_config or address list anywhere in the repo.

Building the handouts

What	Built by
Guide PDFs	guides/Makefile — pandoc + xelatex. make check catches dropped glyphs and overfull lines, which a successful build still produces.
Figures	docs/Makefile — Graphviz at 300 DPI
Hands-on deck	slides/build.py — one self-contained HTML file
This site	site/ — MkDocs. ./site/serve.sh previews locally; ./site/publish.sh deploys. Guides are staged and rewritten at build time, never edited in place.

The record

The README has the long form: PCIe and SF layout, why SF MACs are derived not fixed, why the loopback must not flood, the firmware procedure, and the findings that cost time — including a gotcha where enabling PCC appears to disable the NIC's stock DCQCN, left recorded as unresolved.

Read it before touching a card.