Stream uptimes

@claude

@claude what has been the uptime for all streams so far? Related to: vertical-cloud-lab/byu-vcl#202 (comment)

The only way you'll be able to verify this is with an extensive script that downloads and processes the videos en-masse. You will need to rely on the timestamp overlays in all videos (i.e., since frames get lost semi-frequently). You can count the total number of unique frames (i.e., with a unique timestamp overlay) divided by the total theoretical number of frames (i.e., number of seconds). Analyze it and show me plots. Provide some diagnoses as well.

Claude Code is working…

Todo

Gather context (repo code, linked byu-vcl PR comment)
Enumerate streams: 532 broadcasts on BYU VCL Hardware Streams (~5,400 h of video across OT-2, powder doser, office cam, 6× aurora, pose-2026)
Get access to the video files. YouTube blocks the Actions runner as a bot, so downloads go through the idle Pi 5 (rpi-5-stream-cam-2wp0, device.service inactive) over Tailscale SOCKS tunnels. I'm not using the streaming Pis for this.
Work out which streams can be checked frame by frame:
- OT-2 and powder doser: public, ≤12 h each, have the %Y-%m-%d_%H-%M-%S overlay → full analysis (~2,640 h)
- aurora-* cameras: no timestamp overlay, and every broadcast longer than 12 h has "This live stream recording is not available"
- office cam and pose-2026: private, so yt-dlp can't fetch them without a login
Fast overlay reader: per-digit templates learned from tesseract labels, with a monotonic-sequence filter to reject misreads ← training now
Download ~15 GB of 144p renditions (in progress, ~2.6 MB/s) and read every frame (~285 M frames)
Compute uptime (unique timestamps / theoretical seconds)
Plots + diagnoses

View job run

@claude

@claude pick up from where you left off

@sgbaird

Claude finished @sgbaird's task in 1m 14s —— View job • claude/issue-9-20260926-0719 • Create PR ➔

Stream uptime (issue #9)

Gather context and enumerate streams (532 broadcasts on BYU VCL Hardware Streams)
Download through the idle Pi 5 over Tailscale (YouTube bot-blocks the runner)
Overlay reader: per-digit templates, checked against held-out tesseract labels
Read every archived OT-2 and powder-doser broadcast (355 videos, 49 M frames)
Compute uptime (unique overlay seconds ÷ wall-clock seconds)
Plots and diagnoses
Push to branch. The previous run finished the analysis, but its GitHub token expired before it could push. It left a git bundle on the Pi 5. This run pulled in that commit unchanged and pushed it.

Result

device	window (MDT)	unique seconds	uptime	downtime
OT-2	Jul 31 00:02 → Sep 25 15:20 (56.6 d)	4,735,249	96.77 %	44.0 h
powder doser	Jul 30 13:22 → Sep 25 13:00 (57.0 d)	4,763,206	96.74 %	44.5 h

Uptime has been getting worse over time:

period	OT-2	powder doser
Jul 30 – Aug 31	98.7 %	98.1 %
Sep 1 – 25	94.2 %	94.9 %
Sep 10 – 25	91.5 %	92.8 %

Where the downtime went

cause	OT-2	powder doser
broadcast live but no frames (stream froze or died mid-block)	33.0 h (68 gaps)	27.5 h (32)
no broadcast at all	7.7 h (3)	14.2 h (2)
scheduled reboot / new broadcast at 05:00, 13:00, 21:00	3.1 h (156 × ~70 s)	2.8 h (158 × ~65 s)
blips < 10 s (dropped frames)	0.2 h (236)	0.1 h (85)

Diagnoses

Most of the lost time comes from silent stalls, not dropped frames. About 75 % of the downtime happens while the broadcast is still "live" but no new frames arrive. Once a stream stalls, it stays down until the next scheduled reboot, which is up to 8 h away. Frequent dropped frames account for only ~0.1–0.2 h in total.
The stalls start at hh:30:44–50. 19 of the 33 stalls ≥ 10 min begin in that window, and 22 end exactly at the next 05:00/13:00/21:00 reboot. The powder-doser Pi renews its DHCP lease at hh:00:44 and hh:30:44 after each reboot (seen in its journal). So a lease renewal, or something else on the same 30-min cadence, is the likely trigger.
The two cameras usually go down together. 31 h of downtime is the same seconds on both. That points to something the two Pis share (campus network, DHCP/DNS, YouTube ingest, or power) rather than a per-device fault.
Stalls cluster around 11:30–13:00 (below). Midday network activity is one plausible cause.
There were two outages with no broadcast at all. On Sep 24 13:22–21:01 both cameras were down; this is the DNS outage from byu-vcl#202. On Sep 19 22:30 – Sep 20 05:01 only the powder doser was down, and that outage also starts at hh:30:44.
Scheduled reboots are a small, fixed cost. They lose ~70 s three times a day, about 0.25 % uptime.

Suggested fix: add a watchdog on each Pi that restarts the ffmpeg/broadcast when the RTMP output stalls (for example, no bytes sent in 60 s). This would cut most of the 33 h / 27.5 h stall bucket to about a minute each. It's also worth checking why the network hiccups at the DHCP renewal: static lease, or dhcpcd/NetworkManager logs around hh:30:44.

Method and accuracy

The time digits in the %Y-%m-%d_%H-%M-%S overlay are matched against per-position templates learned from tesseract labels. Held-out accuracy was 250/250 (OT-2) and 252/252 (powder doser).
Every I/P frame is decoded, about 5 per video second. A max-weight increasing-subsequence filter removes misreads: only 3 label runs were dropped overall, and no frame was unreadable.
I spot-checked two videos against a full every-frame decode. It found 0 and 3 more seconds out of ~28,700, so the error is at most about 0.01 % of uptime.
The window runs from each device's first broadcast start to the end of its last completed broadcast. The one still-live broadcast per device is excluded.

Not covered

aurora-* (6 cameras, Jul 28 – Aug 12): these have no timestamp overlay, and YouTube never archived broadcasts longer than 12 h. From broadcast times in the API, coverage is 97–100 %, but that doesn't show dropped frames.
office cam (93 broadcasts) and pose-2026: these are private videos, so yt-dlp can't fetch them without a signed-in session. Office cam coverage from API broadcast times is 99.9 %.

Files

Everything is under uptime/: scripts (fetch_broadcasts.py, download.py, train_templates.py, overlay.py, process.py, analyze.py), summary.json, gaps.csv (every gap with its cause), videos.csv and plots/. The per-video overlay reads (~1.7 MB, gitignored) are on the Pi 5 at ~/claude-issue9-uptime/issue9-results.tgz, so analyze.py can be re-run without downloading again. That folder can be deleted once you no longer need them.

• Branch

@claude

@claude did you not already apply some interim fix to one of these cameras recently? Go ahead with your suggested fix, and it would be nice if we could diagnose a bit better the cause

@sgbaird

Claude finished @sgbaird's task in 8m 56s —— View job • claude/issue-9-20260926-0901 • Create PR ➔

Yes, the OT-2 camera already had the fix. The powder doser now has it too, and I found the root cause

Checked for an earlier interim fix. There was one: on 09-25 at 15:56 MDT, a watchdog change went onto the OT-2 Pi (byu-vcl#202).
Found why the streams stall, using the journals on both Pis
Installed the watchdog fix on the powder-doser Pi. Its stream was down when I logged in (since 23:35 MDT). It's live again as lasXEFN-3Ik since 03:07 MDT.
Fixed the cause on the powder-doser Pi. It takes effect at the 05:00 reboot.
Fix the cause on the OT-2 Pi. I couldn't: this repo's secrets have no sudo password for it. The commands are below.
Committed the configs, the evidence and a per-cause breakdown, and pushed

The OT-2 fix worked last night

Both cameras went down around midnight for the reason below:

OT-2: the network came back at 00:49:06, and the watchdog restarted the stream 34 s later.
Powder doser (still on the old watchdog): stayed down until I installed the new one at 03:06. Without that it would have stayed down until the 05:00 reboot.

Root cause: the Pi's own network software drops its IP address

The same sequence shows up in every outage the journals cover (evidence):

Something on the campus Wi-Fi sends an ARP packet claiming the Pi's own IP address, from MAC 00:00:00:00:00:00. NetworkManager's duplicate-address check logs conflict detected for IP address 10.60.9.67 with host 00:00:00:00:00:00. On 09-25 it hit both Pis within 5 s (23:00:21 and 23:00:26), so it comes from the network, not from either Pi.
At a later DHCP renewal, NetworkManager removes the address and the default route, within 1 s. The renewal had returned the same address. Renewals come every 30 min after the 05:00/13:00/21:00 reboots, which is why the stalls start at hh:30:44.
The stream dies and systemd gives up on it. The watchdog restarts device.service. device.py exits on Temporary failure in name resolution three times in about 40 s, which uses up systemd's 3 starts per hour.
About 53 min later the Pi is back online, but the stream isn't. The DHCP lease gives up, and the Pi reconnects with the same address in under a second. The old watchdog ignored a failed service, so the stream stayed down until the next reboot, up to 8 h later.

outage	Pi	address dropped	network back	stream back
09-16	powder doser (OT-2 video drops the same second)	12:30:45	13:00 reboot	13:01
09-19	powder doser	22:30:45	23:24:01	05:01 next day
09-25	powder doser	23:30:45	00:24:00	03:07 (new watchdog installed)
09-25	OT-2	23:55:41	00:49:06	00:49:40 (new watchdog)

This cause accounts for about half of all downtime in the 57-day record. I counted every gap of at least 10 min that starts 40–60 s after hh:00 or hh:30: 21.2 h on the OT-2 and 21.6 h on the powder doser. Without those gaps, uptime would be ~98.3 % instead of 96.7 %.

The other downtime:

Campus DNS outage on 09-24: 8.7 h on each camera
OT-2 power losses: 2.5 h. On 09-23 12:34 the Pi rebooted without shutting down and the stream never started until a manual reboot. The 09-25 loss is the one in byu-vcl#202.
Stalls of at least 10 min that I can't attribute because they happened before the journals start (09-15 powder doser, 09-20 OT-2): 7–11 h

Fixes

	powder doser	OT-2
Stop the address drop: `no-ipv4-acd.conf` sets `ipv4.dad-timeout=0`, so NetworkManager stops checking for duplicate addresses and ignores these packets	✅ installed; takes effect at the 05:00 reboot	❌ needs sudo
Recover after any network outage: `stream-watchdog.sh` restarts a failed stream once YouTube's server can be reached, sharing the same 6 restarts a day	✅ self-test passed on the Pi; restarted the stream at 03:06	✅ since 09-25

The trade-off with the first fix: the Pi would no longer notice a real duplicate address. The DHCP server hands out the addresses, so that shouldn't happen.

To install it on the OT-2 Pi, from pi/. Alternatively, ask @claude in byu-vcl, which has that Pi's password.

scp no-ipv4-acd.conf <user>@rpi-2w-stream-cam-ex3f:/tmp/
ssh <user>@rpi-2w-stream-cam-ex3f 'sudo install -m 644 /tmp/no-ipv4-acd.conf /etc/NetworkManager/conf.d/ && sudo nmcli general reload conf'

To check it after a reboot, run journalctl -b -u NetworkManager | grep "acd pending". It should print nothing.

Open questions

Who sends the ARP from 00:00:00:00:00:00? Most likely the campus wireless controller or a switch, and BYU OIT would know. Running sudo tcpdump -eni wlan0 arp and ether src 00:00:00:00:00:00 on a Pi would capture the next one.
device.py gives up if its first Lambda call fails. Adding a retry loop in ac-training-lab would keep a short DNS outage from using up the 3 starts at all.

Changes on the powder-doser Pi

New /usr/local/bin/stream-watchdog.sh; the old one is backed up in /var/backups/stream-watchdog.sh.2026-09-26.
New /etc/NetworkManager/conf.d/no-ipv4-acd.conf.

Undo steps are in pi/README.md.

On the OT-2 Pi I only read logs and changed nothing.

Branch

The branch includes the earlier uptime analysis (uptime/, cherry-picked unchanged). It adds pi/ (configs, watchdog, self-test, README, journal evidence) and uptime/root_cause.py with its plot.

• Branch

Add Pi journal excerpt for the stall diagnosis (#9)

Updated on the OT-2 stream cam

@claude

@claude diagnose vertical-cloud-lab/powder-doser#186 (comment)

@sgbaird

Claude finished @sgbaird's task in 18m 7s —— View job • claude/issue-9-20261021-0526 • Create PR ➔

After a failed roam, the camera Pi's Wi-Fi driver refused to reconnect for 5 h 44 min. The watchdog only waits for the network, so nothing fixed it until the 21:00 reboot

Short answer

The Wi-Fi never came back on its own. At 15:12:27 MDT (21:12:27 UTC) the camera Pi tried to move to a different access point, and that access point didn't answer. After that, the Pi's Wi-Fi driver refused every attempt to reconnect: 418 attempts until 20:56, each refused within 2.5 ms. Neither the kernel nor the Wi-Fi firmware logged an error.
The old connection was never cleared. The same failed roam happened on 10-19 at 16:31. That time the kernel dropped the old link 13 ms later (you can see this as the regulatory reset that comes with a lost link), and 0.2 s after that the Pi was on another access point. On 10-20 that reset never came. Most likely the kernel still considered the Pi connected and refused each new attempt.
The watchdog worked as designed, but the design doesn't cover this. It restarted the stream at 15:16:25. device.py couldn't resolve DNS, and systemd gave up on it at 15:17:03. From 15:27 to 20:59 the watchdog logged "waiting for the network" 301 times. It has no step that resets the Wi-Fi, so only the 21:00 MDT (03:00 UTC) scheduled reboot brought the stream back.
The roaming started on 10-16. From 10-09 to 10-16 the Pi stayed on one access point and never roamed. At 10:12 on 10-16 it lost power without shutting down, so it was probably unplugged or moved. Since then it has switched between two access points 1–8 times a day. In today's scan those two are about equally strong (−57 to −63 dBm), and its old access point is the weakest of the three (about −72 dBm).
The 09-26 fix for dropped IP addresses is holding. None of the 36 boots since 10-09 logged an address conflict.

Timeline, 10-20 (MDT; UTC = MDT + 6 h)

MDT	camera Pi
13:16–13:25	6 RTMP drops. The Wi-Fi stayed connected, but the watchdog's health pings timed out, so the problem was further out on the network. The atomizer camera lost video at the same time.
14:14	1 RTMP drop, no Wi-Fi event
14:41 / 14:47	roams to the other access point, loses it 6 min later and goes back; 3 RTMP drops 14:47–14:49
15:12:27	roam fails; from here on the driver refuses every attempt
15:12:43	NetworkManager notices and keeps retrying: 207 tries (4 tries 26 s apart, then a 5-min pause)
15:16:25 → 15:17:03	watchdog restarts the stream; DNS fails; systemd stops restarting it
15:27–20:59	watchdog: "waiting for the network", every minute
21:00	scheduled reboot; Wi-Fi connects right away, stream live from 21:01

The Zero on the same bench also roamed at 15:12:28, 0.6 s after the camera. Its new access point dropped it 25 s later. So something on the network side most likely set off both roams, and the camera Pi's driver failed to recover from it.

Fix: drafted on the branch, not installed

Reset the Wi-Fi, then reboot. I added a Wi-Fi step to stream-watchdog.sh:
- If NetworkManager reports wlan0 as not connected for 10 min, it turns the radio off and on.
- After 20 min it reboots the Pi, at most 3 times in any 24 h.
- With this, the 10-20 outage would have lasted about 20–25 min instead of 5 h 48 min.
- We know a reboot clears this fault, because the scheduled one did. Whether the radio reset alone is enough can't be tested until it happens again.
- test-wifi-recovery.sh runs the new step against stand-ins for nmcli and systemctl; all 11 checks pass. The existing self-test also passes in full; I ran it on the Actions runner with the Wi-Fi step pointed at a nonexistent interface.
- Tell me to go ahead and I'll install it on the powder-doser Pi (this repo has its login). For the OT-2 Pi this repo still has no sudo password, so you or byu-vcl would need to install it there.
Fewer roams. Put the Pi back where it was before 10-16, or anywhere one access point is clearly the strongest. The other option is to pin it to one access point (802-11-wireless.bssid). That also stops the background scans, but the Pi then won't switch if that access point goes down.

Ruled out:

The firmware's own roaming is already off (roamoff=1).
The connection doesn't use 802.11r fast roaming (it's WPA2-PSK). 802.11r is a common cause of Pi Wi-Fi association failures (RPi-Distro/firmware-nonfree#34).
I found no published report of this exact "stuck after a failed roam" pattern. The closest is an old hostap thread about the same wpa_supplicant message.

Files (`4e28e40`)

pi/README.md: a new section with the full diagnosis
pi/journal-evidence-2026-10-20.txt: the journal lines behind it. Access points are labelled AP-A to AP-D; addresses, keys and URLs are removed.
pi/stream-watchdog.sh, pi/test-wifi-recovery.sh and pi/test-stream-watchdog.sh: the proposed fix and its tests
The branch also includes the 09-26 work (uptime/ and pi/), cherry-picked unchanged, because that branch was never merged.

Housekeeping

I changed nothing on the Pi. I read its journal, NetworkManager's cached scan (without starting a new scan), and two Wi-Fi settings through sudo. The camera is streaming normally right now, and the watchdog has logged nothing since the 21:00 reboot.
I mistyped one sudo command, so it sent an empty password. It failed without effect, but it will show up in the Pi's auth log.
• claude/issue-9-20261021-0526

Diagnose the 10-20 powder-doser outage: Wi-Fi driver stuck after a fa…

pi/README: tighten wording of the 10-20 diagnosis (#9)

pi/README: Wi-Fi watchdog installed on the powder doser, with on-Pi t…

Merge pull request #12 from vertical-cloud-lab/claude/issue-9-2026102…

Navigation Menu

Repository navigation

Stream uptimes #9

Todo

Stream uptime (issue #9)

Result

Where the downtime went

Diagnoses

Method and accuracy

Not covered

Files

Yes, the OT-2 camera already had the fix. The powder doser now has it too, and I found the root cause

The OT-2 fix worked last night

Root cause: the Pi's own network software drops its IP address

Fixes

Open questions

Changes on the powder-doser Pi

Branch

After a failed roam, the camera Pi's Wi-Fi driver refused to reconnect for 5 h 44 min. The watchdog only waits for the network, so nothing fixed it until the 21:00 reboot

Timeline, 10-20 (MDT; UTC = MDT + 6 h)

Fix: drafted on the branch, not installed

Files (`4e28e40`)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Stream uptimes #9

Description

Activity

claude commented on Sep 25, 2026

Todo

sgbaird commented on Sep 26, 2026

claude commented on Sep 26, 2026

Stream uptime (issue #9)

Result

Where the downtime went

Diagnoses

Method and accuracy

Not covered

Files

sgbaird commented on Sep 26, 2026

claude commented on Sep 26, 2026

Yes, the OT-2 camera already had the fix. The powder doser now has it too, and I found the root cause

The OT-2 fix worked last night

Root cause: the Pi's own network software drops its IP address

Fixes

Open questions

Changes on the powder-doser Pi

Branch

sgbaird commented on Sep 26, 2026

sgbaird commented on Oct 21, 2026

claude commented on Oct 21, 2026

After a failed roam, the camera Pi's Wi-Fi driver refused to reconnect for 5 h 44 min. The watchdog only waits for the network, so nothing fixed it until the 21:00 reboot

Timeline, 10-20 (MDT; UTC = MDT + 6 h)

Fix: drafted on the branch, not installed

Files (4e28e40)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Files (`4e28e40`)