Last time, this was a phone in my hand, walking a kitchen. This time the phone is bolted to an RC truck (the same one that was an obstacle in the kitchen video), and the truck drives itself. I call it the Robo Bump Truck.

Nothing about the method changed. That's the point of this log.

1 · What changed on the truck

It now carries a Jetson Orin Nano in a printed cradle, a Pixhawk flight controller running ArduRover, a RadioMaster MT12 transmitter talking to an ExpressLRS receiver, and an iPhone on a wedge mount looking down the trail. The phone's video camera is the only thing the model sees, and it's all the model uses to drive.

2 · Training: two models, no hand-drawn labels

The whole project rests on one idea: expensive sensors teach, cheap sensors ship. The phone carries LiDAR, motion tracking and an accelerometer. All of it is used to label the training images automatically. None of it is used to drive. The field calls this self-supervised learning, in a specific sense: the robot's own other sensors write the labels, and no human does. The training step itself is ordinary supervised learning against that answer key — the word "self" refers to who wrote the labels, not to how the model learns. The point of it, beyond the shipping argument, is time: every hour the sensors spend labelling is an hour a human doesn't. There are two models, and each has a different teacher. If you want to dig deeper, the obstacle model is in the lineage of Berkeley's BADGR, which labels images from the robot's own collision and bumpiness sensors; the steering model is in the lineage of ETH's Wild Visual Navigation, which labels images from the path the robot actually traversed.

2.1 · The obstacle model (DangerNet) — taught by bumps

This is Log #1's method. I walk up to each obstacle, and at arm's length I bump the phone body with my free hand — the phone never touches the obstacle. The accelerometer marks the moment; the LiDAR marks where the obstacle is; and that spot is projected back into every frame of the approach as "no-go." The floor I walk over in the next few seconds is projected back into those same frames as "go" (Figure 2). Thirty bumps became 147,731 labelled image patches.

2.2 · The steering model — taught by where the truck went

The model that drove the truck has a different teacher: the truck's own path. I drive a lap by hand while the phone records video and its motion-tracking pose. Afterwards, the path the truck actually took is projected back into every frame, and the image patches that path passes through are labelled "trail" (Figure 3). No LiDAR in this chain at all, and no human drew a line to teach it.

I did paint trail centrelines by hand on 53 frames — but to check the result, never to train it. On held-out data the model's centre landed about twice as close to my hand-painted centre as the pose labels it learned from.

2.3 · One backbone, two heads

Both models are the same shape (Figure 4): a frozen DINOv2 vision backbone that turns each image into a grid of 850 patch features, and on top of it a tiny logistic-regression head, one per model, that scores each patch. The obstacle head answers "must I not hit this?"; the steering head answers "is this trail?". Each head is 385 numbers and trains in a couple of minutes on a laptop. At runtime they see RGB pixels, nothing else.

Each head answers one yes/no question for every patch, independently, and the answer is a score from 0 to 1 (Figure 5). A heat map is just those 850 scores painted over the frame.

On this trip the obstacle model ran display-only. It was trained handheld at walking height, and at truck height on a trail it's out of its depth — literally. It gets retrained from the truck next. The steering model is what drove. Which expensive sensor does the teaching depends on which model you mean (Figure 6), and neither one keeps it.

3 · Live inference: getting the picture to the robot, fast

3.1 · The robot's eye is a phone, and it talks over one cable

The video gets from the phone to the Orin over a single USB cable. No capture card, no wifi, no radio link for the video. That sounds unremarkable until you try it. The published failure reports from people trying this describe the phone never appearing at all: usbmuxd #281, filed on the same phone and iOS version I'm using, and record3d #120. It works here because the Orin's usbmuxd is started with preflight disabled: the device enumerates and a tunnel binds before the pairing handshake completes, which is the whole bypass.

Once it worked, I wanted to know whether it kept working, because the neighbouring reports describe setups that connect and then die: usbmuxd #215, "it works fine for a random amount of minutes, until it dies", and record3d #72, a robot streaming from an iPhone over USB with "variable lag in the video stream (1-3 seconds)", open since 2023. So, a soak test. Thirty minutes continuous:

That was indoors, on a bench, in a cool room. Outside is a hotter place, and other people's phones have overheated doing this. Hold that thought — the field had opinions about it.

3.2 · How long does the robot take to see something?

This matters because it sets the floor under everything downstream. The loop has three stages: perception (what am I looking at?), control (what should I do about it?) and actuation (move the steering servo). This delay sits in front of all three. Control engineers call it dead time: however good the rest of the loop is, it is acting on a world that has already moved on. At 1 m/s, 60 ms is 6 cm of travel before the robot sees anything at all. Dead time is also what turns a confident controller into an oscillating one, so I needed the number.

The obvious approach doesn't work. You can't ask the phone "when did you see this?" The phone's clock and the robot's clock are different clocks, and the question spans both.

Here's the intuition, and it's a yodel. You're alone on one side of a valley and you want to know how long your voice takes to get across it. There's nobody over there to shout back. So you rig a billboard on the far side that flashes the instant it hears you, you yodel "Yo!", and you run a stopwatch until you see the flash. No partner required: you made something happen at a moment you knew, and you timed the consequence coming back.

(The yodel is for human intuition only. Nothing in the real rig makes a sound: light crosses a valley far too fast to time with a stopwatch, which is exactly why the real measurement is all visual, and why the slow thing being measured is the camera pipeline rather than any distance.)

So the robot rigs the same trick with light, on a billboard it controls (Figure 7). It serves a web page that flashes black to white on a schedule set by its own clock; I put that page on a monitor and pointed the phone at it. However long it takes for that flash to turn up in a frame the robot has actually received — that's how long the robot takes to see the world.

About 60 milliseconds. Eleven clean flashes, mean 60.5 ms, standard deviation 4.6 ms.

Two things I have to say about that number. It's an upper bound: it includes the monitor's own display lag, which this method can't separate out, so the camera path alone is faster. And "eleven clean" means I threw four out: they came back at 322 to 990 ms, and in each of those the camera was still recovering its exposure after the screen went dark, so it never saw a flash at all — excluded on a brightness rule I'd set before seeing the numbers. I posted the 60 ms figure on a forum before writing that down, which was the wrong order.

3.3 · How a frame becomes a steering command

Every frame, the steering model scores each patch for "trail." In each row, the probability-weighted centre of the patches it calls trail is that row's trail centre, rows where that mass is spread out are dropped, and a line is fitted through the rest. That is the drawn trail centre. A steering law called pure pursuit picks a goal point on that line a fixed distance ahead, the lookahead, and computes the curvature (in 1/m: how tightly the truck must turn) that would carry it through that point; a gain scales how hard it steers for that curvature. The Orin converts the result to a servo pulse width (in microseconds) and hands it to the flight controller, which drives the steering servo (Figure 8). Every stage (the fitted line, the curvature, the command, and what the radio and servo actually did) is logged per frame, so every pass can be replayed afterwards. That log is how every fault in this post was found.

4 · Day one: it drove itself

First real trail, and it worked.

Five passes down the straight at the settings I'd ended up on (lookahead 2.0 m, gain about 1.8), judged by a tripod I set up on the bank — a fixed camera watching the truck, with a tracker reading the ground colour underneath it. 94 to 100 % on the trail on every pass, one excursion longer than two seconds in 227 seconds of driving.

I want to be exact about what that number covers: the tripod's view was fixed down the straight, and the corner was out of frame. So it's a measurement of the straight. It is also not the robot grading its own homework, which is the part I care about — the robot's own confidence had no vote.

Then the corner. It saw the turn and started it, and then it went one of two ways. At a low gain it turned too gently: the curvature it was commanding climbed from 0.33 to 0.74 1/m, it asked for about 55 % of steering lock, and it arced wide into the grass. With the gain turned up it hunted everywhere, straight or corner: 152 sign changes of the steering command in 319 seconds, weaving back and forth across the trail.

The fix for the zigzag wasn't the gain. Standing there watching it, my read was "we need to look farther up the trail," which turns out to be pure pursuit's damping knob. Pushing the lookahead from 1.5 m to 2.0 m cut the weaving by more than half (0.45 to 0.19 direction changes per second). Nothing at that lookahead fixed the running wide.

5 · Day two: the app broke, then the corner didn't yield

Day two was the first field use of Robot Eyes, the iPhone app I'd built overnight. I'd been recording training data with a scanner app and streaming live video with a separate one, and no app did both from the same camera session. I wanted to gather training and validation data while the model drove on the same frames. It's the tool that later lets you replay exactly what the model saw and did. So I wrote one. It failed in the field in a way I'm glad happened early.

The phone got hot. ARKit throttled the camera to 20 frames/s, which is documented behaviour. The bug was mine: the app decimated frames by a fixed divider, so when capture dropped to a third, what reached the robot dropped to a third with it: 5 frames/s. The steering loop ended up running at 5 Hz. I fixed it sitting in the car between the afternoon and evening sessions, so I could keep recording training data and driving live off the same frames that same day; the evening ran at the full rate; the divider is now recomputed every second from the measured capture rate, so the app targets a frame rate instead of a ratio. The proper bench acceptance that night found two more defects, which is the argument for benches.

Heat is still the open problem, and I'd rather say what I'm doing about it than pretend it's solved. What's actually in: the app now reports its own thermal state every second and halves the video bitrate when iOS says the phone is struggling, so the robot at least knows, and I can read it back afterwards from the session file. What I want to try, none of it tested yet: airplane mode, to stop the phone spending power on radios it doesn't need; a small travel router on the truck so the phone isn't running wifi at all; and settling whether running it on the charger helps or hurts, because a charging battery is its own heat source. The phone is doing camera capture, depth, motion tracking and video encoding at once, in the sun, which is a lot to ask of something designed to live in a pocket.

The second fault I found with my hands rather than any log. Driving by hand on the way to the trail, the truck tracked along the left edge, in the grass, exactly parallel to the trail. Strange. I trimmed it straight on the transmitter. Then, under the model, it did exactly the same thing again. The transmitter's trim and the model's idea of "straight" turn out to be two different numbers that can't see each other: a steering servo is commanded with a pulse width, in microseconds, and the flight controller holds a centre value it calls straight (1500 µs). The truck's actual straight was 1463 µs (Figure 9). So every time the model asked for "no steering," it got a little left: a constant bias, in control-theory terms a steady-state error. Setting the flight controller's centre to 1463 fixed it. Pose data later confirmed the value independently: fitting the truck's recorded path curvature against the servo pulse, the zero crossing lands at 1461.2 µs — 1.8 µs from the number I set by feel.

With both fixed, the straight held: 99–100 % on the trail across three passes, measured on the straight. But that number is graded differently: the tripod wasn't up, so it comes from a second small classifier on the robot that reports whether the camera is looking at trail, the robot's own trail head (Figure 10). It's the robot marking its own homework, which is exactly what day one's number wasn't, so it isn't comparable and I'm not claiming it as an improvement.

And then I spent the rest of the session on the corner, systematically: one knob at a time. I kept a matrix table of the settings, and I moved the tripod to the corner and used it to count how many times the truck left the trail, because I wanted empirical validation and it was too much to hold in my head. Every lookahead from 1.0 to 3.0 metres at one gain, plus one pass at a higher gain: seven passes, one per cell, and the log of each one pulled and graded before the next.

It didn't work. Not one of them held the turn. Every pass left the trail in the bend or just after it, and rejoined further on. Short lookahead, long lookahead, more gain: the truck ran wide every single time.

And the sweep couldn't even tell me which way to go next. A 2.5 m lookahead graded worst and 2.8 m graded best, which isn't physics — it's two single passes at different speeds on different entry lines. Across two field days I'd tried every lookahead from one to three metres and gains from 1.0 to 2.1. None of them made the corner, and the differences between them were noise.

6 · The trail can't measure this problem

That's the actual finding, and it took me embarrassingly long to see it as one.

I was trying to tune the steering law with a test that couldn't resolve the thing being tuned. The throttle was my trigger finger, so no two passes entered the bend at the same speed or on the same line, and with one pass per setting I couldn't rank two lookaheads half a metre apart under that spread. Speed and entry line were uncontrolled variables, and they were bigger than the variable I was changing.

So the answer isn't a better gain. It's a controlled course, where speed is held and the same lap repeats, and a grader that needs neither a tripod nor the robot's opinion.

I built a racetrack out of blue tape on my kitchen floor.

That's the next log.

What I'm not claiming

• Corners. At all. Two field days, every setting I could turn, unsolved.
• The 94–100 % and 99–100 % figures are the straight, not the trail as a whole, and they were graded by different referees.
• The 30-minute streaming soak was indoors on a bench. The one outdoor test thermally throttled, which is the opposite of a pass.
• One park, one operator, one vehicle, daylight, dry. No transfer claim of any kind.
• The USB cable carries video. Whether it also charges the phone on the truck, I haven't measured.
• Three global-shutter cameras are on the bench being evaluated as a future upgrade for the truck. Everything above runs on the phone.

Glossary

• Self-supervised learning (as used here): supervised training where the labels are written by the robot's other sensors (LiDAR, motion tracking, accelerometer) rather than by a person. There is still an answer key; a sensor wrote it. (The pretrained DINOv2 backbone is self-supervised in the other, stricter sense: it learned from images with no labels at all.)
• Per-patch classifier: each head answers one yes/no question for every one of the 850 patches in a frame, independently: DangerNet asks "is this patch dangerous?", the steering head asks "is this patch trail?". The output is a score from 0 to 1 per patch, which is what the heat maps show; a score is an output, not one of the 385 learned weights.
• Backbone / head: the backbone (DINOv2) is a large pretrained vision model that turns an image into a grid of feature vectors and is never retrained; a head is a small classifier on top of those features. Each head here has 385 parameters.
• Patch: one cell of the 34 × 25 grid the backbone divides each image into; every label and every prediction is per patch.
• Pose: the phone's motion-tracking estimate of where the camera is and which way it points, every frame. The steering model's labels come from it.
• Pure pursuit: a steering law: aim at a goal point a fixed distance ahead on the desired path and steer along the arc that reaches it.
• Lookahead: how far up the path the goal point sits, in metres. Longer damps oscillation but cuts corners.
• Gain: a multiplier on the computed steering. Too low turns too gently; too high oscillates.
• Curvature (1/m): how tightly a path bends; the reciprocal of the turning radius. 0.74 1/m is a 1.35 m radius.
• Pulse width (µs): how a servo is commanded: a pulse between roughly 1000 and 2000 microseconds, with ~1500 as centre.
• Steady-state error: a constant offset the controller never corrects on its own. The 37 µs neutral offset was one.
• Control loop rate (Hz): how many times per second the robot sees a frame and issues a command. 15 Hz by design; 5 Hz when the app broke.
• Glass-to-application latency: the time from light hitting the camera to the frame being available to the robot's software. ~60 ms here, as an upper bound.
• Thermal throttling: the phone reducing camera rate to control temperature. Documented iOS behaviour, not a fault.
• Soak test: running the system continuously for a long time to find failures that only appear with time.
• On-trail %: fraction of driving time the truck was over trail rather than grass, per pass. Day one: judged by a tripod camera with a colour tracker. Day two: the robot's own trail head.

Full build notes, including every wall I hit and how I got past it, are in the troubleshooting guide. Video: Log #1's film is here; Log #2's six short clips land on the same channel, one a week. Previous log: I rearranged my kitchen to fool my robot. It found the truck anyway.

How to read the tags: [measured] we ran it, method stated · [reported] someone identifiable measured it, on their hardware · [claim] a vendor said so and nobody checked · [estimate] reasoned, never measured · [none found] we looked and found nothing, which is itself a result.

Every entry is something that actually cost us time. Nothing here is hypothetical.

1 · ExpressLRS won't bind

Your receiver is powered through an ESC, and power-cycle bind does nothing

Symptom. The receiver blinks steadily, never enters bind mode, and pressing Bind on the transmitter does nothing. Link quality stays at zero.

What worked. Stop cycling the ESC's switch. Unplug the receiver's servo lead and reseat it: that cuts power at the receiver itself. Three cycles, then leave it seated, then press Bind. Bound on the first attempt after doing this. [measured]: RadioMaster ER5C-i on a Traxxas XL-5, 2026-09-05.

Why we think it happens. The ESC's capacitors likely hold the receiver's supply up through the switch, so it never sees clean interruptions to count. [estimate]: we did not put a scope on the rail and did not measure how long the 6 V line stays up. The fix is verified; the explanation is not.

Each power-cycle has to be brief

Roughly one second powered, then off. More than about two seconds and ExpressLRS resets its counter, so a slow, careful sequence fails where a brisk one works. [measured], consistent with documented ELRS behaviour.

You're reading the wrong version number

On an EdgeTX radio, SYS → Version is EdgeTX, the radio's operating system. It has nothing to do with binding. The number that matters is the ExpressLRS module firmware, at the bottom of SYS → Tools → ExpressLRS. [measured]

The bottom of the ELRS Lua screen. [Bind] sits directly above the version line 3.3.1 ISM2G4, which is both the firmware version that governs binding and the regulatory domain. (The transmitter's binding UID, shown beside it, is redacted here: it is the bind key.)

Checking FCC vs LBT

Read it off the same ELRS screen, not the box. ISM2G4 is the non-LBT domain (full power); an EU/LBT build reads CE2G4. [measured]

Versions don't have to match exactly

ExpressLRS breaks over-the-air compatibility across a minor version change (3.3 → 3.4), not a patch. Our TX on 3.3.1 bound to a receiver on 3.3.0 with no flashing. [measured]

Still no link — check these three before reflashing anything

The 0/250-style figure in the Lua screen header is link quality. Zero means no packets at all, which points at binding rather than configuration. [measured]

A known-good ELRS configuration, with 0/250 in the header: link quality zero, meaning no packets at all. Note Model Match: Off; a mismatch there prevents a link with no error anywhere.

Internal RF in CRSF mode, External RF OFF. Both screens are the same Model Setup page, scrolled.

Last resort: talk to the receiver directly

Leave the receiver powered with no bound transmitter for ~60 seconds. It gives up and broadcasts its own WiFi access point; connect and browse to 10.0.0.1. That page reports the receiver's real firmware version, whether a binding phrase is set (a phrase makes traditional binding impossible, which is the other common cause of exactly these symptoms), and it offers a flash button. If no access point appears after two minutes, the receiver isn't booting. [claim], vendor-documented behaviour; we did not need to use it.

Don't trust port labels when swapping receivers

Traxxas receivers carry three channel ports plus a battery port, which reads at a glance as a duplicate channel. Map by what the wire physically connects to (steering servo to CH1, ESC to CH2), not by matching printed numbers between two different manufacturers. Both the Traxxas TQ #6519 and the RadioMaster ER5C-i use black for negative, so plugs transfer without adapting. [measured]

The stock Traxxas TQ #6519. The port block carries three channels plus a battery port — at a glance it reads as a duplicate channel. Its own label states 6V / 15mA, which is the supply the replacement receiver will see.

The ER5C-i. Five shrouded 3-pin headers labelled CH1–CH5, a BOOT button, and a separate 2-pin EXT-V header — that last one is an external voltage telemetry input, not a channel.

The TQ #6519's own label reads 6 V, comfortably inside the ER5C-i's stated 4.5–8.4 V input range [claim], so no level shifting is needed on this combination.

1b · Getting CRSF out of an ExpressLRS PWM receiver

Your flight controller can't read individual PWM channels: it wants one serial stream. On a PWM receiver you get that by repurposing an output pin. Everything below is [measured] on a RadioMaster ER5C-i, firmware 3.3.1 ISM2G4, 2026-09-05.

First: reaching the receiver's web page at all

The receiver hosts a config page on its own WiFi. Four things cost us most of an hour.

The transmitter must be OFF. If the radio is on, the receiver stays linked and never starts WiFi. Turn it off and leave it off.

It takes minutes, not the 60 seconds the setting implies. The "WiFi auto on interval" reads 60, but the network took several minutes to appear. Don't conclude it failed at ninety seconds.

Better: command it into WiFi mode from the radio. SYS → Tools → ExpressLRS → WiFi Connectivity has an option to enable WiFi on the receiver. That's far more reliable than waiting for a timeout that fires only when the receiver gives up looking.

Network ExpressLRS RX, password expresslrs.

🔴 Escape iOS's captive-portal browser. Connecting pops a cut-down "Captive Wi-Fi" window. The page renders, but the tabs don't respond — you can look and not touch. Close it, choose Use Without Internet when iOS asks, then load 10.0.0.1 in Safari proper.

Choose Use Without Internet. "Dismiss" or "Use Other Network" drops you off the receiver.

The captive window. It looks right and the tabs do nothing. This is the trap.

🔴 And 10.0.0.1 is also a very common home-router address. The moment the phone falls back to house WiFi, the same URL quietly serves your gateway instead of the receiver — for us, an Xfinity page. Check the page header says ExpressLRS before you trust anything on it.

Same address, wrong device. The phone had dropped back to the house network.

What the page should look like

Correct page. Four tabs: OPTIONS · WIFI · MODEL · UPDATE. Serial output lives under MODEL.

🔴 Serial TX is only offered on ONE output row

This is the thing that cost the most time. The receiver's own help text says "Serial (for MCU pins 1 and 3 only)" — that is internal MCU numbering, not the output numbers in the table. Going by it sends you to the wrong row, where the Mode dropdown simply ends at On/Off and you conclude the firmware doesn't support serial.

Row 1's Mode dropdown. Ends at On/Off. No serial — and nothing tells you why.

On the ER3Ci / ER4 / ER5 series, it is output 2. Set output 2 → Serial TX and output 3 becomes Serial RX automatically. [reported]: Oscar Liang, confirmed [measured] on our own ER5C-i.

Row 2's dropdown. Identical list plus Serial TX, and row 3 has already greyed itself out as the paired Serial RX.

Then

1. Serial Protocol: appears below the output table only after a pin is set to Serial TX, which is why you can't find it beforehand. Set it to CRSF.
2. SAVE. You want "Configuration updated."
3. UART baud on the OPTIONS tab. It already read 420000 for us. Check before changing.

What success looks like, with Serial Protocol: CRSF visible behind the dialog.

OPTIONS tab. UART baud 420000 is the CRSF default — likely nothing to change.

Leave the Binding Phrase field EMPTY. If the page says the UID was set by the traditional method, typing a phrase generates a new UID and breaks the bind you already have.

What you lose, and it is expected

Output 2 stops being a servo output the moment it becomes serial. If your ESC was on it, the throttle goes dead at the vehicle while still showing movement on the transmitter's channel monitor: the radio is transmitting fine; the receiver pin changed job. Throttle comes back through the flight controller. Any PWM outputs you didn't touch keep working.

2 · 🔴 ArduPilot's companion-computer failsafe is off by default, and fails silently

This is the most dangerous thing in this document. If your companion computer is sending guidance commands and it dies, you may have no failsafe at all, and nothing will tell you.

Three defaults combine:

• FS_GCS_ENABLE defaults to 0: the failsafe is disabled
• MAVROS's system_id defaults to 1, while ArduPilot expects 255
• Rover's gcs_failsafe_check() returns early if it has never seen a qualifying heartbeat, so a failsafe that was never armed never fires

[measured] against ArduPilot source and MAVROS defaults.

Minimum working configuration:

MAV_GCS_SYSID   255     # named SYSID_MYGCS on Rover <= 4.6
FS_GCS_ENABLE   1
FS_GCS_TIMEOUT  2
FS_TIMEOUT      1
FS_ACTION       2       # Hold
GUID_TIMEOUT    0.5
FS_THR_ENABLE   0       # only if no RC receiver is bound

plus MAVROS system_id: 255.

Verify by breaking it, not by reading parameters. Pull the companion computer's power in the middle of a command and watch the vehicle go to Hold. If you cannot make it fail on demand, you do not have a failsafe — you have a parameter list.

3 · ROS 2 on a Jetson

MAVROS below 2.15.1 is broken on Humble

Latched publishers broke on Humble and were fixed inside the 2.15.0 cycle. Below that, topics including ~/state never publish, which presents as a bridge that connects and then does nothing. Require ≥ 2.15.1. 2.15.1 also fixed duplicated launch namespaces, whose symptom is topics appearing at /mavros/mavros/.... [measured] against the MAVROS changelog.

The MAVROS node exits immediately

GeographicLib datasets are mandatory, not optional: the node shuts itself down without them, which reads like a crash.

sudo ros2 run mavros install_geographiclib_datasets.sh

[measured]

Discovery is flaky on a Jetson specifically

Two things, both cheap:

• Use CycloneDDS: export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp. This is field consensus rather than official guidance [reported], but MAVROS's own 2.15.0 changelog cites FastDDS shared-memory and teardown hangs [measured].
• Pin ROS_DOMAIN_ID. A Jetson typically has both WiFi and a USB-gadget interface up, and multicast discovery gets confused across them. [estimate]: reasoned from the interface layout, not isolated experimentally.

4 · JetPack and the Jetson itself

Your working setup breaks after an unattended upgrade

Ubuntu's automatic updates break JetPack compatibility. Disable unattended-upgrades and apt-mark hold the NVIDIA/L4T packages before it matters, not after. [reported]: widely reported on the Jetson forums, and hit here too.

nvv4l2decoder fails and /dev/nvhost-nvdec* doesn't exist

The device nodes being absent is not the cause. We measured nvv4l2decoder working on JetPack 6.2.1 with those nodes absent, on an Orin Nano Super 8 GB. If you're chasing missing device nodes, you're chasing the wrong thing; NVIDIA's own guidance for the related reports points at the GStreamer registry cache (rm ~/.cache/gstreamer-1.0/registry.aarch64.bin, install nvidia-l4t-gstreamer). [measured]: written up in full at NVIDIA forums topic 382387.

Don't commit TensorRT engines

Engines are keyed to SM architecture and TensorRT/CUDA version, so a checked-in .engine is portable to exactly one machine. Commit ONNX plus a deterministic builder script plus the calibration cache instead. [measured]

FP16 gains nothing under TensorRT on an Orin ViT — but 3.4× in plain PyTorch

On Vision Transformers specifically, we found FP16 gives roughly 0% speedup under TensorRT on Orin: the ONNX exporter never emits a multi-head-attention pattern for TensorRT to fuse. In eager PyTorch the same precision change is a different story: 3.4× faster (18.9 ms vs 63.7 ms per batch) at no measurable accuracy cost for us. Measure your own path before assuming either way. [measured] for both numbers; [estimate] for the generalisation beyond the models we tested.

5 · Streaming an iPhone into a Jetson over USB

The phone doesn't enumerate

Run usbmuxd with --no-preflight. The default preflight step races the iOS trust pairing and produces the "0 devices found" reports that dominate search results for this problem. [measured] on iOS 26.6 → JetPack 6.2.1.

Don't do this on Ubuntu 20.04

Focal ships libimobiledevice linked against GnuTLS, which breaks trust pairing. Use 22.04. [measured]

What we found when we looked for prior art

[none found] — and that is the finding, not a gap. The NVIDIA developer forums return five usbmuxd topics forum-wide, none about iOS video, and zero for libimobiledevice. The iPhone-on-Jetson space is empty. If you're doing this, you are close to the frontier and should expect to write the documentation rather than read it.

Reported failure modes worth knowing about

Neither reproduced for us, but both are real reports from people doing very similar things:

• record3d #72: "variable lag in the video stream (1-3 seconds)", filed by a Dobb·E author streaming from an iPhone on a robot over USB, open since 2023. [reported]
• usbmuxd #215: "works fine for a random amount of minutes, until it dies." [reported]

We ran 27,000 frames over 30 minutes with zero stalls and zero reconnects on our own path (different app, different client), which does not refute either report — it establishes that our path does not have it. [measured], indoors on a bench; outdoor thermal behaviour is untested.

6 · Capturing data from a phone

Recordings come out corrupt

Video finalization races sharing. We lost video on 6 of 9 recordings in one session and 3 of 8 attempts in another. A fixed post-stop wait is not sufficient: we specified five seconds and still lost one in six. The recording appearing in the app's own list is the actual finalize signal; wait for that event, not a timer. [measured]

A truncated MP4 is partially recoverable by rebuilding annex-B HEVC from mdat using headers from a known-good clip from the same app — but we could not prove frame alignment to single-frame precision, and shipped nothing that depended on it. [measured] mechanically; alignment [none found].

ARKit pose dies within a minute on a moving vehicle

Visual-inertial odometry on a phone is not rated for vehicles. Every session on one device died in 19–47 seconds. What fixed it was speed discipline, not weather: a newer phone, 5 fps capture, and keeping p95 speed at or below ~5 m/s gave clean full-session pose across six consecutive outdoor runs, validated by loop-closure error of 1.6–6.0 m over 465–636 m of driving. [measured]

Check what your "frame rate" actually is

Ours was decimation, not capture rate: ARKit ran at 59.976 Hz and the app kept every twelfth frame. That matters because every thermal event we had recorded happened with the sensor at full rate, so conclusions about "5 fps is sustainable" were about a much heavier load than they appeared to be. Verify against your own timestamps. [measured]

7 · Self-supervised labelling pipelines

Don't use the centre pixel's depth to locate an event

If the object isn't centred, the centre pixel reads the floor beside it and mislocates the label by roughly a metre. Use the nearest above-floor cluster instead. [measured]

A suspiciously good score is a bug report

We hit AUC 0.998 (obstacle model, Log #1) on geometry that was broken, twice. The number was the warning; what caught it both times was looking at the overlays. If you have a self-supervised label chain, your metric measures agreement with your labeller, not with the world — so build the visual check first and trust it over the score. [measured]

Contributing

If you hit something on this list and our fix didn't work, or you hit something that isn't here, send it with exact versions, exact commands and exact output: reply on the post, or email captaincolinr@gmail.com until the code release opens a public tracker. That is the same standard we hold ourselves to when reporting upstream, and it is the difference between a report someone can act on and one they can only sympathise with.

The idea: expensive sensors teach, cheap sensors ship. During training, an iPhone's LiDAR, ARKit pose tracking, and accelerometer do all the labeling work. The model that comes out consumes none of them — RGB images are its only runtime input. The first truck prototype still carries the whole phone (it provides the camera and the compute); what's being tested is whether plain camera images are the only perception signal the finished model actually requires.

I've shipped this pattern before. I spent twenty months building a running-gait system where an expensive camera rig generated the labels that trained a watch model — and the watch shipped without the camera. This project is that pattern, made visible.

How the labels happen

I walk the room for a few minutes. When I'm at arm's length from something the truck shouldn't hit, I bump the phone body with my free hand — the phone never touches the obstacle. Walking never exceeds about 0.6 g of accelerometer deviation (g as in units of Earth's gravity); bumps land at 1.6–4.7 g, so detection is a threshold. The accelerometer identifies when a bump occurred, LiDAR identifies the nearby obstacle in 3-D, and ARKit projects that location back into every earlier frame that saw it on approach — those patches are labeled "no-go." Floor I walk over seconds later labels itself "go." Thirty bumps became 147,731 labeled image patches. No manual image labeling — nobody draws a box.

The model is deliberately tiny: frozen DINOv2 image features feed a logistic-regression head with 385 trained parameters. It retrains locally in about two minutes. Model size doesn't make an evaluation trustworthy by itself, so the experiments use held-out time blocks and pass/fail gates defined before training.

The gate, then the experiment

Before training, I wrote down a pass/fail gate with two separate conditions: at least 0.85 AUC against LiDAR-assisted labels on held-out time blocks, AND heatmap overlays that visibly outline the obstacles. A good score doesn't prove a good picture. The first session — a 3.5-minute walk, 21 bumps — passed both: 0.932, and you could see the obstacles in the heat.

Same-session held-out is the easiest honest test there is. So when the kitchen got rearranged, I treated it as a free experiment — and evaluated the old model on the new room before retraining anything. What changed: the parked RC truck moved to a new spot; a battery and charger were removed; a 3D-printed plastic hand appeared (made for last Halloween; never seen, never bumped); a stool and a drone case stayed put.

Results, old model on the new room: the heatmap outlined the truck at its new location — it found the truck, not the truck's old spot — and showed no comparable response in the empty space where the battery and charger had been. Then I re-ran the whole loop in the new room: 30 bumps, ~2 minutes of training → AUC 0.9403 on that recording's held-out labels. The old model on those same held-out labels: 0.9224 — a 1.8-point gap on identical test data, for a model that had never seen this arrangement. (0.932 and 0.9224 come from different recordings, so I'm not attributing their difference to the rearrangement alone.)

Why test like this at all? An earlier wrist-IMU model from my running project dropped 21 percentage points in a matched test on another day — different task, different metric, so the numbers aren't comparable; together, though, those two experiments are why I now test every system under changed conditions instead of trusting a single held-out score.

The useful failure

The 3D-printed hand is the best part. Under the old model it produced a strong response nearby but only a faint one from across the room: predicted no-go probability 0.19, in a frame whose median was 0.00. After retraining with bumps that included the hand, the same distant viewpoint — measured on a separate no-bump walk — reads 1.00, in a frame whose median is 0.16. The retrained model runs hotter overall, so the margin over the median is the result — and it's one object at one viewpoint, not a generalization claim. But that's the loop: new dangers need new bumps — and the bumps work.

Honest limits

Not tested yet, and therefore not claimed: the truck's ~15 cm camera viewpoint (all data is hand-held so far), another building, another operator, dynamic obstacles, closed-loop control. One person, one home, real measurements. Goal for November: phone mounted on the truck (the mount is real — it's in the video's first seconds), the danger overlay on-device, and driver assistance — a camera-only model flags danger while a human drives. If testing supports it, the next step is shared autonomy, with the human keeping immediate override.

Lineage: BADGR (UC Berkeley) and Wild Visual Navigation (ETH Zürich), shrunk to consumer hardware. Every number above traces to a dated findings ledger.

Next log: the truck goes outside.

Video: youtu.be/4TfdhlGQR4A · Full build log with receipts: coming to Hackaday.io — link will land here.

Both innocent. cProfile said 73% of wall-clock was cv2.VideoCapture.set(CAP_PROP_POS_FRAMES) — seeking the source video.

Why seeking dominated

The composer builds ~8 segments, each revisiting different, overlapping frame ranges out of order. Frame access went through a small LRU cache (60 frames); on a miss it did:

cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
ok, img = cap.read()

The trap: VideoCapture.set(CAP_PROP_POS_FRAMES) is not O(1). Video is inter-frame compressed, so OpenCV decodes forward from the nearest keyframe on every seek. For a 600-frame clip accessed out of order across 8 segments, the 60-frame cache thrashed — nearly every access re-seeked. Measured locally: ~1,251 seeks × ~61 ms ≈ 76 s of a 104 s render, pure seeking. Drawing was ~10%; encoding ~11%. The render was never the bottleneck — frame I/O was.

The fix: decode once, sequentially, into memory

Read the whole clip front-to-back one time into a dict keyed by frame index; frame(idx) becomes a memory lookup. Guard against huge clips so a long or 4K source can't OOM the container:

def _predecode(self):
    self._decoded = True
    nframes = int(self.cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
    if nframes <= 0 or nframes * self.W * self.H * 3 > 6 * 1024**3:
        return  # too big — keep the per-access seek fallback
    self.cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
    i = 0
    while True:
        ok, img = self.cap.read()
        if not ok:
            break
        if img.shape[1] != self.W:
            img = cv2.resize(img, (self.W, self.H))
        self._cache[i] = img
        i += 1

Sequential decode is what video codecs are optimized for; you pay the decode cost exactly once.

Results (measured)

Byte-identical output (same md5) — a pure speedup, which also made it safe to ship without re-validating the video content.

The check most optimizations skip: does the rest of the pipeline have this bug?

No — and knowing why is the useful part. The main renderer reads frames sequentially (cap.read() in a loop, no seeking), so it was already fine. One other POS_FRAMES use grabs ~3 frames per event — bounded, ~1 s. The thrash was unique to the one component with an out-of-order, multi-segment access pattern. The fix stayed scoped to that file, and nothing else needed touching.

Limits

• Cost still scales with source pixels: the renderer draws on full-res frames even though the output canvas is small. A 4K source is ~63 minutes and should be downscaled before rendering (that change touches pose-coordinate scaling, so it carries correctness risk and wasn't done). Our capture path emits 1080p, and the memory guard degrades gracefully, so production is safe as-is.
• The 6 GiB guard is sized for a 16 GiB container; scale to yours.

The transferable rule

For any renderer that revisits frames out of order: decode once into a cache; never random-seek per frame. And before optimizing a slow multi-pass renderer, profile — python -m cProfile -o out.prof … then sort by cumulative — because the plausible bottleneck (drawing, encoding) and the actual one (I/O) are routinely different, and one run of the profiler settles it.

If you stop reading there, the takeaway is "AI agents can ship iOS apps now." That takeaway is wrong in an important way, and the wrongness is the useful part.

Here's the claim I actually believe, and the evidence for it: the agent executed, in one night, a recipe that my failures had spent months writing. Nothing about that night was improvised. Every step the agent took either followed a runbook that a previous failure had paid for, or hit a brand-new failure — and the new ones are now in the runbook too.

What the agent actually did that night

The condensed flow, for people who came for the recipe (all raw App Store Connect API — fastlane turned out to be a dead end on my machine, more on that below):

1. Build & upload: bump CURRENT_PROJECT_VERSION, xcodebuild archive → -exportArchive with an export-only options plist (method=app-store-connect, no destination) → upload the IPA with xcrun altool --upload-app using an ASC API key. 163 MB, 12-second transfer.
2. Wait properly: poll GET /v1/builds?filter[app]=…&sort=-uploadedDate until processingState=VALID. Not "wait a bit and hope" — poll for the actual state.
3. Screenshots via API: for each image — delete the stale appScreenshots in the display set, POST /v1/appScreenshots to reserve upload operations, PUT the bytes, PATCH uploaded=true with an md5 sourceFileChecksum, then poll assetDeliveryState until COMPLETE.
4. Attach the build to the version record, patch reviewer notes, keep releaseType=MANUAL.
5. Submit: POST /v1/reviewSubmissions → POST /v1/reviewSubmissionItems linking the appStoreVersion relationship — not appStoreVersionForReview, which 409s — then PATCH {"attributes":{"submitted":true}}.
6. Verify: read back the version and the review queue; both said WAITING_FOR_REVIEW. Only then report done.

Two things went wrong that night, which is exactly the point. First: we'd deleted the Apple Watch screenshot set, reasoning that v2.0 has no user-facing Watch surface. ASC rejected the review submission item with SCREENSHOT_REQUIRED.APP_WATCH_SERIES_4 — because the binary still ships a dormant, flag-gated Watch app, and if the binary ships it, the listing needs its screenshots. The agent recovered the previous version's Watch images through each localization's old screenshot set (imageAsset.templateUrl, fill in {w}x{h}bb.{f}) and re-uploaded them. Second: one of the new iPhone screenshots came back black — a staged camera capture the agent couldn't retake, because a camera pointed at nothing is one of the few things an agent genuinely cannot fake. We shipped three good screenshots and deferred the hero shot to a metadata update.

Both failures are now permanent lines in the runbook. That's the loop this whole post is about.

The staircase this stood on

Here is what that one night actually consumed, with dates. I went back through the repo's own records to check my memory, because "I couldn't have done this cold" is exactly the kind of claim people flatter themselves with.

May 26 — the first submission took ~7 hours, by hand. Four build cycles, two screenshot redos, a privacy-declaration expansion. It produced a 12-item gotcha list that reads like a hazing ritual: screenshots must be exactly 1284×2778 (a native iPhone 17 Pro screenshot is 1206×2622 — "close" is a rejection); macOS screenshots carry an alpha channel that ASC rejects, and sips silently fails to strip it while PIL and ImageMagick work; the listing's "Requires iOS X" line is derived from the binary's deployment target, not your marketing copy; every data type you declare in App Privacy must appear, by Apple's exact name, in your privacy policy, because the reviewer reads both and matches them.

June 21 — credential archaeology. A headless upload needs an ASC API key. The .p8 had been created back in September 2025 and later removed from the repo; the working copy was recovered from a git blob in the repo's history, validated with openssl, and stored properly. The same session proved the "normal" path — uploading through the signed-in Xcode account — is flaky from a headless CLI context (Xcode 26's account layer can't reliably be reached from a spawned xcodebuild), which is why the API-key path matters. fastlane was also a dead end on this machine (system Ruby, no bundler). Every one of those dead ends cost real hours, once.

June 25–26 — the recipe gets validated on a real release. Issuer ID resolved, device registration via POST /v1/devices proven (the CLI's -allowProvisioningUpdates does not register unknown devices, another thing you learn exactly once), and then v1.5.7 shipped through the full headless path: archive → export → altool → poll → attach → submit. A pre-submission audit that same day caught a genuine rejection blocker — a feature that had drifted out of its internal-build gate and contradicted a note we'd previously given Apple's reviewers.

July 4 — the release-order trap. A marketing version closes for TestFlight beta review the moment it's released on the App Store (ENTITY_UNPROCESSABLE.CLOSED_VERSION). Learned on v1.5.8, encoded as "do not click Release until beta review clears." This is why the agent left releaseType=MANUAL and why the human keeps the Release click.

And under all of that: the app itself. The "48-hour" v2.0.0 was a carve-out from an app that took 18 months to build — the capture flow, the cloud GPU rendering pipeline it calls (already deployed, already canary-tested), the build system, the signing setup, the device-automation harness the agent used to screenshot its own UI changes on a physical iPhone. The velocity of the last 48 hours was almost entirely stored potential energy.

The part that isn't about Apple at all

There's a second dependency that's easy to miss because it doesn't live in any runbook: knowing when to believe the agent.

Agents report success with the same fluent confidence whether or not the thing succeeded. Early on, I got burned by this repeatedly — "done and verified" claims where the verification was the agent's own optimism. The discipline that fixed it is boring and absolute: verify the effect against an independent signal, never the self-report. It's baked into the submission runbook in small, specific ways: don't trust the upload script's exit code (a trailing tee masks altool's real status — look for UPLOAD SUCCEEDED and a Delivery UUID); don't trust "submitted" until the API reads back WAITING_FOR_REVIEW; don't trust a config change until the serving system proves it changed.

That discipline is why the agent could run for hours without me watching. Not because it doesn't fail — it failed twice that night — but because the process doesn't let a failure masquerade as a success. It took months of being burned to internalize that, and I don't think there's a shortcut. If you hand an agent a task you've never done yourself, you have no independent signal to check it against, and you've built an unsupervised liar with API access.

What the human still did

Worth being precise about, since "end-to-end" invites exaggeration. I decided what the app was and what got cut. I reviewed the store description and What's New before they went up. I confirmed the privacy labels in the ASC web UI — there is no official API path for privacy labels (the endpoints 404 with a valid key; the one tool that automates them rides a private Apple-ID session API). I kept the Release click. And I was the reason there was anything to submit: the judgment calls about what a two-day pivot should contain came from 18 months of watching real users hit the previous app.

The transferable version

If you want an agent to do some multi-step, high-stakes process for you headlessly, I now believe the path is:

1. Do it by hand first, badly. The 7-hour version is not wasted time; it's the tuition.
2. Write down every trap as it bites you, in a runbook the agent will literally read. Mine says things like "READ IT" at the top, and agents do.
3. Make verification independent of the agent. Every step needs a check the agent can't wish into passing: a state read back from the API, a checksum, a serving system's own report.
4. Keep the irreversible clicks human until the boring parts have earned trust. releaseType=MANUAL is a philosophy, not a config value.
5. Feed failures back into the runbook the same night. The Watch-screenshot 409 is already in mine, so the next agent won't pay for it twice.

The headline version of that night — AI ships app in 48 hours — is technically true and deeply misleading. The truthful version is better, I think: an agent turned 18 months of accumulated, written-down, independently-verifiable experience into one night of execution. The leverage is real. It's just not free, and it's not the model — it's the runbook, and the failures that wrote it.

Update 2026-08-28: watchOS 26.6 shipped publicly on July 27, 2026. I have not yet re-tested the install path on 26.6, and Apple's release notes don't mention this bug — treat 26.6 as unknown. If you're pinned on 26.5/26.5.1, everything below still applies; if you've tested on 26.6, reporting either way in thread 827053 helps everyone.

If your Apple Watch is on watchOS 26.5 or 26.5.1 and third-party apps won't install — the Watch app on the iPhone spins for about a minute on "Install" and then fails with "could not be installed at this time" — it's not your app, your provisioning, or your pairing. It's an acknowledged Apple regression, and there's a reproducible workaround that doesn't involve waiting for the next watchOS release.

I hit this as a developer trying to get my own dev build onto my watch, but the bug is device-wide: on the same watch, Snapchat and American Airlines failed to install the exact same way. Store apps, TestFlight, enterprise, and dev builds all ride the same broken path.

The bug

On watchOS 26.5/26.5.1, installs from the iPhone's Watch app ("Available Apps → Install", or the "Show App on Apple Watch" toggle) fail after ~1 minute. It worked fine on ≤26.4. Reboots don't help.

You're not alone: Developer Forums thread 827053 has at least six developers reporting the identical failure across enterprise distribution, dev-provisioned OTA installs, and plain Xcode builds — one with an academic project deadline riding on it. Apple has acknowledged it (feedback FB22807635); an Apple engineer replied in June: "it is most likely a regression we have known, and folks are actively working on that." The underlying failure is the transfer socket timing out:

ACXError Code=8 "Failed to create socket"
identityservices Code=20 "Socket open timed out"

As of August 4, 2026, no shipped watchOS release is confirmed to fix it. The newest report in the forum thread (late July) reproduces the failure on watchOS 26.5.2 with iOS 26.5.2; the 26.6 release candidate went to developers on July 20, and as of this writing nobody in the thread has confirmed the fix either way.

What's actually breaking (the plain-English version)

When you tap Install in the iPhone's Watch app, the app doesn't come from the internet to the watch — the iPhone transfers it to the watch over their private phone↔watch link (Apple's Identity Services layer, riding Bluetooth). On watchOS 26.5, opening that transfer socket times out — so the install fails after a minute, no matter how many times you retry, reboot, or re-pair. The app never leaves the phone.

That also explains why the workaround works: take Bluetooth away, and the watch falls back to WiFi — where a Mac can hand it the app directly, skipping the broken relay entirely.

What does NOT work (I tried all of it)

• Rebooting phone and watch (in every order)
• Toggling the app off/on in the iPhone's Watch app
• Unpairing and re-pairing the watch (others in the thread also ruled out factory resets and fresh certificates/provisioning profiles)
• Toggling Bluetooth off/on on both devices — this one is worth trying first, because it has worked for several people in the forum thread. It didn't take for me. If it doesn't for you either, the fix below is the reliable path.
• TestFlight instead of a dev build — same phone↔watch transfer path, same failure

If you're a developer, Xcode "Run" against the watch also stalls, for a different reason — see below. That combination (store path broken + Xcode path broken) is what makes this bug so disorienting: every install route fails, each with a different symptom.

The fix that worked (reproducible)

The trick is to take Bluetooth out of the equation entirely and push the app over the network path instead. The forum thread independently converged on devicectl as the working install route; what follows is the complete recipe — including the two pieces that cost me the most time (why Bluetooth has to be off first, and which of the tooling's status reports to ignore).

One honest caveat: this is a developer workaround. devicectl ships with Xcode, so if you're distributing to enterprise users or testers who don't have a Mac with dev tools, they're stuck with the Bluetooth-toggle lottery until Apple ships the fix. (Non-developers: also try installing from the App Store on the watch itself — that path downloads directly to the watch rather than relaying through the phone. Reported to help for adjacent variants of this failure, though I haven't tested it on 26.5 myself.)

1. Turn the iPhone's Bluetooth OFF.

This forces the watch onto WiFi and lets the Mac↔Watch CoreDevice tunnel come up. Keep Mac, iPhone, and watch on the same real WiFi network — not the iPhone's Personal Hotspot, which isolates the devices from each other. (In my case the tunnel had been dead for a day — tunnelState=disconnected, ddiServicesAvailable=false — and came up once Bluetooth was off.)

2. Install with devicectl from the Mac — not Xcode "Run":

xcrun devicectl device install app \
  --device <WATCH-COREDEVICE-ID> \
  --timeout 240 \
  "$HOME/Library/Developer/Xcode/DerivedData/<YourApp>/Build/Products/Release-watchos/<Your Watch App>.app"

Find your watch's CoreDevice ID with xcrun devicectl list devices. Use the Release-watchos build product — it's clean, with no __preview.dylib inside.

Two verification notes, because this path lies to you in both directions:

• devicectl device info details may still report tunnelState=None from stale cache even when the install succeeds. Trust the App installed output of the install command, not the info query.
• If the install command itself succeeds, the app is on the watch — you don't need the iPhone Watch app to agree, and it may take a while to notice.

Why not Xcode "Run"?

Xcode blocks on "Copying shared cache symbols (0%)", and on watchOS 26.5 that symbol download fails with an HTTP 403 ("Symbols for watchOS 26.5 — forbidden"). Here's the thing: that download is debugger support only — it has nothing to do with installing the app. devicectl install skips it entirely, which is exactly why it works when Xcode doesn't.

If you don't need to attach the debugger, devicectl is strictly the better path on 26.5.

Summary

If this saved you a day, it cost me one — file feedback on FB22807635 anyway; duplicate reports are how Apple prioritizes.