E04_The_Minute_That_Trains_the_Model

The Minute That Trains the Model

openpilot is a data flywheel dressed up as a driver-assistance product. Every minute of engaged driving produces a 50+ MB artifact that retrains the next release. The roadmap to openpilot 1.0 is the closed loop made explicit — and the 332 supported cars are the asset that makes it possible.

Key Takeaways

• A route is broken into one-minute segments. Each segment is a rlog.zst (a zstd-compressed Cap'n Proto log of every pub/sub message) plus three H.265 camera streams. Total size per minute: roughly 50–60 MB.
• qlog.zst and qcamera.ts are the *uploaded* versions. They are decimated subsets of the rlog and lower-resolution versions of the camera streams, designed to be small enough to upload instantly on slow cellular.
• The decimation values in cereal/services.py are the discipline. can is decimated by 2053 (about 3 messages per logged segment). controlsState by 10. The full 100 Hz is preserved on-device; the bandwidth is engineered for the link.
• "Firehose Mode" (released 0.9.8) is the explicit acknowledgment that the model needs more data, faster. It maximizes the upload rate.
• The "testing closet" in the README — 10 comma devices continuously replaying routes — is the test harness. It is the only way to validate a release against the diversity of the 332-car fleet at CI speed.
• openpilot 1.0 is "a fully end-to-end driving policy". The roadmap makes the data flywheel explicit. The flywheel is the product. The driver-assistance system is the input.

You finish a 22-minute drive. You park the car, kill the ignition. The comma four's LTE modem starts a 7 MB/s upload to comma's storage. In 90 seconds your 22-minute drive is on a server in San Diego. By tomorrow, the gradients from that drive are mixed into a 50 GB training batch. By next week, they are in a model checkpoint. By next month, they are in a release that gets pushed to your car over the air, and the loop closes.

I want to walk you through that loop in detail, because the loop is the product. The driver-assistance system is what the user sees. The flywheel is what comma is actually building. The 332 supported cars are the data-collection platform, the testing closet is the validation harness, the release trains are the deployment pipeline, and the World Model from E03 is the thing the flywheel is producing. The flywheel is the business. The car is the input.

What gets recorded

The data architecture is in docs/concepts/logs.md and the per-service decimation constants are in openpilot/cereal/services.py. I will start with the recording side, because the recording side is the one the device does without asking.

A *route* begins on the rising edge of ignition and ends on the falling edge. The route is split into one-minute segments. For each segment, the device writes four files to local storage:

• rlog.zst — a zstd-compressed stream of every Cap'n Proto message that crossed the bus during the minute. The schema is log.capnp. The decimation values are in services.py.
• fcamera.hevc — the road-facing camera, H.265 encoded.
• ecamera.hevc — the wide road camera, H.265 encoded.
• dcamera.hevc — the driver-facing camera, H.265 encoded. (The driver camera is only recorded if you opt in.)

The size budget for one minute of driving: a raw rlog at 100 Hz across 30+ services runs roughly 100–200 MB uncompressed, and zstd typically gets that to 5–10 MB. Each H.265 stream at the comma four's resolution runs roughly 5–10 MB/min. Add them up and a minute of engaged driving produces roughly 50–60 MB of raw local data before the user has done anything.

sequenceDiagram
    participant Car as Car sensors
    participant OP as openpilot process tree
    participant Store as Local segment storage
    participant LTE as LTE modem
    participant Cloud as comma cloud (San Diego)
    participant Train as Training pipeline
    participant OTA as Release server
    Car->>OP: 100Hz CAN, 104Hz IMU, 20Hz cameras
    OP->>Store: rlog.zst + 3 H.265 streams (~50MB/min)
    Note over Store: 1-minute segment, on-device
    LTE->>Cloud: qlog.zst + qcamera.ts (~3-5MB/min) decimated
    LTE->>Cloud: opt-in: rlog + HEVC streams for "firehose" users
    Note over Cloud: Aggregate, dedupe, ~50GB training batch
    Train->>Train: Train World Model + driving policy
    Train->>OTA: Model checkpoint → release
    OTA->>Car: OTA update to device
    Car->>OP: New model runs at 20Hz
    Note over Car,OP: Loop closes

The next thing that happens is the upload. The device does not upload the full rlog. It uploads a decimated version called qlog.zst, and a lower-resolution H.264 version of the road camera called qcamera.ts. The decimation values are tuned for slow cellular. The user is back in the house, the car is parked, and the device uploads the qlog + qcamera first because they are small enough to fit in a few megabytes of bandwidth. The full rlog + H.265 streams stay on the device until the user plugs into Wi-Fi.

The version 0.9.8 release notes (2025-02-28) introduced "Firehose Mode for maximizing your training data uploads". That is the explicit acknowledgment that the model needs more data, faster, and the explicit opt-in for users who are willing to upload the full rlog + HEVC streams. Firehose users are the corpus. The default users are the safety net. The 332-car fleet is the data-collection platform, with an opt-in tail that does the heavy lifting.

What gets trained

The training side is not in the openpilot repository — it lives in comma's private compute infrastructure. What is in the repository is the *evidence* of what gets trained, and the evidence is the RELEASES.md timeline. Read the release notes from 0.9.6 (2024-02-27) through 0.11.2 (2026-06-15) as a paper trail of the data flywheel turning.

• 0.9.6 (2024-02): "Vision model trained on more data", "Directly outputs curvature for lateral control".
• 0.9.7 (2024-06): "Inputs the past curvature for smoother and more accurate lateral control", "Improved end-to-end bit for phone detection" (the DM model).
• 0.9.8 (2025-02): "Trained on a new dataset, including comma four data".
• 0.9.9 (2025-05): "New training architecture using parts from MLSIM".
• 0.10.0 (2025-08): "New training architecture" described in the CVPR paper, "Longitudinal MPC replaced by E2E planning from World Model in Experimental Mode".
• 0.10.1 (2025-09): "World Model: 2× the number of parameters", "World Model: trained on 4× the number of segments", "Driving Vision Model: trained on 4× the number of segments".
• 0.10.3 (2025-12): "New temporal policy architecture", "New on-policy training physics noise model".
• 0.11.0 (2026-03): "Fully trained using a learned simulator", "Improved longitudinal performance in Experimental mode".
• 0.11.1 (2026-05): "New driver monitoring model", "Improved image processing pipeline for driver camera".

Read those bullets as a *training data trajectory*, not a feature list. The vision model is being trained on more data every release. The World Model is being doubled in parameters and quadrupled in training segments. The DM model is being retrained on the new dataset. The flywheel is the thing that is being versioned. The features are the output of the flywheel.

The testing closet

The 332-car diversity is a strength for the data side. It is a nightmare for the testing side. You cannot validate a release against 332 cars in a CI pipeline. You cannot even validate it against 30 brands. So comma built a different kind of CI.

The README has the line I have been quoting since E01: "We run the latest openpilot in a testing closet containing 10 comma devices continuously replaying routes." The phrasing is the part that matters. "Continuously replaying routes" means the 10 devices are running recorded logs, not real drives. The 332-car fleet produces the logs. The 10-device testing closet replays them. The logs are reproducible. The replay is reproducible. The 10 devices are diverse (different comma hardware revisions, different car platforms, different sensor configurations), and the diversity of the *replays* covers the diversity of the *fleet* over time.

This is a CI harness that looks more like a *continuous-integration data center* than a test farm. It is the same idea as a large-scale recommender-system replay pipeline, repurposed for safety-critical code. The 10 devices are continuously running the latest openpilot against the latest recorded logs, and the regression suite is "did the new model produce a different trajectory than the previous model on this log, and if so, was the difference in the direction we wanted". The harness is the loop. The logs are the test cases. The model is the system under test.

The testing closet is also the *fast feedback loop* for the safety kernels from E02. When a contributor proposes a change to the excessive-actuation thresholds in helpers.py, the change is validated against the testing-closet logs before it ships. When a contributor proposes a change to the state machine in state.py, the change is replayed against every recorded log that ever produced a softDisabling event. The safety code is not tested in a unit test. It is tested against the same corpus the model is trained on, which is the only honest way to test safety code that is supposed to gate a learned model.

The roadmap, read as a flywheel diagram

The roadmap in docs/contributing/roadmap.md is short. I will quote the entire openpilot 1.0 section because it is the entire future of the project, in five bullet points.

openpilot 1.0 will feature a fully end-to-end driving policy.

* End-to-end longitudinal control in Chill mode

* Driver monitoring with sleep detection

* Rolling updates/releases pushed out by CI

* panda safety 1.0

Read those bullets as a flywheel diagram. "End-to-end longitudinal control in Chill mode" — the 0.10.0 Experimental-mode E2E planner moves into the default longitudinal mode. "Driver monitoring with sleep detection" — the DM model gets an additional head pose added. "Rolling updates/releases pushed out by CI" — the testing closet starts pushing OTA updates automatically, not on a release schedule. "panda safety 1.0" — the C-level safety kernel ships its 1.0 release, the counterpart to the openpilot 1.0 release.

The four bullets together describe the closed loop made explicit. The flywheel turns faster (rolling updates). The model gets better (end-to-end everywhere, not just Experimental). The safety kernel gets more rigorous (panda 1.0). The driver monitor gets more accurate (sleep detection). Each of these is a release-train improvement, not a one-time feature. The 1.0 release is the moment the flywheel is no longer an accident of architecture. It is the product.

The flywheel, drawn

flowchart LR
    A[332 cars<br/>running openpilot] -->|engaged driving| B[1-min segments<br/>rlog + HEVC]
    B -->|qlog + qcamera| C[Cloud corpus<br/>50GB+ training batches]
    B -->|firehose opt-in| C
    C -->|gradients| D[World Model +<br/>driving policy]
    D -->|new release| E[release-mici /<br/>release-tizi]
    E -->|OTA| A
    F[Testing closet<br/>10 devices] -->|regression suite| E
    F -->|safety replay| E

That diagram is the project. The arrows are real artifacts (segments, gradients, releases). The boxes are real systems (the 332-car fleet, the cloud, the testing closet). The user is at the bottom-left, driving. The model is at the top-right, being trained. The testing closet is the auditor.

What this is and what it is not

I want to be honest about what the flywheel is and what it is not, because the marketing line and the engineering line have a tendency to diverge on this point.

The flywheel is a data-collection platform. It is the most diverse in the consumer-AV industry. It is open-source at the model layer, the simulator layer, the dataset annotation layer (the comma10k dataset), and the release layer. The data flywheel turns faster than any of the closed-fleet competitors' flywheels because the user base is technical, engaged, and willing to run nightly builds.

The flywheel is not a self-driving car. The current release (0.11.2, 2026-06-15) is a Level 2 driver-assistance system, which is exactly what the README says and exactly what the MIT license with the "ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY" disclaimer says. The 1.0 roadmap is "fully end-to-end driving policy", which is the next milestone, not the destination. The destination — Level 4 autonomy — is not on the public roadmap, and comma has not committed to it.

The flywheel is a business. comma sells the comma four and the car harness. The flywheel produces the model that makes the comma four worth buying. The model is the differentiator. The flywheel is the moat. The 332 supported cars are the surface area. The data flywheel is the company.

The 332 figure, revisited

I opened the series with the 332 figure. I want to close it the same way, with the figure having acquired three more layers of meaning.

In E00, the 332 was a *framing* — the number of cars openpilot supports, the measure of the abstraction.

In E01, the 332 was an *architecture consequence* — the abstraction is sustained by the 100 Hz process bus and the C-level safety kernel.

In E02, the 332 was a *safety surface* — the safety layers have to work for every one of those cars, and the fork policy is what keeps them working.

In E03, the 332 was a *data asset* — the diversity of the fleet is what trains the World Model.

In E04, the 332 is a *flywheel surface* — the cars are the input side of a closed loop that produces the next model, the next release, the next OTA update, the next minute of driving, the next model.

The 332 is the same number at every step. The number has not changed. What the number *means* has changed at every chapter.

I came into this series thinking openpilot is a clever product. I came out of it thinking openpilot is a data flywheel that ships intermediaries on the way to a destination that the company has not yet named in public. The driver-assistance system is the shippable intermediary. The destination is something comma is building with the help of 332 cars, 30+ brands, a few hundred thousand engaged users, and a 10-device testing closet. The destination is not on the roadmap, but the flywheel that will get it there is, and you can read the flywheel in the architecture.

The 332 figure is the asset. The architecture is the engine. The flywheel is the company. The car is the input. The minute is the unit of progress. The release is the heartbeat. Three hundred and thirty-two cars. One loop. The loop is closed.

---

References:

• docs/concepts/logs.md — route/segment data model
• openpilot/cereal/services.py — service decimation table
• RELEASES.md — full version history through 0.11.2
• README.md — testing closet line and data-collection policy
• docs/contributing/roadmap.md — openpilot 1.0
• comma connect — the user-facing portal for uploaded routes
• comma10k dataset — public annotated image dataset for training