openpilot 0.11.1 - comma.ai blog

Blog

Jobs

Got questions? Drop a comment and the team will answer!

Driver MonitoringPermalink

New model with good phone detectionPermalink

The LM GT3 model (#37425) improves phone detection and is the first openpilot DM model trained using labels generated by a locally hosted large Vision-Language Model (VLM).

In this model, the driver’s phone-usage ground truth is generated by asking a VLM a YES/NO question and using the output probability of the YES token. The language model is set to predict only one token. The previous approach relied on a custom classifier network trained on a set of hand-labeled driver images.

To evaluate this new pipeline, we hand-labeled a 1k validation set of driver images with drivers either using or not using phones, called the phone_using set. We also curated a smaller set named phone_mounted, consisting of 80 driver images with a phone mounted on the dash but not in use. The phone_mounted set represents difficult negative samples.

The VLM notably reduces False Positives and improves detection of active handheld phone use as True Positives, while keeping all mounted phones as True Negatives.

ROC curves for phone_using on the 1k validation set. phone_mounted not depicted since all are negative samples, whose false positives reduce to 0 when phone_using true positive rate is ~50%.

With the model deployed in the field, we were also able to see its positive effects on safe driving habits when using openpilot. We compared phone usage time, as detected by the improved model, across the same group of devices before and after this model change. We saw a significant drop when openpilot was enabled, and even observed a small drop when it was not enabled, suggesting that drivers did reduce their phone usage overall.

Phone usage behavior progression on the same sample group of 71 devices running master/nightly/staging: ‘before’ segments sampled from 60-day window before new DM model; ‘after’ segments from 20-day window after. All data re-evaluated offline using the same model.

Improved driver camera image qualityPermalink

The two road cameras use a different image processing path on the ISP than the driver camera, and have had better image quality due to some missing pieces on the driver camera processing side. We have now enabled equivalent tone mapping blocks (#37873), as well as moved from on-sensor binning to post-demosaic downscaling (#37876). This means the driver camera will get the same high level of dynamic range and pixel details as the road-facing cameras.

Driver camera pictures old vs new, day and night.

Fewer alerts where there shouldn’t bePermalink

Alerts from looking around are reduced during active maneuvers (#37751). When steering through a turn, the head pose yaw tolerance is widened in the direction of the steering angle, so glancing where the car is going no longer triggers an alert.

Adaptive head pose thresholds during turns. This drops ~7% of all alerts. Angles shown are for illustrative purposes only, as these thresholds are also based on the scene and speed.

This PR also extends the standstill exemption for DM alerting, so that an audible alert no longer unnecessarily kicks in the moment the car starts moving.

Improved system transparencyPermalink

The driverMonitoringState message has been restructured (#37799) to be both more readable and have a cleaner abstraction boundary. Downstream processes, such as selfdrived, can access top-level states and handle alerts without needing to know about DM policy details. Debugging with the new logs should also become easier.

In the comma four onroad UI, we added a color change to the DM icon to subtly indicate distraction (#37826) before any alert is triggered. It also serves as feedback when the driver gets an audible DM alert and tries to look back at the device.

DM icon turns orange as soon as alert countdown starts, and returns to green as attention is restored.

The DM icon is now calibrated to the road instead of showing raw pose from the model (#37149), so the green arc will be at the top of the icon when the driver is looking straight ahead.

Comparison of DM icon behavior when driver is looking straight ahead.

Thermal improvementsPermalink

Reducing thermal onroad blocksPermalink

Direct sunlight on a parked comma four can heat the device above the fixed 75°C onroad threshold, temporarily blocking openpilot from going onroad even though the device would cool down once the car started moving.

This release raises the threshold to 85°C and updates the comma four onroad thermal bands (#37891). Fleet analysis projected that raising the threshold from 75°C to 85°C would reduce affected devices by ~90%.

Number of comma four devices thermally blocked over the last 30 days. Raising the threshold from 75°C to 85°C reduced affected devices from 125 to 11.

Field observationsPermalink

Thermal blocks were not isolated to a specific...