AEB: a Case Study Using comma.ai Dataset | by comma ai | MediumSitemapOpen in appSign up<br>Sign in

Medium Logo

Get app<br>Write

Search

Sign up<br>Sign in

AEB: a Case Study Using comma.ai Dataset

comma ai

6 min read·<br>Sep 10, 2019

Listen

Share

A data driven approach to evaluate driving assistance features<br>Features, such as Automatic Emergency Braking (AEB), are meant to improve the safety of the vehicle by reducing both the occurrences and the severity of accidents. Unfortunately, they usually have a long development cycle and effectiveness that is difficult to quantify.

Figure 1: Automatic Emergency Braking event present in our data set.A precise estimate of the real-world benefits deriving from a safety feature isn’t easy: car manufacturers and tier 1 suppliers are in charge of the technical development of Driver Assistance Systems (DAS), but other than Tesla, they lack the ability to collect data and feedback from the field. Car manufacturers also lack the ability to issue Over-The-Air (OTA) software updates, which is another strong limitation in developing and deploying new features.<br>On the other hand, insurance companies, government agencies (e.g., NHTSA) and independent organizations (e.g., IIHS) have large data sets available, but they are not in the position to deploy safety features or propose technical changes. Also, the available data (insurance claims, incident reports etc…) is not sufficiently detailed for in-depth technical analyses.<br>Thanks to our fleet of EON users, we have collected over 13 million miles of driving data all around the globe. Currently, our users upload over 35,000 miles each day, which is almost three times what an average car drives in the US per year. The data includes full CAN traffic of the car and outputs from the EON sensors, such as videos from the forward facing camera (plus driver’s facing camera, if user opts-in), accelerometer, gyroscope and high precision GPS.<br>Press enter or click to view image in full size

Figure 2: a comma EON installed in a vehicle.We can scan our large data set in search for driving events of any kind, as long as they are definable through signal patterns of what we collect. The real advantage of such a large data set, though, is the ability to look for events with very low occurrence. Events that happen every 100,000 miles have most likely happened over 100 times in our rapidly growing data set.<br>Finding rare events: AEB in Honda vehicles equipped with Bosch DAS.<br>An example of a rare event is the car’s driver assistance system issuing an AEB maneuver in order to prevent or mitigate a collision.

Figure 3: another example of AEB event.We have already discussed using our data to develop a FCW algorithm for openpilot in 2018. In this post we show a similar approach to evaluate the effectiveness of the stock AEB feature as the starting point to develop our own AEB algorithm for openpilot.<br>We started by looking into how many AEB events we captured in our fleet. For the sake of this study, we decided to limit the scope of the analysis to Honda vehicles equipped with Bosch DAS system: this represents the majority of the Honda vehicles sold today. This choice filters the number of driven miles that we can use in our analysis to 1.04 million.<br>In order to be able to look for AEB events in our data set, we first had to reverse engineer the vehicle’s API over CAN. We built a fake vehicle target and we staged an imminent rear-end collision while driving our 2018 Honda Accord (Figure 4). We recorded the test using a comma EON.

Figure 4: testing AEB using a dummy lead vehicle target.We repeated the maneuver to collect several cases where the vehicle issued an automatic braking maneuver. We then used our open source CAN reverse engineer tool, Cabana, to find out which message and signals carry the AEB information.<br>Press enter or click to view image in full size

Figure 5: our Cabana reverse engineering tool was used to identify the AEB interface on Honda vehicles with Bosch DAS.Once we were able to tell which driving segments include an AEB event, we scanned our data set and found 308 AEB occurrences. This is about one AEB event every 3,300 driven miles: not too rare!<br>Press enter or click to view image in full size

Figure 6: peak AEB commanded deceleration VS peak vehicle deceleration.As it can be seen in Figure 6, there is quite a high number of AEB events with a commanded peak deceleration that does not exceed 2 m/s²: this probably represents some sort of “Stage 1” AEB state, where moderate automatic braking force is issued by the system to alert and/or assist the driver during the braking maneuver, before eventually escalating to a more severe AEB event. We were left with 194 AEB events after excluding what we defined as moderate “Stage 1” AEB events from our analysis.<br>True positives VS false positives<br>There are several ways to evaluate the performance of an AEB system. We propose the following 2 key performance indicators:<br>Ratio between the number of...