While performance varied by car make, model and year, there were no major issues with mileage accuracy. Unfortunately, braking detection results were not as strong.

To understand the reasons behind braking event inaccuracy, let us first take a close look at how mileage and braking events are calculated.

Mileage Calculations

An OBD device is alerted that driving has begun when it detects an electrical spike from the ignition. Ideally, as soon as the driver turns the key, the electrical spike is detected and the OBD device starts measuring mileage. Conversely, when the key is turned off, the OBD device stops measuring mileage. Nevertheless, the OBD device is always plugged in and slowly draining the car battery. During our study, we found that OBD devices did not always detect the ignition spikes in cars with older electrical systems, accounting for the deterioration in mileage tracking results in older vehicles.

Braking Event Calculations
Braking event calculations are more complicated. OBD devices take vehicle speed readings from a car's onboard computer approximately once every second. A braking event occurs if a vehicle slows down by at least 7 mph per second.

To detect a braking event, the device measures the Speed at Time Interval 2 minus the Speed at Time Interval 1 divided by the time difference between the two events and compares it to a certain threshold. The formula is (V[t2]-V[t1]/(t2-t1).

So, for example, if a vehicle is travelling 50 miles an hour in one second and 40 miles an hour in the next second, there is a drop of 10 miles per hour in a one second timeframe, triggering an OBD braking event, which is usually defined as 7mph/sec.

However, because the OBD device was not designed for continual braking event calculations, this method has some critical flaws:

1. OBD speed values have no attached timestamps. This means that interval (t2-t1) is actually unknown, although the OBD device assumes it to be 1 second. However, if there are small variations, and the interval is actually 0.7 second or 1.3 seconds for example, it will cause an incorrect calculation and braking events may be over or understated.
2. Car computers send OBD speed values as a truncated integer (a rounded number). In some cases a 10.5 mph speed value may come through as 10.0 mph and a speed of 19.7mph could come through as 20 mph. These small variances can add up to braking event oversights and in some cases, overstatement.
3. Multiple speed readings. Some vehicles send multiple speed readings at the same time, causing even further inaccuracy – particularly since there are no timestamps.

Based on initial findings, the OBD device did not appear to be worthy of a "gold standard" status. To take the analysis one step further, we asked, "Can a smartphone perform any better?"

Smartphone Platform Challenges and Requirements
Many questions have been raised about the viability of a smartphone app used as a telematics device, including:  

• What if the driver turns the smartphone off or the battery runs out?
• How can the smartphone differentiate between a braking event and an event in which the phone is simply dropped?
• How will drivers remember to press a "start" button before each trip?
• How will rides on public transportation affect UBI scoring?

Because of these variances, and based on years of extensive testing and research, we determined that a smartphone app, on its own, could not compete with the data accuracy of an OBD device. Even a highly developed and sophisticated smartphone app, with auto trip detection, battery-efficiency and no fixed position requirements would have difficulty achieving comparable data accuracy.

To compete with OBD accuracy, a smartphone platform must be able to capture all trips, remove unrelated trips and compensate for missed trips. To achieve these three goals requires more than an app. The smartphone app must be paired with powerful cloud analytics.

Five requirements for accurate smartphone UBI data measurements:

1. The smartphone app must feature very reliable automatic trip detection. The user cannot be relied upon to press a "start" or "stop" button. The accuracy of trip detection has to be above 90%. Our studies showed that only 4% of drivers would remember to press "start" each time they begin a trip if asked to do so.
2. The app must have very low battery overhead so it can continuously capture all trips without killing phone functionality. Ideally, it should have battery overhead of 10% or less.
3. The cloud analytics component of the smartphone platform must include unrelated ride filtering capability. It must capture all activity and then filter out rides in trains, airplanes, boats and taxis as well as other situations.
4. The smartphone platform must have a mechanism to compensate for missing data during times in which the phone was compromised–left at home, the battery ran out, or it was just turned off. Periods of missing data should not influence the overall UBI scoring.
5. For accurate event interpretation (i.e. to understand the difference between a dropped phone and a hard brake), all of the sensors in the phone must work together through a process known as Sensor Fusion. GPS alone is not enough.

Our findings suggested that if a smartphone platform met these five requirements, then the data collected would be on par, and in some ways better than UBI data collected by an OBD device.

Smartphone Platform vs. OBD
To further test this theory, we used three test subjects. For the purpose of brevity, I will cover the results of one test subject below. During the test period, each test subject's driving behavior was measured by both an OBD device and a smartphone platform which included a smartphone app combined with cloud analytics.

Test subject activity is shown above. The smartphone platform had reliable automatic trip detection and on average, it used 7% of phone battery per day.

The smartphone successfully captured everything that occurred while the phone was charged and turned on including passenger rides via train, boat, airplane and taxi. Cloud analytics detected and removed the passenger rides so that only driving activity was included in the UBI calculation.

Furthermore, the cloud analytics detected three days in which the phone was compromised (dead battery, left at home, left the phone turned off), accounting for the 9 missed car trips. To ensure that the overall UBI score was not skewed by missed activity, the three days missed were subtracted from the total activity and the scoring formula was adjusted accordingly.

Mileage Results

As you can see from the Mileage Results chart above, the smartphone platform results were on par with the OBD device results after the smartphone successfully flagged and removed unrelated trips and accounted for missed trips. Both the OBD device and the smartphone platform were within 90% of the actual odometer results indicating a strong measure of reliability.

Braking Event Results

As mentioned earlier, the detection of braking events is highly reliant on the collection of accurate second-by-second speed measurements. OBD devices were originally designed to diagnose engine alerts at a moment in time–not on a continuous basis. Therefore, continuous, second-by-second speed measurement by OBD devices have many inaccuracies.

Likewise, if a smartphone platform uses only a GPS device to detect braking events, it also has data flaws. That's because the typical smartphone has an average low accuracy of 10 meters measured at one hertz, causing many false positives.

Conversely, if a smartphone platform is able to merge data from the GPS, the accelerometer, gyroscope and magnetometer, it captures the majority of braking events–actually performing slightly better than an OBD device. Sensor Fusion also allows the smartphone to interpret events that occur and differentiate between a phone being dropped and acute braking.

The UBI Decision Made Easier
Based on this data, we conclude that when a smartphone app is combined with smartphone analytics and Sensor Fusion technology, mileage and braking event data is on par with OBD data accuracy as long as the five key requirements are met. Those five requirements are automatic trip detection; low battery overhead; passenger ride removal; missing trip accountability; and accurate event interpretation.

Now that we've answered the question of data accuracy, executives can look at the big picture factors of program cost, user experience and overall logistics to determine which usage based insurance approach is most viable for their organizations.