By one measure, Alphabet’s Waymo told California officials that last year it doubled the reliability of its autonomous vehicle prototype software in public road tests in the state, largely in suburban Silicon Valley where it is headquartered. By the same measure, known as “disengagements,” or when a human sitting behind the wheel of the prototype must take over the wheel to prevent a dangerous situation, General Motors’ Cruise showed road-test data that were half as good as Waymo’s and Apple’s self-driving car prototype data were 10,000 times worse.
What does it mean? Not much. The disengagement disclosures, which self-driving car developers have submitted to the California department of motor vehicles in January for the past three years, leave a lot of wiggle room for companies to decide whether certain human takeovers qualify as a disengagements. (The DMV prematurely published the reports this morning and quickly pulled them from the web, but Reddit user Walky22Talky published much of the data.)
Furthermore, the closely watched disclosures have an apples-and-oranges problem: Every company makes frequent changes to its software and tests multiple versions simultaneously, meaning some cars in a fleet might be performing a lot better than others. And each company tests in different locations. Plus, disengagements don’t tell us about capabilities of the prototypes and how quickly they recognize the cars and pedestrians around them and predict what they will do, known as latency. They do show the relative maturity of various programs: Waymo drove more than 1.2 million miles in California last year while Apple’s cars drove 80,000 miles.