This morning, as it’s done for the past several years, the California Department of Motor vehicles released a batch of reports from the companies piloting self-driving vehicles in the state. By law, all companies actively testing autonomous cars on public roads in California are required to disclose the number of miles driven and how often human drivers were forced to take control of their vehicles, otherwise known as a “disengagement.”
Formally, the DMV defines disengagements as “deactivation of the autonomous mode when a failure of the autonomous technology is detected or when the safe operation of the vehicle requires that the autonomous vehicle test driver disengage the autonomous mode and take immediate manual control of the vehicle.” Critics say it leaves wiggle room for companies to withhold information about certain failures, like the running of red lights in order to avoid crosswalks. But while there’s credence to this, in lieu of federal rules, the reports are one of the few metrics by which the progress of the industry’s pack leaders can be compared.
According to the DMV, AV permit holders — of which there are 60 — traveled approximately 2.88 million miles in autonomous mode on California’s public roads during the reporting period — an increase of more than 800,000 miles from the previous reporting cycle. Currently, 64 companies have valid permits to test autonomous vehicles with a safety driver on California public roadways.
Waymo
Waymo drove 1.45 million miles in California in 2019, eclipsing the company’s 1.2 million miles in 2018, 352,000 miles in 2017, and 635,868 miles in 2016. Indeed, it was a year of mileage milestones for the Alphabet subsidiary, which passed 1,500 monthly active riders in Phoenix, Arizona — the only state of the nine in which Waymo has driven where its commercial taxi service, Waymo One, is available. Waymo announced earlier this year that its autonomous Chrysler Pacificas and Jaguar I-Pace electric SUVs have driven tens of billions of miles through computer simulations and 20 million miles on public roads in 25 cities (up from 10 million a year ago).
The company’s disengagement rates, on the other hand, dropped from 0.09 per 1,000 self-driven miles (or one per 11,017 miles) to 0.076 per 1,000 self-driven miles (one per 13,219 miles). That’s compared with 2017, when Waymo reported 63 engagements for the entire year.
The improvements are perhaps partly attributable to Waymo’s AI data-mining techniques inspired by Google Photos and Google Search, as well as the company’s ongoing collaboration with Alphabet’s DeepMind on AI techniques inspired by evolutionary biology. DeepMind’s PBT (Population Based Training), which starts with multiple machine learning models and replaces underperforming members with “offspring,” managed to reduce false positives by 24% in pedestrian, bicyclist, and motorcyclist recognition tasks while cutting training time and computational resources in half.
Separately, Waymo says it’s currently developing its fifth-generation Waymo Driver in Silicon Valley, San Francisco, and Los Angeles, and it says it runs disengagements through its simulation program. Later this year, it plans to share more on the safety framework it’s developing.
Waymo hasn’t shared the number of customers who’ve ridden in its fleet of over 600 vehicles to date, but it said last December that over 1,500 people are using its ride-hailing service monthly and that it’s tripled the number of weekly rides since January 2019. Additionally, it revealed that it’s served over 100,000 total rides since launching its rider programs in 2017, and it reiterated that, shortly after announcing a partnership with Lyft to deploy 10 cars on the ride-hailing platform in Phoenix, a portion of its self-driving taxis no longer have a safety driver behind the wheel. Completely driverless rides remain available only to a “few hundred” riders in Waymo’s Early Rider program, the company says.
In a move that laid bare Waymo’s ambitions for the lucrative freight transportation industry, the company recently announced it will begin testing Peterbilt trucks retrofitted with its tech stack on “promising” commercial routes in Texas, New Mexico, San Francisco Bay Area, Michigan, Arizona, Georgia, and on Metro Phoenix freeways (as well as on the I-10 between Phoenix and Tucson). This came after Waymo began mapping the streets of Los Angeles to study congestion and expanded testing to highways in Florida between Orlando, Tampa, Fort Myers, and Miami.
GM Cruise
Cruise Automation, the GM subsidiary that’s estimated to be worth over $14.6 billion, reported a significant uptick in miles driven. Its retrofitted Chevy Bolts racked up 831,040 miles in 2019, up from 447,621 miles in 2018, 131,676 miles in 2017, and 9,776 miles in 2016.
Cruise recorded disengagements for the first half of 2019 separate from those recorded in the second half of 2019, reflecting what it says was a combination of upgrades and revamped testing procedures that drastically cut down on disengagements per mile. In the first half of 2019, Cruise reported 43 disengagements (or 7,635 miles per disengagement) in 328,285 total miles driven, compared in the second half of 2019 with 25 disengagements (or 20,110 miles per disengagement) in 502,755 miles. That works out to 68 total disengagements in 2019, versus 86 disengagement in 2018 (once every 5,205 miles) and 105 disengagements in 2017.
For a frame of reference, that’s across the roughly 180 self-driving vehicles Cruise was operating in California as of early 2019, up from 130 in June 2017.
Dan Ammann, Cruise’s CEO, recently gave a glimpse at the progress it’s made toward a fully self-driving vehicle fleet. During GM’s investor day conference in New York, he revealed that Cruise has reduced the amount of time between software updates while cutting the time it takes to train its AI models by 80%. Major new firmware rolls out up 45 times more frequently than before, and as often as twice weekly.
Cruise runs lots of simulations across its suite of internal tools — about 200,000 hours of compute jobs each day in Google Cloud Platform as of April 2019 — one of which is an end-to-end, three-dimensional Unreal Engine environment that Cruise employees call The Matrix. Over 30,000 instances spin up daily across 300,000 processor cores and 5,000 graphics cards, each of which loops through a single drive’s worth of scenarios and generates 300 terabytes of results.
Cruise operates an employees-only ride-hailing program in San Francisco called Cruise Anywhere that allows those who make it beyond the waitlist to use an app to get around mapped areas. Separately, building on the progress it’s made so far, Cruise has a partnership with DoorDash to pilot food and grocery delivery in the Bay Area for select customers. The company also continues to make progress toward its Origin vehicle concept, which features automatic doors, rear seat airbags, and other redundant systems and lacks a steering wheel.
Apple
Apple’s ongoing self-driving effort — code-named Project Titan — made incremental progress in 2019, this year’s report shows. Compared with 2018, a year in which Apple’s roughly 52 autonomous vehicles covered 79,745 miles, its ….. drove ….. miles in total.
On the subject of disengagements, the company’s cars experienced …… disengagements or … miles per disengagement, versus 6,951 disengagements in 2018 or 1.15 miles per disengagement.
It hasn’t exactly been smooth sailing for Project Titan, which reportedly kicked off as early as 2014. Apple was only permitted to test autonomous vehicles on California roads in 2017, lagging behind rivals like Waymo. And proposed technical collaborations with BMW and Mercedes-Benz failed, as did potential alliances with Nissan, BYD Auto, McClaren, and others.
Eventually, Apple found a partner in Volkswagen, with which it produced an autonomous employee shuttle van based on the automaker’s T6 Transporter commercial platform. But in 2018, Project Titan suffered a blow when a former Apple employee was arrested by the FBI for allegedly stealing trade secrets. In 2019, it suffered another when Apple cut the staff of roughly 5,000 employees by 200, shifting some to machine learning projects.
Shortly afterward, former senior vice president of engineering at Tesla Doug Field became the leader of the Titan team, preceding Apple’s acquisition of self-driving startup Drive.ai.
Uber
Uber didn’t report much in the way of progress regarding miles or disengagements in California for the year 2019. (In 2018, it reported 2,608 disengagements per 1,000 miles, or 0.4 miles per disengagement.) That’s because in May 2018, the company announced it wouldn’t renew its permit to test self-driving vehicles in California, citing caution in the wake of a fatal accident involving one of its autonomous cars in Arizona.
Above: An autonomous Uber in San Francisco
That’s likely to change next year. In a bid to close the gap with rivals including GM’s Cruise Automation and Alphabet’s Waymo, Uber reapplied for a permit, which was granted earlier this month. It cautioned at the time that it didn’ have immediate plans to engage in autonomous driving, although it’s eyeing San Francisco as a potential site. Instead, Uber will kick off driving with trained drivers behind the wheel and notify local, state, and federal stakeholders prior to launching driverless tests.
In an S-1 filing ahead of its initial public offering last year, Uber noted that its Advanced Technologies Group — the division responsible for its autonomous transportation projects — has grown from a team of 40 Pittsburgh-based researchers in 2015 to a 1,000-person workforce spread across offices in San Francisco and elsewhere. Furthermore, it said that it’s collected data from “millions” of autonomous vehicle testing miles to date and completed “tens of thousands” of passenger trips to date. And it’s gathering map data in Washington D.C., San Francisco, Dallas, and Toronto.
Aurora
Aurora, which last year raised investments from Amazon and other totaling $530 million at a valuation of over $2 billion, reported that its lidar sensor-, radar-, and camera-equipped Lincoln MKZs (which might in the next year be swapped out for Chrysler Pacific minivans) drove 39,729 miles (26,300 manually) and disengaged 10.6 per 1,000 miles in 2019. (Aurora blames 25% of the 142 reportable disengagements on a software issue that was fixed early in the year.) That’s compared with 11.5 per 1,000 miles in 2018.
Auora says that after a year of focusing on capabilities including merging, nudging, and unprotected left-hand turns, its autonomous system — the Aurora Driver, which has been integrated into six different types of vehicles to date, including sedans, SUVs, minivans, commercial vans, and Class 8 freight trucks — can perform each seamlessly “even in dense urban environments.” As it expands its vehicle fleets for data collection, testing, and validation this year, it plans to improve how the Driver predicts and accounts for “non-compliant actors,” or people who aren’t following the rules of the road, like jaywalkers and drivers who aggressively cut into the lane.
Aurora has prioritized investment in its Virtual Testing Suite, which allows it to run millions of off-road tests a day and feed driving decisions into motion planning models, allowing them to learn from experience. Thanks to the Suite, Aurora can model tests involving pedestrians, lane merging, and parked cars; in point of fact, CEO Chris Urmson estimates that a single virtual mile can be just as insightful as 1,000 miles collected on the open road — and that in 2019, virtual testing grew to over 735,000 tests per day, an increase of over 100 times from 2018.
Aurora says that in a typical mile of driving in 2019, its vehicles encountered approximately three times more cars on the road and nearly ten times more pedestrians compared with 2018. Plus, it says that the Aurora Driver improved in its ability to nudge (i.e., move around stagnant objects, like double-parked vehicles); navigate pedestrians, crosswalks, and traffic lights; make right-on-red onto roads with speed limits no greater than 25 miles per hour; make unprotected left turns that do not involve multiple lanes; conduct lane changes; negotiate merges; and share the road with cyclists, where it slows its speed behind them or merges with them if appropriate.
On the virtual testing side of the equation, Aurora claims that the Aurora Driver completed 2.27 million unprotected left turns in simulation before attempting to perform one on the road (in September 2019). Now, that number stands at 31.28 million in simulation.
Nuro
Driverless delivery startup Nuro reported 68,762 total miles driven and 34 disengagements total (2,022 miles per disengagement) in 2019, versus 0.97 disengagements per mile (1,028 miles per disengagements) in 2018. It listed 33 vehicles in its report as having collected autonomous miles in California, though it noted that this isn’t the entirety of its fleet.
Nuro was cofounded in 2016 by Dave Ferguson and Jiajun Zhu, both veterans of the secretive Google self-driving car project that eventually spun out as Waymo. The Mountain View, California-based company has about 400 employees and 100 contract workers and has so far deployed over 75 delivery vehicles, and it plans in the coming months to test 50 vehicles on roads in California, Arizona, and Texas, with safety drivers behind the wheel.
In a step toward that goal, the company last week announced that its R2 vehicles had been granted an autonomous vehicle exemption by the U.S. Department of Transportation’s (DOT) National Highway Traffic Safety Administration (NHTSA).
The R2 features a more durable body that’s able to handle a greater variety of roads, climates, and weather conditions than the outgoing R1. Its smooth and rounded cabin, which has contours where side mirrors would otherwise be placed, creates room for bicyclists and other “vulnerable” road users while improving lateral maneuverability.
The R2 will soon join a fleet of self-driving Prius vehicles in Houston, Texas, making deliveries to consumers on public roads from partners including Domino’s, Walmart, and Kroger. Under the terms of the exemption, Nuro will be permitted to produce and deploy no more than 5,000 R2 vehicles during a two-year period, and it’ll have to report information about the R2’s operation (including the automated driving system) and conduct outreach in communities where it will deliver goods.
Tesla
Tesla reported zero miles driven autonomously on public roads in California during all of 2019, as it has for the past three years. The company says that it conducts its testing via simulation, in laboratories, on test tracks, and on public roads in various locations around the world, and that it “shadow-tests” its cars’ autonomous capabilities by collecting anonymized data from over 400,000 customer-owned vehicles “during normal driving operations.”
Telsa’s Autopilot — the software layer running atop its custom chips — is effectively an advanced driver assistance system (ADAS) that taps machine learning algorithms and an array of cameras, ultrasonic sensors, and radars to perform self-parking, lane-centering, adaptive cruise control, highway lane-changing, and other feats. The company previously announced that cars with Full Self-Driving Capability, a premium Autopilot package, will someday gain the ability to “automatic[ally] driving on city streets” and “recognize and respond to traffic lights and stop signs.”
In its most recent voluntary safety report covering Q4 2019, Tesla said that it registered one accident for every 3.07 million miles driven in which drivers had Autopilot engaged. And according to a model developed by MIT researcher Lex Fridman, its cars are estimated to have driven nearly 2 billion miles in autonomous mode,
Disengagements
In total, the DMV says that …. companies reported driving …. million miles in autonomous mode on California roads and highways, up from 48 companies and 2 million miles in 2018. About 65 companies have the DMV’s go-ahead to test self-driving vehicles on public California roads, which is different than the permits issued by the California Public Utilities Commission (CPUC) to transport passengers in autonomous vehicles. Only five companies — Aurora, AutoX, Pony.ai, Waymo, and Zoox — have permits under the CPUC’s pilot program, with Zoox being the first to receive one in December 2018.
But whether the disengagement metrics communicate anything meaningful remains the subject of ongoing debate.
Aurora, for one, has noted that they don’t adequately capture improvements or their impact over time. That’s why the company records two types internally: reactionary disengagements, where a vehicle operator disengages the system because they believe an unsafe situation might occur, and policy disengagements, where an operator proactively disengages ahead of an on-road situation the system hasn’t been taught to handle.
Urmson claims that technical or engineering velocity is a superior measure of progress because it captures advancements made on core technology. “Historically, the industry and media have turned to tallying on-road miles and calculating disengagement rates as measurements of progress,” he wrote in a Medium post. “If we drive 100 million miles in a flat, dry area where there are no other vehicles or people, and few intersections, is our ‘disengagement rate’ really comparable to driving 100 miles in a busy and complex city like Pittsburgh.”
In a blog post earlier this year, Kyle Vogt, cofounder and CTO of Cruise, posited it might be time for a new metric for reporting safety of self-driving cars. “Keep in mind that driving on a well-marked highway or wide, suburban roads is not the same as driving in a chaotic urban environment,” Vogt wrote. “The difference in skill required is just like skiing on green slopes vs. double black diamonds.”
Vogt and Urmson aren’t the only ones who have voiced their disapproval of disengagement-based safety measures.
San Francisco-based self-driving truck startup Embark, which last year released a voluntary disengagement report, declined to disclose numbers in favor of a new system of metrics “that [capture] the different scenarios” its system encounters. (Despite this, it revealed that in 2019, it tested 449,837 automated miles with 0 crashes.) “Within the company, we have migrated away from using disengagement rate as a performance metric, and to remain consistent with our internal thinking, we have decided not to release a 2019 disengagement report,” wrote Embark CEO Alex Rodrigues in a Medium post.
Apple has called on the DMV to “amend or clarify” its position on disengagement and testing without safety drivers, and in a blog post this month, Waymo wrote that “the key to self-driving technology safely improving and scaling is through a robust breadth of experience and scenario testing, represented by a wider array of data points beyond disengagement alone.”
“We appreciate what the California DMV was trying to do when creating this requirement, but the disengagement metric does not provide relevant insights into the capabilities of the Waymo Driver or distinguish its performance from others in the self-driving space,” wrote Waymo in a series of tweets today. “While most of the development, learning, and validation of the Waymo Driver comes through billions of miles driven within our simulation environments, our real-world driving experience is primarily outside of California, in markets like Detroit, Los Angeles, and Phoenix. Most of our large-scale real-world driving, which is critical for full-system validation (including validating the realism of our simulator) comes from Phoenix.”
Nuro CEO Jiajun Zhu said in a statement that the metrics for miles driven and miles driven per disengagement are “not a comprehensive measure” for technological success, business maturity, or safety. “We view the autonomous vehicle disengagement reports as an opportunity to amplify the comprehensive safety strategies used by Nuro to develop our autonomous technology,” he continued. “We also continue our engagement with regulators on how our unique vehicle design and operations prioritizes the safety of others with whom we share the roads.”
In a conversation with VentureBeat, Dmitry Polishchuk, the head of Russian tech giant Yandex’s autonomous car project, noted that Yandex hasn’t released a disengagement report to date for this reason. “We have kind of been waiting for some sort of industry standard,” he said. “Self-driving companies aren’t following the exact same protocols for things. [For example, there might be a] disengagement because there’s something blocking the right lane or a car in the right lane, and [the safety driver realizes] as a human that [this object or car] isn’t going to move.”
Stalled regulation and skepticism
Unfortunately for companies like Yandex, Cruise, and Aurora, less regulatory guidance 00 not more — seems the likelier near-future path, at least in the U.S. At CES on January 8, Transportation Secretary Elaine Chao announced Automated Vehicles 4.0 (AV 4.0), new guidelines regarding self-driving cars that seek to promote “voluntary consensus standards” among autonomous vehicle developers. It requests but doesn’t mandate regular assessments on self-driving vehicle safety, and it permits those assessments to be completed by automakers themselves as opposed to by a standards body.
Advocacy groups including the Advocates for Highway and Auto Safety almost immediately criticized the policy for its vagueness. “Without strong leadership and regulations … [autonomous vehicle] manufacturers can and will continue to introduce extremely complex supercomputers-on-wheels onto public roads … with meager government oversight,” Advocates president Cathy Chase said in a statement. “Voluntary guidelines are completely unenforceable, will not result in adequate performance standards, and fall well short of the safeguards that are necessary to protect the public.”
In the U.S., legislation remains stalled at the federal level, unfortunately. More than a year ago, the House unanimously passed the SELF DRIVE Act, which would create a regulatory framework for autonomous vehicles. But it has yet to be taken up by the Senate, which in 2018 tabled a separate bill, the AV START Act, that made its way through committee in November 2017.
There’s evidence the standstill is contributing to public consternation about self-driving cars. J.D. Power’s inaugural 2019 Mobility Confidence Index Study found that a majority of respondents harbor doubts about the technology’s robustness, with 71% saying that they’re worried about driverless system failures or errors and 57% saying they fear malicious vehicle hacks. It was roughly in line with results from a PSB Research survey commissioned by Intel last year, which indicated that nearly half (43%) of people don’t feel safe around them, and a recent CarGuru survey of 1,146 automobile owners that found that 87% wouldn’t rely on autonomous cars given the choice.
Source: Read Full Article