Sign up to receive 5 free days of our daily summary for tech news.

Join us for just $10/month for our analysis of Facebook's new cryptocurrency announcement on 6/18.

At Ring’s R&D Team, Security Gaps and Rookie Engineers

Jamie Siminoff had flown to frigid Kiev, Ukraine, to give a pep talk to the roughly 30 people who worked there for his fast-growing video doorbell startup, Ring. It was December 2016, and the Santa Monica, Calif., company had recently opened a satellite office in Ukraine to develop products that would use artificial intelligence and motion detection to improve home security.

At one point during the meeting, Mr. Siminoff asked the team how he could make their jobs easier. One of the engineers in the room said that to improve Ring’s software, the Kiev office needed access to customer video feeds. The information trove contained images from security cameras pointed at home entrances across the globe that could be traced back to individual customers.

On the spot, Mr. Siminoff agreed to arrange for the team to get administrative access to Ring’s web-based interface, where customer videos could be streamed, according to multiple people either present or briefed about the meeting.

The decision worried Jason Mitura, a prominent computer vision engineer in Ukraine whom Mr. Siminoff brought in to help build the team. Mr. Mitura expressed concerns to colleagues about granting access to customers’ personal video feeds to the Kiev team, which consisted largely of inexperienced engineers who lacked training in managing sensitive personal information. The risk was heightened in Ukraine, a hotbed of cybercrime. If the information were to fall into the wrong hands, it could give would-be thieves a virtual guide for when to break into someone’s home.

But Ring was burning through cash, and Mr. Siminoff was eager to demonstrate to investors and potential suitors that his startup was making progress in developing AI-powered products, former employees said. Giving the research team access to customer videos would help them to more quickly build image-recognition capability that would enable doorbell cameras to alert homeowners when a person appeared at their front door.

The security vulnerability was one of several problems with Ring’s Ukraine office. The engineers hired for the team often had little experience, former employees said, hindering the development of AI-powered doorbell cameras.

Shortly after the December 2016 meeting, Ring’s Ukraine employees were given access to a database of customers’ video history. A little more than a year later, Ring was acquired by Amazon for $853 million.

In an interview, Mr. Siminoff said he didn’t recall the 2016 meeting with the Kiev team, and that he delegated to senior managers the decision to make customer video feeds available there. Mr. Siminoff disputed the notion that customers’ private information was more vulnerable in Ukraine.

“I believe humans are the same everywhere,” Mr. Siminoff said when asked about sharing customer videos with the team in Ukraine, which he referred to as “one of my prouder places I’ve been involved in.” He added that the core principles of Ring’s business were “privacy, security and consent,” and that that’s been the case “from day one.”

To understand Ring’s operations in Ukraine, The Information interviewed 24 current and former employees as well as business partners and others familiar with the company. The Information also reviewed scores of internal documents, including employee training guidelines, PowerPoint presentations and communications sent to investors. What emerged is a picture of a startup that cut corners in the pursuit of growth.

‘Higher Risk’

Sharing personally identifiable video with employees in Ukraine left the data more vulnerable to being hacked, digital security experts said. For months after the Ukraine office opened, videos were transmitted there unencrypted, according to multiple people with knowledge of the arrangement, adding to the risk.

Fed by weak state oversight and proximity to criminal actors in Russia, Ukraine is viewed by security experts as both a source and a victim of cyberattacks. Hackers targeting the country’s power grid have caused widespread power outages on several occasions. Earlier this year, the U.S. Justice Department indicted a Ukrainian-led global cyber criminal group accused of stealing people’s identities and personal information.

“I would certainly place sending data to and from Ukraine” as “higher risk” than operating elsewhere, said Joshua Motta, a former CIA analyst who now runs a cybersecurity insurance provider. “Ukraine would be on a list of countries where I’d advise people to be more careful about what it is they share, and who it’s shared with.”

A spokeswoman for Ring said customer videos are encrypted “today” but didn’t respond to questions about when Ring began encrypting the videos. Encryption is a standard security measure that makes it more difficult for people outside the company to gain access to the material.

“We take the privacy and security of our customers’ personal information extremely seriously,” the spokeswoman said. She added that Ring trains its AI software only on videos that customers shared with the company as part of its neighborhood watch feature, a relatively small subset of Ring’s user videos, as well as a “small fraction” of users who have provided the company with written consent to use the videos for machine-learning purposes.

But former employees said that wasn’t always the case, and that when the Kiev office was launched, customer videos were widely shared there. It couldn’t be learned when Ring began to restrict this access. Ring’s terms of service don’t inform customers who opt in to the community watch feature that their videos are used for image-recognition research. Mr. Siminoff said he believed Ring’s disclosures to customers were sufficient.

Ring has had other issues with security. As The Information reported earlier this year, a software flaw allowed former users of shared accounts to continue to view doorbell video.

Ring has added additional security measures since being acquired by Amazon in April. Employees in Ukraine are no longer allowed to download and store videos on their computers, for example. An Amazon spokeswoman didn’t respond to questions about security measures, or what due diligence the company conducted prior to acquiring Ring.

Different Approach

Ring’s approach to security stands in contrast with some of its rivals, including Google’s Nest Labs, Canary and August Home. Nest, Ring’s primary competitor, relies only on an in-house lab and cameras kept at employees’ homes to train its AI, which works similarly to Ring’s. Nest customers can train their devices to recognize their own faces, but a company spokesman says it doesn’t use customer video for training or product improvement, and that employee access to video feeds is “extremely limited.”

August’s doorbell camera doesn’t currently feature image recognition. Only a handful of its engineers, all of whom are based in San Francisco, have access to customer videos. Canary’s algorithms, meanwhile, are trained not with customer video, but a combination of open-source data and “internally generated content,” a spokeswoman for the company said.

In addition to storing users’ video history, Ring stores metadata, including home Wi-Fi network information and timestamps for when motion is detected, internal documents viewed by The Information show.

“Video coming from a person’s home is the most sensitive kind of personal information you can imagine,” said Jacob Snow, a technology and civil liberties attorney with the American Civil Liberties Union. “The company should be telling customers how they are using their data, and what business or commercial purposes they are using it for.”

Current and former Ring executives said they aren’t aware of customer videos ever being used inappropriately by company employees. A Ring spokeswoman said the company has a “zero tolerance for abuse of our systems, and if we find bad actors who have engaged in this behavior, we will take swift action against them.”

Move Overseas

Founded in 2012, Ring quickly became a market leader in the nascent video doorbell industry. Mr. Siminoff spent much of his time pitching the company to investors, securing more than $280 million in funding from prominent backers including DFJ Growth and Goldman Sachs Investment Partners. At one point, the company was valued at more than $1 billion.

Yet the rapid growth masked problems at the company that went beyond lax security, current and former employees said. Much of the research and product development was carried out by inexperienced engineers who lacked the skills to develop products, they said.

As Ring grew, Mr. Siminoff decided that the company needed to introduce advanced features such as motion detection and, eventually, image recognition to Ring’s doorbell cameras to stay ahead of the competition and attract new investors.

About six months before his first visit to the Ukraine office, Mr. Siminoff began laying the foundation for an engineering team to work on these initiatives. It was the summer of 2016, and at the time there were more than 500,000 smartphone-connected Ring doorbells in the world, internal documents show, the majority in the U.S.

Dubbed Ring Labs, the initiative was to be based in San Francisco. The idea was to build a team that could vacuum up the vast amount of doorbell video feeds being recorded by customers and use the data to train an algorithm that could notify users anytime there were people or unusual activity outside their home.

But doing so would be expensive. At the time, Ring was burning between $10 million and $12 million per month, according to people with direct knowledge of the company’s finances. To gain traction in machine learning without a large upfront investment, Mr. Siminoff moved the company’s efforts to Ukraine.

The move allowed Ring to hire engineers at a fraction of the cost of expanding in the U.S., while giving the company a presence in the competitive AI sector. The Ukraine team eventually grew to more than 500 people, most of whom are employed as contractors, employment documents show.

Lack of Experience

From the beginning, there were concerns about the Ukraine engineering team.

Denys Popov, a software engineer who opened the office with Mr. Mitura, its CEO, was assigned the responsibility of building the team, even though he had no recruiting experience. Mr. Popov conducted more than 300 interviews in less than four months, personally hiring 78 people. He described Ring Ukraine’s workforce, which operates largely independently from Ring’s headquarters, as “young and ambitious,” but said many early employees didn’t have the experience needed to create a successful product.

This was particularly problematic for the image recognition team. Cameras that use artificial intelligence to spot potential intruders by registering unfamiliar faces or patterns are considered the holy grail of home security businesses. In practice, it has proved extremely difficult to build a reliable AI-based tool.

For years, Ring has used off-the-shelf motion detection software in conjunction with its own image-recognition technology. But the system often didn’t work the way that Mr. Siminoff had envisioned, former employees and business partners said. Users routinely complained to customer support about receiving alerts when nothing noteworthy was happening at their front door; instead, the system seemed to be detecting a car driving by on the street or a leaf falling from a tree in the front yard. There also were instances where the doorbell camera would suddenly stop recording altogether.

“At Ring, the AI isn’t very smart,” said a former Ring Ukraine customer-support specialist. “We just apologized and said, ‘We’re working on it.’ Sometimes, it couldn’t recognize a human from a dog.”

For a brief period prior to the Amazon acquisition, it appeared that Ring was trying to address the problems with the software the Kiev team was developing, and former employees said the R&D work showed promise. A team of engineers working on the project demonstrated it to a delegation of Amazon representatives in November 2017.

“[Amazon] liked what we were doing,” a former employee with direct knowledge of the effort said. But as the talks between Ring and Amazon accelerated, Ring’s leadership seemed to lose interest in the project, neglecting to ask the Ukraine office for updates and communicating less frequently.

Once it was clear Amazon was going to make the acquisition, Ring scaled back plans to implement the team’s work, said former employees with knowledge of the project. “Why bother?” asked a former Ukraine employee. In February, the acquisition was announced.

Under Amazon, Ring has kept the bulk of its R&D work in Ukraine, but the team continues to struggle to develop reliable image-recognition software, former employees said.

Experiments the Ukraine team has done using customer videos are, for the most part, "still pretty far from implementation" into Ring's products, said Vitaly Bondarenko, a former vice president of engineering in Ukraine.

Limited image recognition is currently available in half a dozen Ring products, including its flagship video doorbell as well as spotlight and floodlight cameras. But the products still mistake humans from other objects, former employees said, prompting frequent user complaints.

Mr. Siminoff acknowledged that users often complain about false positives, and said it “takes a long time” to build a system that’s as good as the human eye at motion detection. “I think we’ve made great strides in this,” Mr. Siminoff said.

Ring plans to release updated AI software early next year, which it says will improve motion verification and person detection. In May, Amazon filed a patent describing how Ring’s doorbell could use facial recognition, an area in which Amazon has invested heavily, to monitor neighborhoods and report suspicious activity to law enforcement. The patent lists Mr. Siminoff as its inventor.

Amazon didn’t comment on Ring’s efforts with image recognition and motion detection. The Ring spokeswoman said that the patent doesn’t necessarily reflect “current developments to products and services.”

Persistent Gaps

After a visit by Amazon representatives to the Ukraine office in May, Amazon moved to restrict access to sensitive customer information, former employees said, requiring a digital key that could only be used from within the Kiev office.

But employees quickly found ways around the restriction. “We had to apply and get access. The Ukraine office wasn’t comfortable with this, so we found a workaround,” a former Kiev employee said. “Workers could then access the system from any computer, at home or anywhere.”

The former employee said that once, when he had an issue logging into his account, he wrote to a colleague based in the U.S. to see what happened. When the head of his team in Kiev found out, he was scolded for alerting Ring’s U.S. personnel of the issue and told that his access could be restored from Kiev by essentially giving him administrative privileges, without any help from Ring headquarters.

Neither Amazon nor Ring responded to multiple requests for comment on the matter.

As recently as October of this year, even low-level employees were able to access user information and video shared by customers.




‘Video coming from a person’s home is the most sensitive kind of personal information you can imagine.’

You might also like ...

AI

AI Hiring Frenzy Settles Down

By Kevin McLaughlin

Enterprise Amazon Cloud

Larry Ellison Returns to War (and Diaper Changing)

By Kevin McLaughlin

Amazon E-commerce Policy

What’s Amazon’s Share of Retail? Depends Who You Ask

By Priya Anand

Autonomous Vehicles

SoftBank’s Ronen Says Self-Driving Market “Big Boys’ Game”

By Cory Weinberg, Aaron Tilley, and Kevin McLaughlin