Sign up to receive 5 free days of our daily summary for tech news.

Q&A

Lighting a Spark Beneath Big Data

The so-called Big Data industry is a jumble of open-source software projects with such odd mascots that it resembles a box of animal crackers. There’s Hadoop (an elephant), Pig (see Porky, The) and Hive (an elephant’s head and a hornet’s body). The latest flavor is called Spark, whose logo is (mercifully) just a spark.

Spark has become wildly popular with developers in a short time, gathering more than 400 contributors last year, according to Open Hub, which tracks activity around open-source projects. It promises to compute data up to 100 times faster than the standard methods used for bits sitting inside Hadoop, the software suite sold by Cloudera and Hortonworks.

Adatao, a startup with $13 million in funding from Lightspeed Venture Partners and Andreessen Horowitz, is one of the early firms building an app that uses Spark to analyze data stored in Hadoop file systems.

Adatao CEO Christopher Nguyen argues that Hadoop forms a foundational “storage layer” for data that companies could one day unlock using Spark’s computing speed. Mr. Nguyen talked with The Information about Spark’s promise and the direction the industry is headed. Edited excerpts follow.

No subscription? You're missing out.

Join the high-powered community of tech and business leaders who rely on The Information's original news and in-depth reporting.

In the future, the data will speak for itself: ‘Your customers appear to be in three distinct segments, and here’s some information about them.’ Now I don’t need to be a great marketer with lots of industry experience to make those decisions.

Arthur P. Johnson commented on this article.
Read comments from top tech and industry leaders
Joe Lonsdale
Joe Lonsdale
Founding Partner, Eight
Chamath Palihapitiya
Chamath Palihapitiya
Founder & Managing Partner, SocialCapital
Tina Sharkey
Tina Sharkey
CEO, Brandless
Jonah Peretti
Jonah Peretti
CEO, Buzzfeed
Adam D'Angelo
Adam D'Angelo
CEO, Quora
Brit Morin
Brit Morin
Founder & CEO, Brit + CO
Dustin Moskovitz
Dustin Moskovitz
Co-Founder, Asana
Christina Miller
Christina Miller
President & General Manager, Turner
Max Levchin
Max Levchin
CEO, Affirm
Adam Mosseri
Adam Mosseri
Director of Product, Facebook
Alex Mather
Alex Mather
The Athletic
Martha Josephson
Martha Josephson
Partner, Egon Zehnder
James Murdoch
James Murdoch
Co-Chief Operating Officer, 21st Century Fox
Andrew Kortina
Andrew Kortina
Founder, Venmo
Ben Chestnut
Ben Chestnut
Co-Founder & CEO, Mailchimp
Ruchi Sanghvi
Ruchi Sanghvi
VP Operations, Dropbox
Login or Subscribe to follow the discussions happening here and real-time in our   Slack Community.