Market Overview

Databricks Survey Gets to the Heart of the AI Dilemma: Nearly 90% of Organizations Investing in AI, Very Few Succeeding


Technology, Data and Organizational Silos Are Slowing AI Projects; IT
Executives Point to Unified Analytics as Key Solution to Address AI

the leader in unified analytics and founded by the original creators of
Apache Spark™, today announced the results of a commissioned survey
looking at the AI dilemma. Today, only one in three AI projects are
succeeding, and, perhaps more importantly, it is taking businesses more
than six months to go from concept to production. The primary reasons
behind these challenges are that 96 percent of organizations face
data-related problems like silos and inconsistent datasets, and 80
percent cite significant organizational friction like lack of
collaboration between data scientists and data engineers. IT executives
point to unified analytics as a solution for these challenges with 90
percent of respondents saying the approach of unifying data science and
data engineering across the machine learning lifecycle will conquer the
AI dilemma.

"Data is the fuel that powers AI. Large amounts of reliable data that
data scientists can iterate on is the key to AI success. But
organizational silos between data science and engineering cripples the
iterative model development process. And to make matters worse the
divide between today's data and AI technologies increases complexity
throughout the lifecycle further slowing down AI," said Bharath Gowda,
vice president of product marketing at Databricks. "Unified analytics
addresses this AI dilemma by providing an end-to-end analytics platform
that unifies big data and AI, while fostering better collaboration
between data science and engineering teams."

View the complete survey results within the "Conquer the AI Dilemma by
Unifying Data Science and Engineering" report found here:

The survey, commissioned by Databricks through IDG's CIO Research
Services, surveyed 200 IT executives at larger companies (1000+
employees) across the U.S. and Europe. The results speak to the
complexity and organizational confusion being creating as companies
pursue AI initiatives:

  • 98 percent of those surveyed believe preparation and aggregation of
    large datasets in a timely fashion is a major challenge;
  • 96 percent of respondents found data exploration and iterative model
    training challenging;
  • 90 percent cited the deployment of models to production quickly and
    reliably as a significant challenge
  • 87 percent of organizations invest in an average of seven different
    machine learning tools, adding to the organizational complexity

So, what will help these organizations conquer the AI dilemma? The
surveyed executives said they need end-to-end solutions that combine
data processing with machine learning capabilities. These streamlined
solutions would simplify workflows, improve efficiency and ultimately
accelerate business value.

In fact, nearly 80 percent of executives surveyed said they highly
valued the notion of a unified analytics platform. Unified analytics
makes AI more achievable for enterprise organizations by unifying data
processing and AI technologies. Unified analytics solutions provide
collaboration capabilities for data scientists and data engineers to
work effectively across the entire AI development-to-production
lifecycle. With more than 90 percent of large companies facing
data-related challenges and increasing complexity driven by an explosion
of machine learning tools, the need for platforms and processes that can
remove technology and organizational silos is more pronounced than ever.
Unified analytics provides an ideal approach for companies facing modern
AI implementation barriers.

Databricks accelerates innovation by unifying data science, engineering,
and business. Through a fully managed, cloud-based service built by the
original creators of Apache Spark, the Databricks Unified Analytics
Platform lowers the barrier for enterprises to innovate with AI and
accelerates their innovation.

About Databricks
Databricks' mission is to accelerate
innovation for its customers by unifying Data Science, Engineering and
Business. Databricks' founders started the Spark research project at UC
Berkeley that later became Apache Spark. Databricks provides a Unified
Analytics Platform powered by Apache Spark for data science teams to
collaborate with data engineering and lines of business to build data
products. Users achieve faster time-to-value with Databricks by creating
analytic workflows that go from ETL and interactive exploration to
production. The company also makes it easier for its users to focus on
their data by providing a fully managed, scalable, and secure cloud
infrastructure that reduces operational complexity and total cost of
ownership. Databricks, venture-backed by Andreessen Horowitz, NEA and
Battery Ventures, among others, has a global customer base that includes
Viacom, Shell and HP. For more information, visit

Apache, Apache Spark and Spark are trademarks of the Apache
Software Foundation

View Comments and Join the Discussion!