How automation speeds up data science projects

In 2018, Nick Elprin, CEO and Co-Founder of Domino Data Lab, said, “Sixty percent of companies plan to double the size of their data science teams in 2018. Ninety percent believe data science contributes to business innovation. However, less than 9% can actually quantify the business impact of all their models, and only 11% can claim more than 50 predictive models working in production.”

The figures for converting data science projects into corporate business success stories hasn’t changed much since and are troubling. Industry practitioners are well aware of them, and some are trying to do something about it.

“We’ve seen studies that report only 4% of companies successfully implement business intelligence (BI) and artificial intelligence (AI),” said Ryohei Fujimaki, Ph.D., founder and CEO of dotData, which focuses on data science automation for enterprises. “It naturally makes you wonder what the other 96% are doing.”

One area that Fujimaki and others focus on is better understanding customer relationships and the factors that generate customer churn.

“There is a great deal of business interest in this,” said Fujimaki. “Data science is a key to business growth if you can unlock its potential. You can predict new products and costs, and even customer churn. The insights that data science can generate cuts across all industries, whether it is pharma, aerospace, manufacturing, retail, finance, or other.”

Data science difficulty

The problem is that it is taking companies an average of two to three months to complete a single data science project.

“Data science is difficult for enterprises because it requires an interdisciplinary team to be successful,” said Fujimaki. “First, you have company ‘domain experts’ who know particular areas of the business and can assist in defining important business use cases. Data science talent is also difficult to hire. Then, you have to collect, clean, and prepare data, which can consume more than 80% of project time. You then must define different data models, algorithms and visualizations and try them out in an iterative mode, knowing that not all of them will work. Finally, when you get a strong project that meets a business case, you have to migrate the project into production. This often impacts business processes.”

At the end of this process, the company may achieve a successful AI project—but many companies are also finding that they want to add machine learning to get even more out of the initial AI work.

Adding machine learning can take another 20% to 30% of project time.”Again, you must continually test and retest, to ensure that data is accurate and that you are realizing your business case objectives,” said Fujimaki.

This is where automation can make a difference. Fujimaki gives an example use case: A large bank was assembling a data science team, and it was finding that it was taking longer to insight than was desired. What it wanted was the agility, and the ability to perform more data science projects faster. They decided to add data science automation—not to replace the data science team, but to make the team more agile and productive. Instead of performing one data science project every two to three months, the team used data science automation and was able to perform ten times that.

Data science automation

How does data science automation work?

With software, a company can automatically perform all of the data cleaning, preparation, statistical analysis, mathematics and AI engineering with minimal internal person resources. If a company wants to go beyond AI and add machine learning, it can automate the ML processes as well.

“With this capability, you still need business domain experts, data scientists, and engineers, but you can automate many of the statistical and mathematical operations of data science,” said Fujimaki. “This makes data science more sustainable in organizations, and it enables companies to cover more ground because they can provide data science products faster.

“There are many elements in this process, but data science automation can help,” Fujimaki continued. “In addition to enabling your enterprise to complete more data science projects and get products to market sooner without having to do all of the data science operations  yourself, a kind of ‘democratization’ of data science begins to occur in organizations. Now, many people who might be business domain specialists can also use the automation without having to become full-blown data scientists.”