6 Lessons Learned from the AI Projects Graveyard
Blog
Blog

6 Lessons Learned from the AI Projects Graveyard

Most common pitfalls of unsuccessful AI projects and actionable lessons learned so you can avoid making the same mistakes.

6 Lessons Learned from the AI Projects Graveyard
Ken Stibler

Artificial intelligence and machine learning technologies have promised to unlock endless possibilities for decades—everything from better customer selection to more efficient operations to robots that think like humans. However, research shows that AI efforts consistently struggle to deliver on these promises. Gartner estimates that 85% of big data projects fail. What's behind this failure rate? According to Dimensional Research, 96% of organizations engaged with AI and machine learning say they have faced data quality, data labeling, and building model confidence problems, causing eight of ten AI projects to stall.

Given the numerous potential pitfalls, Devron's Founder & CEO, Kartik Chopra, sat down with Publicis Sapient AI Lab's Lead Data Scientist, Larry Berk, for a webinar to discuss the most common reasons AI projects fail and the lessons we can learn from them.

Kartik is a former Technical CIA Intelligence Officer with a breadth of experience in machine learning projects, including covert financial intelligence, geospatial, and distributed computing. Larry is a former Principal Data Scientist at Hitachi with extensive decision science experience across industries, including IIoT and intelligent data centers. 

1. Understand Your Why: Spend More Time Framing the Question

AI projects are doomed from the start if you focus on the wrong thing. Often, you'll be presented with a fully-formed problem and a presumed solution. If we can solve for X, it will improve Y. However, it can be dangerous to proceed down this laid-out path. You could spend months solving a problem that doesn't address the underlying business need.

Let's take a look at an example. Early in his career, Larry worked for a company that built network design software for trucking companies. Many trucking companies that deliver small parcels operate just like airlines—they need to consolidate their routes through hubs to optimize costs and preserve as much of their margins as possible.

In this case, Larry was asked to automate driver dispatching and scheduling, preferably in real-time, because driver turnover was exceptionally high. Therefore, the supply of drivers to fulfill routes was ever-changing. Assuming the problem was a simple dispatch issue, Larry's team built an algorithm that improved the efficiency of the dispatching process. Unfortunately, it didn't make a difference to the bottom line because driver turnover remained high.

After months of work, the team realized they had solved the wrong problem. In fact, optimizing the matching of trucks to drivers only made driver retention worse because it forced drivers to make fewer stops and complete longer hauls away from home. The real problem was that drivers were hauling very low-margin freight, making optimizing margins difficult. Therefore, they should have been focused on finding the right freight to optimize margins instead of putting drivers in trucks. 

As a data scientist, it's essential to take a step back and understand what problem you are actually solving and why. Framing the question properly to address the business need is the foundation on which your entire effort is built.  Ensuring that your team takes the time to explore the problem upfront will prevent you from wasting time and money pursuing the wrong question. 

2. Explore More Data & Test Your Understanding of the Problem

In the same trucking dispatch example, the data science team's initial assumption was that the underlying issue was efficiently matching drivers with demand. This fix-mindedness resulted in narrower data collection and a shorter period of exploratory data analysis (EDA). While this saved time, it also meant key assumptions were left unchecked.

To combat this, you should also expand the scope of your project to allow for alternative hypotheses. For Larry's team, this expansion could have improved their understanding of the problem and enabled them to pivot faster to a more practical solution. 

3. Build Privacy & Security into Data Acquisitions

While working with the CIA, Kartik faced life and death situations that required near-immediate insights from highly distributed and air-gapped data. Situations like this can result in a unique quandary: do you choose data privacy and security but delay insight, or do you sacrifice privacy for faster development and higher-performing models? 

Although someone's life may not always be on the line, this challenge is representative of many AI projects. To build effective models, you need access to a substantial amount of training data. But privacy and security restrictions can delay your access to data and thus delay your results. 

Rather than waiting for manual data access or navigating through numerous data silos, you can use a distributed data science platform like Devron. Devron leverages federated machine learning and privacy-preserving technologies to unlock access to data where it resides. The decentralized data architecture maximizes data privacy without compromising the accuracy of your predictive models.  It also has the ability to provide insight at the edge, sooner.

4. Leverage Decentralized Computing to Shorten Time Spent Finding, Obtaining, & Blending Data

It is said that data science is 80% cleaning data and 20% complaining about cleaning data. Unfortunately, this saying is painfully true for both Kartik and Larry. 

Although, as with the trucking example, choosing to cut corners on the amount of data used can also compromise the outcomes. Meanwhile, in the CIA example, critical operations can't afford to wait for data access and months of cleaning.

Time can therefore be very impactful in a competitive situation, and the mishandling of data—especially in today's climate—can result in compliance, security, and even reputational and franchise risk. With decentralized model training, the traditional extract, transform, and load (ETL) process is replaced with a faster, local deployment approach that bypasses data movement and centralization to train on raw data at the point of collection. 

5. Iterate Faster Instead of Waiting for a Big Reveal

When Larry's data science team developed their optimized dispatch algorithm, they designed and tested it in a vacuum. The siloed team had very few interactions with the customer and gathered very little feedback. The consequence was confusion and disappointment from both sides when the grand reveal of their model failed to solve the underlying problem.  

Data scientists need visibility into the business requirements and frequent communication with non-technical stakeholders. Adopting a rapid experimentation approach while working alongside the customer can help you remain focused on the business problem and profitability when questions do arise.

6. Deemphasize Elegant Code

Another byproduct of a team that focuses on the big reveal is an over-emphasis on elegant code. Unfortunately, focusing too much on this output can also cause you to lose sight of the business outcome you're trying to achieve. Instead, try to adopt an explorer or problem-solver mindset to push your data science team beyond the code. Such approaches can ensure all outputs are tailored to efficiently use resources and optimize for the original "why" of the projects.

________________________________________________________

These six common AI project failures illuminate the continued complexity of the data science process despite a proliferation of platforms and frameworks. Whether spending enough time on EDA or managing cross-functional communication, ensuring the success of a machine learning project means separating the art from the science of data work to mitigate human pitfalls. Understanding and addressing these cognitive heuristics can keep the core 'why' in focus and create processes that keep projects on track. 

Even when the human element is optimized for success, the speed and safety of data access appear to be a fundamental tradeoff. Luckily, a new paradigm in data science enables faster experimentation, lowers risk, requires less overhead, and delivers more consistent insights. To learn more about this new decentralized paradigm of model training, watch the recording of our webinar, Accelerating AI Business Value