Managing Artificial Intelligence Projects: Framework for Success. Part 1
This article addresses the challenges related to managing AI projects and builds on the latest academic research, industry best practices and experience in successfully delivering AI projects by proposing an AI project management framework. In addition, it gives an overview of the team required to ensure the delivery of the AI project.
Content
Part 1. Introduction / Existing framework and best practices
Part 2. Proposed framework for managing AI projects
Part 3. Problem statement / Compliance assessment / Technical literature review
Part 4. Data collection / Data exploration / Data pre-processing / Feature engineering
Part 6. Model deployment / Post-deployment review / Operationalization
Introduction
Artificial Intelligence (AI) is increasingly becoming critical for organizations to enhance customer value, boost productivity, and uncover insights. The AI revolution is recognized as a crucial driver of innovation across various sectors, including healthcare (Wessel et al., 2019), banking and financial services (Wong et al., 2012), manufacturing, education, and many others. Furthermore, advancements in AI technology continue to expand our capacity to harness data for value creation, with significant progress in areas like machine learning, robotic process automation, natural language processing, and computer vision.
Many AI techniques are not new; for example, neural networks date back to the 1980s (Mead and Ismail, 1989). However, recent advances in data availability and computing power have vastly expanded AI's applicability (Bernardi, Mavridis and Estevez, 2019). AI is now integral to critical business operations. This rise has introduced new lifecycle challenges for AI systems, from conception to retirement. These projects must be planned, tested, debugged, deployed, maintained, and integrated into complex systems, much like traditional software.
Leading AI and tech companies are revolutionizing their development processes and devising new solutions. Their experiences offer valuable lessons for other organizations and guide industry-relevant research. This is crucial for regulated industries like healthcare and financial services. These industries must comply with evolving regulations and ensure product usage is compliant. Therefore, new processes are needed to ensure AI systems meet all required standards.
Despite the benefits AI brings to organizations, managing AI projects is a very challenging endeavour. One report highlights that 85% of AI projects fail to achieve the value businesses expect (DeNisco Rayome, 2019). While some research focuses on the impacts of AI on users (University of Cologne et al., 2021) organizations (Chinese University of Hong Kong et al., 2021), and industries (Lou and Wu, 2020), there are only a limited number of studies available to help practitioners manage their AI projects effectively. For instance, research has explored the conflicts that can occur between domain experts and developers during AI system development, as well as issues related to data accessibility (Universiteit Amsterdam et al., 2021).
When managing AI projects, practitioners cannot rely solely on their previous management experience because of a unique challenge that sets them apart from other IT projects - AI workflow. The AI workflow is characterized by sequential dependencies, significant feedback loops, and numerous cycles of data exploration and experimentation (Amershi et al., 2019). The nature and quality of data heavily influence the fine-tuning of these models, making it difficult to predict, plan, and manage the experimentation cycles.
Existing frameworks and best practices
Describing the renewed interest in AI and machine learning—which are practical forms of data mining and computational intelligence—the Cross-Industry Standard Process for Data Mining (CRISP-DM) has become widely recognized in both academia and industry as the go-to method for using AI in decision-making. CRISP-DM includes six phases: understanding the business, understanding the data, preparing the data, modelling, evaluating the results, and deploying the solution.
Building on the success of CRISP-DM, various adaptations have been suggested, such as those focusing on automation, geographically distributed teams, context diversity, and knowledge sharing, as well as versions tailored to new fields like big data, cybersecurity, and fintech. More recently, as terms shifted from "data mining" to "data science" and "data analytics," the Team Data Science Process (TDSP) and Microsoft's best practices have gained popularity over CRISP-DM.
TDSP offers a structured approach to managing data science projects that use machine learning or predictive analytics. It's seen as a blend of the SCRUM framework and CRISP-DM. Microsoft's best practices emphasize three key insights for AI projects: the need for a thorough data discovery and management process, the limitations on customization and reuse for different scenarios, and the lack of modular AI components, which requires the entire team to collaborate closely on all aspects of the project.
Based on the understanding of life cycles and methodologies, I found that CRISP-DM, TDSP, and Microsoft's best practices are the main frameworks used in AI projects today. This shows a clear need for a life cycle approach specifically designed to tackle the unique challenges of creating and managing AI solutions.
Currently, the technical specifics of each step in the process are often left unaddressed. Despite AI development being a collaborative effort, the roles and required expertise of team members are not clearly defined. Furthermore, the inclusion of pre-trained models, third-party code repositories, AI ethics, and governance frameworks is often missing in existing development methodologies.
This paper addresses these limitations by introducing a AI project management framework, which outlines the design, development, and deployment stages of AI systems and solutions.
The AI project management framework benefits from our experience in industry working with clients across various sectors and functions. This work builds on the latest academic research, industry best practices and experience in successfully delivering AI projects by proposing AI project management framework.
By Yesbol Gabdullin, Senior Data Scientist, MSc in Artificial Intelligence, MRes in Engineering, BEng in Electronics and Control Engineering, Microsoft Certified
The article was written for the Forecasting the Future of AI Applications in the Built Environment session of the Project Controls Expo 2024.