Managing Artificial Intelligence Projects: Framework for Success. Part 7
Monitoring
In the final stage of the AI model's lifecycle, it's crucial to monitor and evaluate its performance. This evaluation revolves around three main criteria: the technology itself, the diverse individuals using it in various settings, and the value it generates.
Technology Evaluation (Model Drift and Model Staleness):
Model Drift: This happens when the AI model's accuracy drops because the data it's based on changes over time. For example, a retail recommendation system might need updates to stay relevant to shifting consumer trends. Addressing model drift involves retraining the model with up-to-date data to ensure it remains accurate.
Model Staleness: This occurs when the underlying problem or environment that the model was designed for changes. For example, a pricing model for airline tickets might become stale if significant changes occur in the travel industry due to new regulations or economic conditions. Solving this requires reevaluating the model’s architecture, inputs, algorithms, and parameters, potentially leading to the development of a new model.
People Evaluation (End-User Activity): Continuous monitoring of how end-users interact with the AI model is essential. For instance, in a hospital, evaluating how doctors and nurses use an AI-enabled diagnostic tool can provide insights into its effectiveness and user engagement. Metrics might include adoption rates, frequency of use, feedback, and requests for new features.
Value Generation Evaluation (Return on Investment, ROI): Though ROI from AI implementations isn't always immediately clear, it can be measured using various metrics:
Cost Reduction: An AI-powered customer service chatbot might reduce operational costs by decreasing the need for human agents.
Increased Revenue: A personalized marketing AI could boost sales by targeting the right customers with the right offers, leading to higher customer satisfaction and market share.
Productivity Gains: Examples include minimizing errors thanks to automated data entry in a finance department or increasing the agility of teams through optimized workflow automation.
By regularly assessing technology performance, user interaction, and overall value generation, organizations can ensure their AI models remain effective and beneficial over time.
Importance of documentation
Although not part of the AI lifecycle, documentation is a very important piece of every AI project. Each model must be documented to achieve several important objectives. Firstly, it ensures that the model complies with regulatory standards, making it easier to assess from a legal perspective. This is similar to how financial institutions document risk models to meet the requirements set by regulatory bodies like the SEC or the FCA. Secondly, thorough documentation facilitates reproducibility, which enables other data scientists to replicate and validate the model's results.
Additionally, extensive documentation promotes a holistic understanding of the model. This can often lead to improvements, as seen in software development where peer-reviewed code often results in higher quality code. Documentation also serves as an audit trail, cataloguing all actions, decisions, and versions, and providing the evidence needed for any subsequent reviews or audits.
Knowledge transfer is another critical benefit. It aids new team members or end-users, who are often not machine learning experts, in understanding the model’s intricacies. Think of how comprehensive medical research documentation can help new doctors understand the efficacy and methodology of a new treatment.
Just like how code is peer-reviewed, documentation undergoes similar scrutiny to ensure its completeness and accuracy. Although the specifics of the documentation may vary between departments, all documentation must at least meet the minimum standards defined by the model risk management framework. Some teams even take it a step further by developing their own templates for documentation.
In general, the documentation for a Machine Learning application should cover the purpose, methodology, assumptions, limitations, and usage of the model. More specifically, a Technical Model Document is typically created. This document elaborates on the model's methodology, inputs, outputs, performance metrics, and testing strategy. It also details any encountered challenges and the solutions applied, as well as the critical decisions made throughout the development process. For example, it should explain why a certain model was chosen and elucidate its inner workings to validate that it performs as intended—similar to a research paper justifying the choice of a particular experimental approach.
Even though creating detailed documentation is often seen as time-consuming, it is indispensable for ensuring the model's robustness, compliance, and ease of use. In industries like autonomous
vehicle development, where safety and reliability are paramount, the value of such comprehensive documentation cannot be overstated.
By Yesbol Gabdullin, Senior Data Scientist, MSc in Artificial Intelligence, MRes in Engineering, BEng in Electronics and Control Engineering, Microsoft Certified
The article was written for the Forecasting the Future of AI Applications in the Built Environment session of the Project Controls Expo 2024.