Managing Artificial Intelligence Projects: Framework for Success. Part 5
Modelling
Once data preprocessing and feature extraction are complete, the next step involves constructing a model to interpret the data. This phase encompasses planning, model selection, and extracting critical properties from the dataset. Effective model building relies on choosing suitable data processing strategies and algorithms. For instance, in predicting house prices, selecting a linear regression model might be preferable due to its simplicity and interpretable coefficients compared to more complex models like neural networks.
To ensure the model functions correctly, we need to train it using labelled data. During each iteration of training, the model is refined and optimized for better performance. The quality of the training data significantly affects the accuracy of the model. For example, in facial recognition software, high-quality, well-labelled images of faces will enhance the system's ability to accurately identify individuals compared to using low-quality or incorrectly labelled images.
Data augmentation
Data augmentation refines the dataset to address model output limitations based on initial model performance. Unlike data transformation, which targets specific variables, augmentation deals with class imbalance, feature engineering, and feature representation. The model is re-run multiple times to evaluate the effectiveness of these measures. Class imbalance is typically resolved using undersampling and oversampling techniques.
Benchmark
During the initial AI model development, an evaluation benchmark is essential. This benchmark often comes from common-sense heuristics, like human expertise in solving the problem or the success rate of human detection using the same data. Alternatively, it can be derived from relevant literature or industry standards. This benchmark not only evaluates the model but also identifies input vectors or attributes that do not fit, guiding the next steps: either developing multiple models or revisiting data pre-processing if the dataset is incomplete.
Primary metrics evaluation
The primary metrics build on the previously set performance benchmark. It's key to know what the metric represents and how it's calculated. This information is available in various sources like API documentation and case studies. Effective evaluation metrics should be accurate, robust, agnostic, and interpretable. Commonly used metrics include accuracy, precision, recall, F1 score, R-square, and root-mean-squared error. The general formula, Outcome = model + error, helps clarify that "accuracy" isn't the only valuable metric. It aids in selecting the right metric among many options. These metrics are also useful for comparing models across different parameter settings.
AI interpretability
Model interpretability (often referred to as model explainability or explainable AI) has gained prominence recently due to ethical and regulatory demands for AI transparency. However, explainability also offers substantial benefits for model development by helping AI researchers understand how specific attributes, the learning process, and model parameters collectively influence the model’s outcomes. This need for transparency is important for complex models like neural networks, where the pathway from input data to the decision output is hidden by multiple layers of computations.
In simpler models such as logistic regression, the flow of information is relatively straightforward and can be easily visualized or unpacked. However, with complex models, this is not possible. Industry across various sectors has recognized the importance of explainable AI. For example, in healthcare, IBM Watson Health uses XAI to provide understandable and reliable insights for medical diagnoses. This is needed to ensure that clinicians trust the AI’s recommendations. In finance, companies like FICO employ XAI to deliver transparent credit scoring, allowing both regulators and customers to understand why a particular credit decision was made.
Essentially, XAI methods can be split into intrinsic and extrinsic categories. Intrinsic methods are those built into the algorithm that work concurrently with the AI's functionality. In contrast, extrinsic methods are typically post-processing techniques applied to the model's output.
Given that most AI algorithms were not initially designed for intrinsic interpretability, XAI predominantly relies on extrinsic methods such as partial dependence plots, individual conditional expectation, local interpretable model-agnostic explanations (LIME), and Shapley additive explanations (SHAP).
Feasibility study
The end of the first iteration is crucial for the project. Stakeholders assess whether to continue based on viability (solving a business issue), desirability (complying with ethics or governance), and feasibility (cost-effectiveness). This assessment process is standard for all innovation projects.
Feasibility checks are critical throughout the project, and a fail-fast approach is important. All projects need a feasibility study early on. At this stage, projects don't yet fit into sprint-based agile planning. An agile approach helps prioritize tasks and engage stakeholders.
By Yesbol Gabdullin, Senior Data Scientist, MSc in Artificial Intelligence, MRes in Engineering, BEng in Electronics and Control Engineering, Microsoft Certified
The article was written for the Forecasting the Future of AI Applications in the Built Environment session of the Project Controls Expo 2024.