With Machine Learning models becoming increasingly integrated into the day-to-day operations of business, it is essential that the predictions that they make are fair and explainable.

Bias within the Machine Learning process can take several different forms. These could include data bias, model bias (otherwise known as algorithm bias) and evaluation bias. Each of these will be explored in more detail later in this blog post.

An example of where this could cause problems is in the Financial Services industry, where a model may be used to decide whether or not to offer someone a loan. If bias were to be present in the process of collecting data to train this model, it’s possible that loan applicants of a particular demographic would be less likely to get a positive outcome.

It’s important to note that bias, in this context, typically occurs unintentionally – it could be down to inconsistencies in the sample data used for training the model, too much complexity in the way a model is built, or false assumptions made when evaluating the model’s performance. Understanding the different types of bias that can occur will help businesses be more aware of potential issues and take the necessary steps to mitigate any unfair or inaccurate predictions.

Types of bias – and how to avoid them

Data bias

Data bias refers to bias introduced in the collection and processing of the data that is used to train a model. This type of bias will likely be the most familiar outside of the ML world; in simple terms, it results when the data is not truly representative of what the algorithm should be learning.

Let’s consider an example. A dataset is being prepared for training a model to predict whether someone will default on their loan. When the data scientist is selecting data for this, they select the most recent 50k defaults and the most recent 50k paid-up loans. There is no deliberate bias being introduced into the process here; however on closer inspection, 75% of the dataset is for individuals aged 35-50. Unless it is statistically accurate that 75% of the loans that the bank maintains belong to this age group, it increases the possibility that the model trained on this data will perform poorly for the minority age groups (<35 and >50).

A way that this could be solved is to carry out exploratory data analysis (EDA) before selecting the data to be used for training. This process seeks to understand the distribution of the data, taking particular care around any features that would be of concern when it comes to bias. Examples of these would be protected characteristics such as age, race and gender. Ensuring that the dataset used for training gives a true representation of the real-world data results in a much better chance of a model making fair predictions.

Model bias

When talking about ML models themselves, bias takes on a slightly different meaning. Bias in this sense can be boiled down to how well the model has captured trends in the dataset. Of course, if the trends in the dataset are incorrect to begin with, then the model has no chance of capturing the correct ones.

High bias in models is typically found when there is a high error rate, and the algorithm has not properly captured the trends. This is also known as underfitting and indicates that the model is too simple.

Low bias is accompanied by another concept within Machine Learning known as variance. A high-variance model is characterised by overfitting, where the algorithm has over-learnt the data trends, capturing noise in the data. This occurs when the model is too complex. The desired state of a model is at the point where bias and variance lines converge on a graph. This is known as the bias-variance trade-off.

One way to detect whether bias in a model is an issue is to perform feature importance analysis. This involves determining which of the features the trained model sees as the most significant when it comes to making predictions. If this analysis finds that the model deems that a protected characteristic such as age, gender or race has a significant impact on the prediction and that is incorrect, then bias has been introduced into the process somewhere.

This may just require changing how the model is trained; however, it would also be worth performing more in-depth analysis of the training data itself to ensure that bias has not been introduced there.

Evaluation bias

Evaluation bias occurs when the data being used for testing the model does not represent the real-world distribution, leading to incorrect assumptions about the quality of the model itself.

This is closely linked with data bias, because evaluation data is usually randomly sampled from the selected dataset before training using a pre-determined split (for example, 70% for training and 30% for evaluation). This would mean that if the original data selected was biased (whether intentionally or not), evaluation bias would also likely be present.

Imagine the scenario where a model has been trained on a truly representative data set – great! This means the model should have learnt real-life patterns and regularities and not been influenced by a poorly selected range of data. Now, time to evaluate the performance of it. Model metrics such as RMSE (Root Mean Square Error), Accuracy, Precision, Recall and F1 score could be used to get a quick idea of how the model is performing using a sample of the original data. However, this does not guarantee that features where bias can cause real problems are being properly evaluated.

For example, using the bank loan default scenario – we have a model that has been trained with a dataset as close as possible to perfect. Each age range is represented in the same proportion as it is in the overall landscape of loans taken out. But when it comes to evaluation, the randomised sample that has been taken does not represent this distribution anymore.

This means that the metric generated is biased more towards those ranges with a higher presence in the data. For example, the model has a high recall score so it is particularly good at predicting which customers will default; however, the evaluation data is heavily weighted towards over 50s. This does not really tell us much about how good it is for the other age ranges.

To really understand whether the model is performing well for all the age ranges, it is essential to evaluate each range distinctly. For example, before training the model, evaluation datasets could be extracted for each of the distinct ranges whilst still maintaining a good distribution of data for training. This would then allow for separate evaluation of each of the age ranges. By carrying out the evaluation stage this way, it is easy to see how the model performs for each range which helps to quickly identify if it is systemically making wrong predictions for people of a particular age.

To use ML responsibly, understanding bias is key

Bias can cause real problems within Machine Learning; it is crucially important that steps are taken to avoid it wherever possible. The ramifications of using an unfair model can range anywhere from poor business decisions being made due to missed opportunities or incorrect predictions, all the way to breaches of law if it is determined that the ML model is discriminatory towards protected characteristics of individuals or groups.

There are ways to reduce the risk of, and subsequently detect bias in models which should aid in preventing an unfair model making its way into use. A thorough understanding of the types of bias that may occur and what steps can be taken to mitigate them is essential.

Machine Learning models are incredibly powerful tools for optimising processes and providing insights into your data – however, if trained with unrepresentative data, they can do more harm than good. As the saying goes: garbage in, garbage out!