What happens if a machine learning model makes nearly all of its training predictions correctly but still fails when it encounters fresh information? Surprisingly, a stronger model isn’t always associated with greater accuracy. Overfitting or underfitting, two frequent causes of model’s inability to generalize beyond their training data could be the issue.
Table of Contents
What is Model Fitting?
When a machine learning model is trained, it searches the training data for patterns and makes predictions based on those patterns. Learning the training examples by memory is not the aim. A good model should be able to accurately anticipate new, unknown data by learning the underlying patterns. This is called generalization.
Consider teaching a model to distinguish between dogs and cats, for example. It can identify a new dog it has never seen before if it learns that dogs often have specific visual characteristics.
However, what would happen if the model learned too little or too much from the training set? Underfitting and overfitting are relevant in this situation.
What is Underfitting?
When a model is too basic to identify significant patterns in the data, underfitting occurs. The model struggles with both training examples and new cases since it does not learn enough from its training data.
Imagine trying to predict property values only based on the number of bedrooms, ignoring location, size, quality and other essential factors. It’s possible that the model is too basic to accurately represent the relationships found in the data.
Signs of Underfitting
- Poor performance on training data
- Poor performance on test data
- Important patterns are missed
- The model may be too simple for the problem
How to Fix Underfitting?
You can often reduce underfitting by:
- Using a more complex model
- Adding useful features
- Training the model for longer
- Reducing excessive regularization
- Improving the quality of the training data
However, increasing the complexity of a model isn’t necessarily the solution. If you push complexity too far, the opposite issue may arise.
What is Overfitting?
When a model learns the training data too closely, including noise and irrelevant data, it is said to be overfitting. On the data it was trained on, the model might perform incredibly well, but on fresh data, it might perform poorly.
Consider a student who commits all of the answers of a practice test to memory. Even if they receive a perfect score on that particular test, their performance drastically declines if the questions are significantly altered.
In machine learning, overfitting basically looks like that.
Signs of Overfitting
- Very high performance on training data
- Much lower performance on test or validation data
- The model captures noise instead of useful patterns
- Performance drops on unfamiliar examples
How to Fix Overfitting?
Common approaches include:
- Collecting more training data
- Using regularization
- Simplifying the model
- Removing unnecessary features
- Using techniques such as dropout in neural networks
- Applying early stopping
- Using cross-validation
Underfitting vs Overfitting
| Parameters | Underfitting | Overfitting |
|---|---|---|
| Model Complexity | Too Low | Too High |
| Training Performance | Poor | Very Good |
| Test Performance | Poor | Poor |
| What Model Learns | Too Little | Patterns and Noises |
| Core Issue | High Bias | High Variation |
Bias-Variance Correction
Bias and variance are two key machine learning concepts that are intimately linked to overfitting and underfitting.
Underfitting → High Bias
The model misses significant patterns and makes overly simple assumptions.
Overfitting → High Variance
Due to its extreme sensitivity to the particular training set of data, the model could experience substantial alterations.
A successful model seeks to strike a balance between the two. This is often called the bias-variance tradeoff.
Example
Consider a graph with data points that show the correlation between study time and test results.
- The data may be underfitted by a straight line that disregards the general trend.
- A good fit could come from a smooth curve that depicts the overall trend.
- By tracking random fluctuations, a highly complex curve that passes through nearly every point may overfit.
The model that passes most of the training points is not always the best one. It is the one that accurately depicts the fundamental pattern in order to generate trustworthy predictions for fresh data.
Why Generalization Matters?
Rarely is a machine learning model created solely to function well on the data it has already encountered. It is usually expected to function with unknown future data. A spam detection model, for example, does not have to commit the emails used during training to memory. It must identify spam communications that it has never seen before.
Test and validation datasets are crucial due to this. They offer a means of determining whether the model has simply memorized its training data or has truly picked up useful patterns.
Conclusion
There are two very different ways that a model can fail: either it learns too little or it learns too much. Overfitting results in the model becoming overly attached to the training set, whereas underfitting results in the model missing significant patterns. Reliable results on data the model has never seen before is the true objective, not flawless training accuracy.
FAQs
Q.1 What is overfitting in machine learning?
When a machine learning model studies the training data too closely, including noise and irrelevant data, it is said to be overfitting. Due to this, it does well on training data but poorly on fresh, raw data.
Q.2 What is underfitting in machine learning?
When a model is too basic to recognize the significant patterns in the training set, underfitting takes place. On both training and test data, it typically performs poorly.
Q.3 Is overfitting better to underfitting?
Both are unacceptable. While underfitting works badly because it misses significant patterns, overfitting may yield good training accuracy but poor performance on fresh data.
Q.4 Does overfitting decrease with additional training data?
Yes, most of the time. Instead of learning specific examples by heart, a model can learn broad patterns with more representative and varied training data. However, overfitting won’t go away just because more data is added.
Q.5 In machine learning, what is a good fit?
When a model learns the key patterns in the training data and does well on fresh, untested data without merely memorizing the training examples, it is said to have a strong fit.
Read More
- Understanding Supervised Learning: Types, Algorithms, Benefits and Applications
- What is Unsupervised Learning? Types, Benefits and Applications
- Data Augmentation: How it Improves Machine Learning Models
- What is Deep Learning? How it Works, Key Components, Types and Applications
- Neural Networks Explained: How They Work, Types & Applications