Image by Editor
In many projects I carried out, companies, despite having fantastic AI business ideas, display a tendency to slowly become frustrated when they realize that they do not have enough data… However, solutions do exist! The purpose of this article is to briefly introduce you to some of them (the ones that are proven effective in my practice) rather than to list all existing solutions.
The problem of data scarcity is very important since data are at the core of any AI project. The size of a dataset is often responsible for poor performances in ML projects.
Most of the time, data related issues are the main reason why great AI projects cannot be accomplished. In some projects, you come to the conclusion that there is no relevant data or the collection process is too difficult and time-consuming.
Supervised machine learning models are being successfully used to respond to a whole range of business challenges. However, these models are data-hungry, and their performance relies heavily on the size of training data available. In many cases, it is difficult to create training datasets that are large enough.
Another issue I could mention is that project analysts tend to underestimate the amount of data necessary to handle common business problems. I remember myself struggling to collect big training datasets. It is even more complicated to gather data when working for a large company.
How much data do I need?
Well, you need roughly 10 times as many examples as there are degrees of freedom in your model. The more complex the model, the more you are prone to overfitting, but that can be avoided by validation. However, much fewer data can be used based on the use case.
Overfitting: refers to a model that models the training data too well. It happens when a model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data.
It is also worth discussing the issue of handling the missing values. Especially if the number of missing values in your data is big enough (above 5%).
Once again, dealing with missing values will depend on certain ‘success’ criteria. Moreover, these criteria vary for different datasets and even for different applications, such as recognition, segmentation, prediction, and classification (given the same dataset) even for different applications (recognition, segmentation, prediction, classification).
It is important to understand that there is no perfect way to deal with missing data.
Different solutions exist, but it depends on the kind of problem — Time-series Analysis, ML, Regression, etc.
When it comes to predictive techniques, they shall be used only when missing values are not observed completely at random, and the variables were chosen to impute such missing values have some relationship with it, else it could yield imprecise estimates.
In general, different machine learning algorithms can be used to determine the missing values. This works by turning missing features to labels themselves and now using columns without missing values to predict columns with missing values.
Based on my experience, you will be confronted with a lack of data or missing data at some point if you decide to build an AI-powered solution, but fortunately, there are ways to turn that minus into a plus.
As noted above, it is impossible to precisely estimate the minimum amount of data required for an AI project. Obviously, the very nature of your project will influence significantly the amount of data you will need. For example, texts, images, and videos usually require more data. However, many other factors should be considered in order to make an accurate estimate.
- Number of categories to be predicted
What is the expected output of your model? Basically, the fewest number or categories the better. - Model Performance
If you plan on getting a product in production, you need more. A small dataset might be good enough for a proof of concept, but in production, you’ll need way more data.
In general, small datasets require models that have low complexity (or high bias) to avoid overfitting the model to the data.
Before exploring technical solutions, let’s analyze what we can do to enhance your dataset. It might sound obvious but before getting started with AI, please try to obtain as much data as possible by developing your external and internal tools with data collection in mind. If you know the tasks that a machine learning algorithm is expected to perform, then you can create a data-gathering mechanism in advance.
Try to establish a real data culture within your organization.
To initiate ML execution, you could rely on open source data. There are a lot of data available for ML, and some companies are ready to give it away.
If you need external data for your project, it can be beneficial to form partnerships with other organizations in order to get relevant data. Forming partnerships will obviously cost you some time, but the proprietary data gained will build a natural barrier to any rivals.
- SEO Powered Content & PR Distribution. Get Amplified Today.
- Platoblockchain. Web3 Metaverse Intelligence. Knowledge Amplified. Access Here.
- Source: https://www.kdnuggets.com/2019/06/5-ways-lack-data-machine-learning.html?utm_source=rss&utm_medium=rss&utm_campaign=5-ways-to-deal-with-the-lack-of-data-in-machine-learning