How clean is your data? Data quality can be one of the greatest impediments to an analytics project. Good data quality leads to more accurate results and better predictions, while poor data quality can create misleading predictions—which in turn may lead to misleading actions for a company.
Here are the most common data problems to look out for—and what you can do about them:
#1. Simple Numeric Typos
The value entered is not in line with the field range. For example, the value entered for a week is likely to be a number from one to seven. A value of eight is incorrect. Another common one is using the letter “O” for zero and the letter “I” for one.
#2. Complex Numeric Typos
The value entered is in line with the field range, but not in context with other fields. A $1,000 expense is normal, but a $1,000 expense for a book is not. In this example, an extra zero could have been added by mistake.
#3. Simple Text Typos
These are simply incorrect values. For example, typing Feb as Deb because “D” is right next to “F” on the keyboard. Deb may make sense as a name, but not as a month.
#4. Complex Text Typos
More complex text typos are also a common problem. For example, typing Belmont for Vermont because “V” is next to “B” on the keyboard. While Belmont is a location just like Vermont, if it’s in the state field, we know it’s a typo because Belmont is not a state.
#5. Missing Values
The data has missing values. For example, the age (numeric) or gender (categorical) is missing in a customer record.
#6. Data Rules Violation
An example of a simple data rules violation would be a date of birth that is set in the future. This is an easy catch. A more complex data rules violation might be a weight of 100 pounds for a one-year-old. You need context to catch complex error (it could be 10 pounds for a one-year-old or 100 pounds for a 15-year-old).
#7. Incorrect Format
Say you are an international company and you are merging product data. Five pounds in the United States is not the same as five kilograms in other countries. Another common format error is related to dates. For instance, 1/12/18 is not the same as 12/1/18.
#8. Duplicate Values
Duplicate values are harmless in some areas, but not always. For example, two people with identical information getting admitted to a hospital at the same time is almost certainly due to duplicate values.
What Can You Do?
If you identify any of these data problems, you can either:
- Solve the problem in the current data. This involves a combination of transformation, imputation, and prediction using machine learning.
- Solve the problem manually in the source data to prevent it from occurring again in the future. Tools like Logi Predict have built-in data cleansing algorithms that can solve a variety of such common data problems.