Create, deploy, and maintain analytic applications that engage users and drive revenue. See a Logi demo

Development Tips

8 Common Data Problems to Protect Against

By Sriram Parthasarathy | November 15, 2018

How clean is your data? Data quality can be one of the greatest impediments to an analytics project. Good data quality leads to more accurate results and better predictions, while poor data quality can create misleading predictions—which in turn may lead to misleading actions for a company.

>> Related: Blueprint to Modern Analytics <<

Here are the most common data problems to look out for—and what you can do about them:

#1. Simple Numeric Typos

The value entered is not in line with the field range. For example, the value entered for a week is likely to be a number from one to seven. A value of eight is incorrect. Another common one is using the letter “O” for zero and the letter “I” for one.

#2. Complex Numeric Typos

The value entered is in line with the field range, but not in context with other fields. A $1,000 expense is normal, but a $1,000 expense for a book is not. In this example, an extra zero could have been added by mistake.

#3. Simple Text Typos

These are simply incorrect values. For example, typing Feb as Deb because “D” is right next to “F” on the keyboard. Deb may make sense as a name, but not as a month.

#4. Complex Text Typos

More complex text typos are also a common problem. For example, typing Belmont for Vermont because “V” is next to “B” on the keyboard. While Belmont is a location just like Vermont, if it’s in the state field, we know it’s a typo because Belmont is not a state.

#5. Missing Values

The data has missing values. For example, the age (numeric) or gender (categorical) is missing in a customer record.

#6. Data Rules Violation

An example of a simple data rules violation would be a date of birth that is set in the future. This is an easy catch. A more complex data rules violation might be a weight of 100 pounds for a one-year-old. You need context to catch complex error (it could be 10 pounds for a one-year-old or 100 pounds for a 15-year-old).

#7. Incorrect Format

Say you are an international company and you are merging product data. Five pounds in the United States is not the same as five kilograms in other countries. Another common format error is related to dates. For instance, 1/12/18 is not the same as 12/1/18.

#8. Duplicate Values

Duplicate values are harmless in some areas, but not always. For example, two people with identical information getting admitted to a hospital at the same time is almost certainly due to duplicate values.

What Can You Do?

If you identify any of these data problems, you can either:

  • Solve the problem in the current data. This involves a combination of transformation, imputation, and prediction using machine learning.
  • Solve the problem manually in the source data to prevent it from occurring again in the future. Tools like Logi Predict have built-in data cleansing algorithms that can solve a variety of such common data problems.

About the Author

Sriram Parthasarathy is the Senior Director of Predictive Analytics at Logi Analytics. Prior to working at Logi, Sriram was a practicing data scientist, implementing and advising companies in healthcare and financial services for their use of Predictive Analytics. Prior to that, Sriram was with MicroStrategy for over a decade, where he led and launched several product modules/offerings to the market.

Subscribe to the latest articles, videos, and webinars from Logi.