Data Quality Tolerance and Business-Case

Data quality should not be perfect, but should meet the expectation of the user. Data quality pursuit has to pass the test of business case. Data Tolerances, leniency on non-financial data and historical data are some examples of staying with imperfect but acceptable data quality.

Everything is relative and so is quality. Quality is defined as ‘ability to meet the expectation of the customer in terms of cost, delivery and timeliness’. Data quality is not perfect due to reasons of the external and un-controllable factors, cost of quality and ‘less than perfect’ expectations of business.

Data Quality Tolerance

Example- While you are accepting payments, your system may keep some tolerances under which you state the payment as complete. This is done to collect the balance amount is not worth the effort. Some systems take the expected amount instead of real amount, and close the balance.

Data Truncation/data Rounding off

In today’s world where time is money, most of the financial figures are rounded off to nearest digit and in many cases to nearest tenth digit place. This truncation of data is done to keep the transaction simple and not to trouble the people at the counter to get hassled for pennies and cents, OR to avoid unruly reconciliation issues. Truncation of data is generally done less compared to the rounding off of data. The rounding-off is generally done in the favor of the customer to avoid any regulatory issues.

Real Time Data vs. Online Data vs. Offline Data

Data does not need to present the real-time world, if you don’t need it to that level. Most of the End of Day batch runs exhibit the same spirit. You might have paid cash in the morning on an outlet, but system may recognize it only in the night, when it gets feed from the front-end capture systems. Similarly an agent might be terminated in the distribution management system, but that is captured in the core administration system by end of the day. Most of the business applications are a combination of online + offline data updation.

Financial Data vs. Non-Financial Data

Most of the businesses are OK, if there are problems with the PIN code of a customer, but the balances against him/her should be accurate. Depending upon your need of analysis, you may find wrong pin codes un-acceptable, if you are doing analysis on the cities. Therefore non-financial data typically has a greater tolerance for data quality issues.

Transactional Data vs. Analytics Data

The transactional data is always expected to be of highest quality and mostly it throws up customer issues, if there is something wrong. The data, which is mainly used for analysis (non-transactional) is sometimes given a materiality lenience of few %age points. This leniency does not impact the quality of decisions. For example even if the states of 2% customers are not correct, that may not impact your pareto/80-20 analysis on state-wise customer value analysis.

Historical Data vs. Immediate Use Data

Companies are less bothered about keeping the historical non-financial data accurate. This is partially due to:

  • The data not impacting customer service
  • The physical effort needed to follow-up on what happened
  • Old data migrated from a junked system

Data Quality Cost-benefit equation for Business-Case

Typically companies find it acceptable to have 15% (say) of the mailing lists to be defective. The cost of cleansing the data to perfection (which will definitely involve the field verification) will outweigh the savings on the wasted mailers. Lesser is the data quality cost-benefit business case, greater is the tolerance for data quality issues.

This entry was posted in Data Quality and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s