Agile Data Governance: The Key to Solving Enterprise Data Quality Problems

By Marty Moseley

Companies today are under increasing pressure to make better business decisions in less time, with less risk, while producing higher quality results. The challenges are enormous, as are the many issues that can arise and potentially jeopardize success. Among the most pervasive problems companies face is the consistently poor quality of internal data that is used to draw conclusions and make decisions. Poor data quality is not a new problem; but solving it is easier than before, because companies no longer have to rely on methods that require them to “boil the ocean.” Streamlined approaches to data governance that incorporate new processes and data stewardship technologies enable more agile methods for improving data quality.


Problems With Top-Down Data Governance

The problem with data governance programs thus far has been that most companies were taking a top-down approach, while more pressing short-term business demands were derailing efforts and distracting resources. Executives responsible for these programs and line-of-business managers, whose cooperation was needed to make the processes successful, were always scrambling to make their revenue numbers, launch new products and meet other required business objectives, rather than focusing on the often laborious data governance process, which only produced intangible results.

As a result, only a very small percentage of companies have an active data governance program in place because in most cases a top-down approach doesn’t work. In a typical top-down waterfall approach to data governance, a company spends six months forming a committee, an additional six months defining and stating the problems, another six months gathering requirements and another six months arguing about terminology. By the end of a two-year process, many people have wasted their time in endless meetings with little to show for it.


A Better Approach: Agile Data Governance

An alternative to top-down data governance and a better way to address the problem is for companies to adopt a more agile approach. By introducing more agile processes, companies can achieve quick wins by implementing data governance processes and policies in smaller pieces, learning from and adapting the approach with each segment, taking time off in between increments to focus on pressing business goals, and then coming back together to address another data domain, while making steady progress over time. For example, with an agile approach, a company could decide to focus its efforts on a master data management (MDM) project for its customer data, which it could actually solve within six to eight months – quite different from a top-down approach which takes years just to get started.

Companies that want to try an agile data governance approach should follow a number of guidelines that will assist them in a successful implementation. Before getting into the specific details of an agile process, let’s take a step back and consider some of the broader issues that are key to the success of any project:

  1. Determining up front which data fixes are going to deliver the greatest business benefits and focusing on fixing those first will help guarantee that the project is a success.
  2. Limiting the size of the data governance team and allowing the team to evolve as different data is being addressed at each stage of the project will help streamline the process and eliminate many of the potential political issues.
  3. Drafting a solid team of data stewards for each piece of the project will help to ensure success.

Before the overall project begins, a company needs to select a small, core data governance board made up of executives who can authoritatively represent the business goals of the entire organization. The board needs to have enough visibility to be able to sort out the biggest data problems the enterprise faces and determine which “critical few” problems should be tackled first, including the very first project, which should deliver the biggest overall bang for the buck. Data governance problems that should be addressed are those where poor data quality has the most direct impact on bottom-line profitability, productivity, cycle times, customer satisfaction, risk, reputation, cost savings and employee morale. The data governance board will ultimately be responsible for overseeing the entire data governance process, but it will not manage individual pieces of the project. Beware of the practice of most organizations to assemble larger, rather than smaller, groups for data governance projects. The thinking is that larger groups help ensure that everyone has a voice, but in reality this can seriously compromise agility and drag out a project.


Steps to a Successful Data Governance Project

Each iteration of an agile data governance project should be managed by an implementation team who will work with others within the organization to follow a number of specific steps that will result in successful completion. The following are recommended steps companies should follow for success:

1. Select the data governance implementation team for the first project.

Once data priorities have been set, the core executive board should select the team that will manage the first piece of the project. Executives who will all benefit from fixing a specific type of data are most likely to work well on the team and ensure that a project gets completed in a timely fashion. Each piece of the project will require different team members, depending on which data area is being addressed. For example, if a company is resolving business to business (B2B) hierarchy data, that problem will require a different group of executives than if product catalog data is being addressed.

2. Define the data problem.

The data governance implementation team should evaluate and determine the size and scope of the data problem it is going to address. This team needs to clearly define the business problems it is trying to solve or improve, and then ruthlessly manage the process to ensure that everyone stays focused on that one data domain. If the project for a single domain of data takes more than eight months to complete, chances are the scope is too big.

3. Draft the data steward team.

The data governance team should create a team of data stewards made up of those individuals with the most knowledge of the data being addressed in each piece of the project. Data stewards do most of the hard work with the specific data and are responsible for acting on behalf of data owners to employ technologies, find poor quality data, remediate and repair data as well as prevent errors in new data. In a global corporation, it would not be unusual to have dozens of data stewards spread across the world with the common task of solving a specific data problem. The data governance board should provide “air cover” to free up data stewards for this task and set aggressive deadlines to avoid analysis paralysis.

4. Validate or disprove assumptions of the data governance team.

The first job of the data steward team will be to investigate data quality problems in the first chosen data domain. This involves data discovery work to understand the data quality issues, profiling data across system boundaries, looking at messages that carry the data payload as well as examining extract, transform and load (ETL) transformations, data integration adaptors and connectors. The job of the data stewards is to determine the extent of data quality issues, root causes of data problems and the extent of potential damage when conducting a proof of concept (POC) on the data. In very large enterprises, for scope management reasons, it may also be wise to begin by focusing on a subset of the global systems.

5. Establish policies for making changes to data.

When the data steward team delivers the results of their analyses, the data governance team needs to step back in to create new policies for data and secure board approval for their recommendations. These new policies should be business policies – not technology policies – that enforce how data is handled within and between participating systems. They should define policies for system-independent data, for transaction-specific data quality and for addressing and solving often contentious cross-organizational data sharing. When participating systems have unique data integrity constraints that do not represent shared business rules, the governance teams should ensure that these constraints do not leak into business policies.

6. Enlist internal IT groups required to fix problems.

The data governance and steward teams should work together to kick off the actual project implementation. This phase involves calling on enterprise architects from internal IT groups to help design the best solution before specific problems are identified and fixes are made. Too often, repairs are made “quick and dirty,” which can cause additional data quality problems. It is critical to specify metrics reporting as part of the project design, so that exceptions and error rates are captured and reported as errors occur. Inevitably the data governance team and board, as well as data stewards, will be called upon to clarify business rules.

 7. Compare results, evaluate and determine next domain.

After testing the solution, bringing data into compliance across all participating systems and implementing new data quality tools, the team should compare the results and exceptions with the new business policies to determine where issues still exist. (It may take several iterations to cure a single data domain completely.) At this point, evaluate what worked in the overall process and what didn’t, and determine which data domain to tackle next.

The key to changing the current mind-set that data governance processes are long and tiresome and yield very few results is to focus teams on achieving quick wins using a more light-weight, agile approach, such as the process outlined in this article.

Delivering Higher Data Quality through Agile Data Governance

What is most critical for companies that want to make better business decisions by leveraging higher quality data is to begin implementing an agile data governance process immediately. Within six to eight months, these organizations will begin reaping the rewards. New approaches to data governance do not require executives to boil the ocean. With a few simple and easily definable steps, a C-level executive tasked with managing enterprise-wide data governance can begin the process and be successful. Remember, it’s better to minimize scope and ensure success, especially for busy executives, than invite casts of thousands to be part of a never ending process. Ultimately, an agile data governance approach will result in faster, measurable results to solving an organization’s pervasive data quality issues. A company can then build on its successes by iterating through the agile data governance process.


Marty Moseley serves as chief technology officer at Initiate Systems where he is responsible for the company’s strategic technology direction, development and future product evolution. Intitiate Systems, Inc. is the leading provider of customer-centric master data management software for companies and government agencies that want to create the most complete, real-time views of people, households and organizations from data dispersed across multiple application systems and databases.

This entry was posted in Data Governance and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s