All Things Data

Master Data Management Discovery Summary

At this point, we have typically completed the Discovery Phase.  Before moving on to the Design Phase, we complete the Discovery Document and send it to our client for review.  The Discovery Document provides the background detail that is necessary so that Design & Prototyping of the master data management solution can take place.  In addition to being a valuable project management tool for our team, it also helps keep the client informed of the project details.  Further, the Discovery Document helps make sure that the client and our team are on the same page.

The last part of the Discovery Document is the Summary.  The Summary provides an overview of the master data management project goals, the tasks performed during the Discovery, and the project deliverables.

It should be pointed out that even during the Discovery Phase that we perform some of the initial Design work.  Some initial prototyping is necessary even before all the facts are discovered, as taking a look at the preliminary Design helps uncover the details and determinations that need to be made in order to develop the Design of the eventual “go live” system. 

Further, Discovery does not stop even when the Discovery Phase comes to completion. Discovery is an on-going process and certain elements shall also be discovered in the other phases.  However, based on the Discovery Document’s completion, we are now at the point where we can begin to configure and develop the client’s prototype master data management system.

Master Data Management Initial Matching

While setting up the Master Data Management match is primarily a Design phase function, it is necessary to at least do some matching during the Discovery Phase.  Doing some initial test matches provides an initial view of the data in match clusters.  These views help determine what data do you need for the import and match, hierarchy levels, and other determinations during the Discovery process. 
 
Also, we find that providing an initial match result helps demonstrate the potential of the Master Data Management system.  While you can talk about it all you want, until you start seeing results, it’s hard to get your head around it.  The initial test match can provide a clearer picture and build enthusiasm that is needed for the project.  With this enthusiasm, you can start to get the commitment and participation needed from the client in order to truly dive into their needs.  Therefore, even before we have completed all of the Discovery documentation, we start with initial match results.

Within our Master Data Management software, SingleVision, we have default match rules for both entity matching and contact matching.  Typically, we start with our default rules for our initial look.  As the Discovery process continues, we start to determine the definitions of the Hierarchy.  Once the hierarchy definitions have been set, the match rules can be adjusted to reflect the determined hierarchy levels. Then, as more and more Discovery is being completed, we further fine tune the initial match.

Master Data Management Import Mappings

At this point in the Discovery process, we have documented the data sources at the table and record level.  Now, we are ready to document the mappings of the fields for the import of each of the data sources that are going to be imported in to our Master Data Management software, SingleVision. 

The match rules for our Master Data Management solution are created by a series of algorithms that are based on combinations of the fields of data from the client’s data sources.  Therefore, we examine all of the fields from each data source in order to determine which fields may be potentially beneficial for the match.  Once determined, we can begin to plan for the importing of those fields. Since different data sources use different field names for the same information, we need to map each data sources’ fields to the corresponding field in the Master Data Management software so that it can be properly transformed and then matched based on the matching criteria that is selected. 

Additionally, there may be other fields that we may not necessarily need for matching purposes; however; we still may want to import them because they are potentially valuable as reference data.  Reference data is information that is not being used to help the master data management software match records, but provides valuable information for someone to see as part of their view.  You may want to import reference data to help you determine if records that are potential matches are in fact actual matches after a person takes a look. 

Another reason you may want to import reference data is the development of a single customer view.  One of the common goals of a Master Data Management solution is to provide representatives a view of all the authorized entity and contact records with authorized information across multiple data sources to help them do their jobs better.  For example, you may want to set-up a single view of the customer in a CRM system that lets a sales rep see all the products and revenue in the financial systems so that the rep can plan for potential up-sell and cross-sell opportunities.  The view in the CRM system would be developed from the Cross-Reference Table from the Master Data Management solution. 

Master Data Management Data Hub

Now that we have examined the data and started to make determinations with the data, we now need to correlate our data into a Cross-Reference Hub as part of our Master Data Management solution.  In order to correlate the data, we import the separate files into our Master Data Management software application’s central database.

As discussed in the previous post, some of the data may need to be pre-processed for better match results.  Based on the data determinations, in addition to running the import process, we also choose the MDM solution’s data transformation processes that are relevant to the client’s data and goals.  

Once the data has been pre-processed and imported, we can run an initial Match Project.  The Match Project generates a Cross-Reference Table by linking every record in the database based on their relationships as indicated by the Entity Level IDs and Contact IDs.

With a Cross-Reference Table, the client can: 

1.  View - See matched and linked records from multiple data systems.  For example, a company may want to give a sales rep a view of all the authorized records and information that are part of the same organization regardless of the data source in order to help up-sell and cross-sell.

2.  Report – Analyze data utilizing information across multiple data systems and properly counting the number of entities and contacts based on their Level and Contact IDs.  For example, a CRM system could be linked to the Cross-Reference Table in order to generate a report indicating how many unique entities that are customers or prospects within a certain market.

3.  Data Quality – Various data quality reports can be developed.  For example, a duplicate record report can be generated.  Then, as the system gets updated, subsequent reports can indicate the progress being made on cleaning-up duplicates as there should be less duplicates each subsequent day, week, or month that you run the report. 

4.  Operational System Enhancement – As part of the Master Data Management project, you may reformulate the data for matching purposes in order to get better match results.  The reformulation does not impact the operational system of origin.  For example, you may want to remove Department data that is in an Address field in the operational system of origin for the Match.  However, if desired, you can then use the Cross-Reference Table to help change the data back in the original operational system. 

In addition to creating a Cross-Reference Table, some organizations need to take the Master Data Management solution a step further and create a Master Record.  Sometimes the Master Records are also referred to as the Record of Record or the Golden Record. 

A Master Record is a record that uses information on the record from any of the data silos that have been linked by the Cross-Reference Table in order to create a record that is complete, accurate, and acts as the standard for that record based on the best available information across the data silos.  Many times a Master Record is created so that reports can be created with a uniform record representing that entity or contact within all reports.

When it comes to creating a Master Record, you can choose a record that already exists and deem it to be the Master Record or you can create one by taking the best information from the different data sources and then build the Master Record. 

Master Data Management Data Determinations

After the client’s files have been profiled both at the data source level and at the record level, we are now in position to start to make some data determinations for the Master Data Management system.   Data determinations for a Master Data Management application typically fall under the following categories:

1. Data Reformulation
2. Native Intelligence
3. Non-reliable Data
4. Missing Data
5. Accuracy
6. SingleVision Functionality
7. Operational Data

Data Reformulation

Sometimes, we need to change the way the client has the data for matching purposes in order to get the best possible Master Data Management match.  For example, the client may have Department data in an Address field.  Depending on the client goals, it may be best to extract the Department Data and put that data into a Department field that we created strictly for matching purposes.

Native Intelligence

By examining the Data Profiles discussed in the previous Discovery Posts, we can determine if there is any Native Intelligence that can be used to improve the Master Data Management match results.  Native Intelligence is data that is already in the client’s files that indicates that one record is somehow related to another record.  Sometimes the client has created the Native Intelligence through their own manual research or other method.  Other times, the Native Intelligence has been provided by purchasing or otherwise accessing information from a third-party resource.

Non-Reliable Data

Sometimes, after examining the Data Profiles, it may be determined that certain data elements are just not reliable enough and cannot be used for the Master Data Management solution.  However, in some cases, it may be determined that a particular data element could be useful after some clean-up.  Other times, we can use the data element if it is used in conjunction with some other criteria that restricts its use so that it is reliable for matching purposes.

Missing Data

After reviewing the Data Profiles, it may be discovered that certain data elements that we would like to use for the Master Data Management match either do not exist in the client’s files or only exist in a limited fashion.  At minimum, this can prompt a discussion with the client to double check to see if they do actually have this data element.  Depending on the missing data element, we could look to an outside source to provide the missing data.  However, this is typically not necessary as it is our general practice to use whatever data the client has.

Accuracy

In addition to determining how to use the client’s data, we also check the data in order to determine if it’s accurate.  For example, the client may have manually researched certain relationships and provided Native Intelligence indicating that certain records have a relationship.  However, after reviewing the data, it may be determined that there were some mistakes made.  Depending on the severity of issues with accuracy, it may be determined to remove the identified mistaken data, to revise the mistaken data, or to add additional criteria to the mistaken data so that it is accurate for matching purposes within the client’s Master Data Management solution.

SingleVision Functionality

Within our Master Data Management system, SingleVision, there is certain functionality to process data for matching purposes.  The review of the data helps us determine which data processing features within SingleVision should be used and how they should be configured.

Operational Data

While we are generally looking to review the Data Profiles for matching within the Master Data Management system, sometimes we review the results for purposes beyond matching.  For example, it may be discovered that there are many contact names in an address field.  The client may want to pull out these contact names and then populate the operational system with the contact data in the appropriate fields.

Master Data Management Data Format Goals

After profiling the customer’s client files, the next step in a Master Data Management initiative is to profile the data at the record level.  

We profile the Master Data Management records by breaking them down to their most basic elements.  For example, take a look at elements for a sample record at my Alma Mater, Rutgers:

Institution – Rutgers University
Campus – New Brunswick
Unit – Business School
Department – Accounting

With this breakdown, we can begin the discussion with our client as to what are the criteria for Master Data Management match at each hierarchy level:
1.  How many hierarchy levels do we need?
2.  What are the definitions of each hierarchy level?

We deem Level 1 matches to be the most granular and the higher levels to indicate more overall relationships.  

Depending on the business goals, one client may choose the hierarchy levels as:
Level 1 – Rutgers University, New Brunswick
Level 2 – Rutgers University

Another client may need further levels in their Master Data Management match:

Level 1 – Rutgers University, New Brunswick, Business School, Accounting Department
Level 2 – Rutgers University, New Brunswick, Business School
Level 3 – Rutgers University, New Brunswick
Level 4 – Rutgers University

Profiling organization records for the MDM match helps with  building corporate hierarchies, developing uniformed reports, and properly organizing the data when building a new database or incorporating data into an existing database

Similarly, we also breakdown Contacts records for the Master Data Match by its elements such as Company Name, Name Elements, Address Elements, Title, and Email.  While Contacts do not have hierarchies, the breakdown helps with the other Master Data Management matching goals.

Master Data Management Data Evaluation

Once we have received Data Extracts, the next step in the Master Data Management Discovery process is to document the files.  The documentation includes the total number of records as well as the number of records that contain each data element that can potentially be used for our Master Data Management match.  We also look to see if each record has a unique ID so that we can properly keep track of the records for importing and matching purposes in the Master Data Management application. 

In addition to examining the quantity of the data, we also take a look at the quality of the data that is to be integrated as part of the Master Data Management system.  For example, we look to see if there is data in an Address field, such as the continuation of the Company Name, that does not belong in an Address field.  It is very common for representatives to put data in a field that does not belong.  Sometimes this happens because the database does not have an appropriate field for the specific data element being entered or the appropriate field in the database cannot handle enough characters.  Other times this happens because the rep is working quickly “in the heat of battle” and mistakenly types the information in the wrong field.

Once we have documented and evaluated the data, we can begin to determine some of the items needed for the Master Data Management match.  Note, we do not interfere with any operational systems and we only make adjustments in the Master Data Management system for matching and linking purposes within the MDM system.  Some of the items to consider are:

1. Data Transformation – What pre-thought out processes within the Master Data Management application should be used to change the data for matching purposes?

2. Data Pre-processing – Do we need to develop any custom processes to change or derive data for matching purposes?

3. Knockout Data – Do we need to remove any data for matching purposes?

4. Hierarchy Data – Does the client’s data contain any information that indicates that records are part of the same corporate hierarchy or in any way related to each other?

5. Third-Party Data – Does the client have any third-party data that indicates that records are part of the same corporate hierarchy?

6. Match Data Elements – What data elements should be imported and used in the match?

7. Hierarchy Levels – What are the elements of each level of the corporate hierarchy match?

Master Data Management Data Extracts

Once we gain a general understanding as to what are the client goals in a Master Data Management solution, the first thing that we ask the client is to provide us with extracts from each of the data sources that is going to be part of the MDM match.   Our mid-market clients have typically been most interested in Customer Data Integration projects (“CDI”).  Therefore, most of our Master Data Management systems are matching customer databases like CRM, financial, separate customer service systems, external marketing files, and other data sources that contain customer and prospect data.

We generally ask our clients to send us data extracts that include all the Match Data from each data source.  Match Data is the data that we use to match records to each other in our Master Data Management solution, SingleVision.  The Match Data includes Organization, Contact, and other data elements.  However, it has been our experience that most of our clients send us partial or sample extracts for the initial Master Data Management match.  Then, after further conversations during the MDM Project Discovery process, we receive updated extracts that eventually include everything.  It’s not that are client are being difficult, it’s that they do not realize all the different data elements that can be utilized to improve the Master Data Management match results until they go through the process.  After a back and forth process with our client, we have all the Match Data that is available from the client that can be used for the initial Master Data Management match.

In addition to Match Data, many times the client will also send us Reference Data.    Reference Data is data that will be useful for viewing and analyzing unified data records.  It can include any data that would help the client evaluate the matched and linked records.

Master Data Management Client Goals

The first step in any Master Data Management or Customer Data Integration project is the Discovery Phase and the first step in the Discovery Phase is to understand what the client’s goals are.   Once you start with the overall goals, then there shall also be some data goals along the way to help reach the overall client goals.  A Master Data Management solution can involve both Parties and Products.  Parties consist of data items such as customers, prospects, resellers, vendors, suppliers, and other data types.  The Parties are made up of both organizations and contacts.  Many times a Master Data Management solution shall consist of the integration of Parties and Products.

In the Mid-Market, we have seen a variety of different types of overall goals for Master Data Management solutions.   Common Master Data Management goals include providing a single customer view for cross-selling and up-selling, marketing analysis, and data quality goals.  Another common Mid-Market Master Data Management goal is to build a new database or incorporate new files into an existing database after a merger or acquisition.

Typically, there shall be a goal that is the main driver of the Master Data Management initiative and there shall be secondary goals that will come out of the Discovery process.  For example, the main driver for a Master Data Management system may be to provide their sales reps a single customer view to help up-sell and cross-sell.  Inevitably, it will be discovered that the client also has issues performing analysis, such as accurately stating how much business the company is doing with ABC Company, and data quality issues.  Discovering what the client’s overall goals are and the data goals that are steps along the way to reaching the overall goals is an interactive process with the client.  Discovering the initial Master Data Management goals is something that should be done as the first step in the Discovery process.  However, new goals, changing goals, and data goals that shall help reach the overall goals will also be discovered at any stage of a Master Data Management project.