After receiving a comment on last month’s post I decided to do a follow-up and detail a little further how Microsoft Dynamics CRM manages the merging of duplicate records.  For the purposes of this post I’ll stick to using Contacts as the example.  However, the same is true for Accounts and many other tables. 

For our sample records let’s say we have just two contacts that are duplicates.  Contact A has four service calls associated with it.  Each of these service calls have relevant data that you want to retain.  Contact B has three service calls associated with it and each service call has data that needs to be retained.

Upon merging Contact A with Contact B (so in this case B is the keeper record), there will be seven service calls associated with Contact B.  This is accomplished through the use of three data elements in each Contact transaction. These fields are MasterID, Merged, and Statecode. 

Merged is an indicator field where 1 indicates that the transaction is indeed merged.  Statecode is another indicator field indicating active and inactive transactions.  In Dynamics a statecode of value of 1 is inactive and 0 is active.  Yes, you read that right.  Zero is active. 

 The magic of the duplicate consolidation lies in the MasterID field.  For consolidated records the MasterID is equal to the unique identifier of the keeper record.  So in our example if the unique identifier of Contact B was 1234, the MasterID of  Contact A will be 1234.  The Merged field would be 1 and Statecode of 1.  This allows all associated transactions of Contact A to be “realigned” to Contact B without having to perform massive updates to many tables.  The MasterID functions as a re-pointer for all related transactions associated to the merged record.  So there are no changes to the related transactions.  The re-pointer directs all transactions related to the non-keeper record to the new keeper record.

For those transactions which are not merged the MasterID is NULL (no need to store the unique identifier twice).

In last month’s edition of the DQC I reviewed some data quality features built into Microsoft’s CRM package, namely detect a duplicate upon create or update, duplicate detection rules and duplicate detection jobs.  I left off with a promise to dive deeper into how you remove the duplicates once you’ve detected them. 

Before I get into the details, I want to emphasize that without customization, removing duplicates is not a batch process.  In other words, you remove duplicates one at a time.  Don’t kill the messager; learn from the message.  If there is one area within the data quality space that Microsoft needs to improve on, it’s this one. 

Duplicate consolidation, in my experience, is rarely so exception based that it can be done in such a tedious manner.  Not to mention that those organizations that are most afflicted with duplicates generally have  a large customer base.  When you have a customer base in the millions, duplication ratios can be as high as 10% or more.  Consolidating 100,000 duplicates one at a time is almost pointless.  By the time you catch up, you’ve created more duplicates.

  That said, let’s move on.  So you’ve detected duplicates and now you want to eliminate them from your data. 

If you remember from last month’s post, read up here if you don’t, a duplicate detection job returns potential duplicates and allows you to browse each one along with it’s potential match.  Consult the screenshot below for a view of what that looks like.

Duplicate Detection Job results

In the lower pane of the screenshot above there is a toolbar option (3rd from the left) is a icon to merge the two highlighted records.  This is where the consolidation effort begins.

One of the best features of the merge functionality is that it has the flexibility to build a composite, or best of available information, master record.  Briefly, the master record is the record which is retained as the active record.  It also allows the end user to select one record over another.  An example of these features is outlined in the screenshot below.  First let’s look at the option where each element of the master record are selected.

An example of an all inclusive master record selection

Here’s a look at the composite option:

An example of a composite master record selection

Notice in the all inclusive example the entire left hand column is highlighted in blue, whereas in the composite option only those elements selected via the radio button are highlighted in blue.  This is a visual indication of what data elements will be retained in the master record.

This is one of my favorite pieces of functionality with regard to the merge option.  Often end users vary in the data they provide and it is always better for an organization to retain as much information about their customers as is possible.

I specifically chose the composite screenshot presented because it illustrates one of those important aspects in customer data quality.  Noticed that the element selected from the right hand side was a middle initial.  This data element is invaluable when performing data matching and having that element can make an important distinction between two different customers later on down the road.

Once you’ve defined what your master record looks like, either via the all inclusive or composite method, it is time to commit that selection.  The screenshot below illustrates how this is performed.

How to commit your master record selection

An important option in the commit process is one enabled through the checkbox provided visible in the screenshot above.  Not every field in a record is in all cases exposed via the merge utility.  As a result the option made available through the checkbox allows you to make sure that, as the label indicates, select every field with data from the chosen master record even if there is a different value in the other record.  Simply put, it is an overwrite function that retains all the data from the selected master record beyond what is visible in the merge screen.

Once you’ve reviewed and are confident in your selection, you simply click on the OK button.  Provided there are no commit locks on the record, which indicates that another user has one of the two records open and is actively working on it, you will receive the following dialog box confirming your consolidation success.

Duplicate elimination success!

It is critical to note that the subordinate, or non-master record, is NOT deleted from the system.  It is simply deactivated.  This is to say that a flag (statecode) is changed to inactivate.  One important note about the statecode field is that unlike conventional notation a value of ‘1′ is not active in Microsoft Dyanmics CRM.  Instead Microsoft chose the value ‘0′ as active and ‘1′ as inactive.  Consequently all non-master records in CRM have a statecode value of ‘1′.  This little fact can save hours of data analysis and perserve the samity of your DBAs, so it is worth noting.

I hope this information was beneficial to you Microsoft Dyanmics CRM users and administrators.  As usual I welcome all comments, questions, and suggestions.  So please feel free to comment on this post and I’ll try and replay in a timely manner.

This month I’d like to talk about my recent experienecs with some of the data quality features of Microsoft’s Dynamics CRM package and how to put them to use in the typical enterprise environment.

One of my favorite features is one that attempts to prevent duplication from occurring by executing a match scan on create or update of a record.  One of the reasons this is a favorite of mine is that it is proactive and aims to prevent duplication where every data quality experts recommends; at the beginning.  One of the issues I have seen with this feature is that in high data volumes it causes significant delays in record creation. 

The feature can be enabled by checking on the “When a rcord is created or updated” option in the Duplicate Detection Settings panel of the Data Management console. 

Data Management Console

One of these features are duplicate detection rules which are a set of criteria used for matching records.  For instance, it is very common for organizations to accumulate more than one record for a single customer.  In this example, a duplicate detection rule can be built to identify all customer transactions that have a match on customer’s first and last name as well as their zip code.  As you may already know, I am a diligent advocate of requiring more than first and last name to truely identify an individual.  Based on research regarding change of address (COA), it makes practical sense to limit this criteria to a geographic area like those from which postal codes are generated.  You may also want to throw in more qualifying data elements such as street name and number but for the purposes of this posting, we’ll stick to a customer’s full name and zip code.

This rule will identify each set of records that share the same values for these three data elements.  It is possible to require an exact match on the data values or a substring of the characters of the values.  Again, I am an advocate of using partial matches on data like last name due to the frequency of data entry errors.

Once you create your match criteria you’ll want to intialize, or publish, the rule.  You can publish a rule, with the proper permissions, by clicking on the greeen arrowed icon labeled “Publish” on the tool bar after the rule is saved.  Be forewarned some rules take quite a long time and a lot of resources to publish so you may want to perform this action as part of your off-hours operations.

Once the rule is published, it is time to schedule when it will be utilized.  This is done by building a duplicate detection job which includes the start time, a setting for execution reoccurance, and an option to provide an email for notification of when the job finishes.  The following is a snapshot of the interface for developing detection jobs.

Duplicate Detection Job

Once you have your rules and jobs created, you completed the basic steps to remove duplicates from your data.  After the job completes and you receive your email you’ll want to review the duplicate matches. 

 This can be done by opening the duplicate detection job from the System Jobs queue and double-clicking on it.  Once the job is open you’ll see an option labeled “View Duplictes”. 

Next month, I’ll dive deeper into the details on how to remove the duplicates with a posting on the merge feature.  I hope this was informative and enough to get most of you started.  I’ll address detailed questions if you have them, so please feel free to comment!

This year the IAIDQ, an international not-for-profit dedicated to developing the profession of Information Quality Management, is 5 years old and is having a series of rolling celebrations, the Blog Carnival “Festival del IDQ Bloggers” being one of the strands of those celebrations. 

I am glad to be hosting the Festival del IDQ Bloggers this month!  I’ve tried to capture the core of each message, but each of these is worth a deeper look.  Don’t forget to follow the submission links and get all the details!

This month’s first submission comes to use from Daragh O Brien.  Daragh poses an interesting question when he asks, Is information quality management a recession proof profession?

One clear take away from this post is that those who express the value proposition of an information quality initiative are more likely to be regarded as valuable and necessary.  In that way those who participate in these initiatives can be thought of a recession proof.

About Daragh: Obsessive blogger, information quality consultant and Director of IAIDQ with over 12 years experience at the sharp end of Information Quality. Taoiseach (CEO) of Castlebridge Associates, a specialist Information Quality consulting business based in Ireland.

Continuing on a theme our second post comes to us from Dylan Jones and shows us How To Deliver A Compelling Data Quality Business Case.

 Dylan recommends several excellent ways to build and deliver the value proposition such as:

• use time/date stamps to show information quality as a long term problem and not a short lived glitch
• use cause and affect analysis to link data quality issues with business process gaps
• review the annual report to gain insight into corporate strategies that can benefit from information quality services
• don’t use PowerPoint
• keep it simple to get the point across

About Dylan: Dylan is the founder and editor of the Data Quality Pro which is dedicated to “helping data quality professionals take their career or business to the next level.”

Switching gears a little, Jim Harris reminds us to keep our theories in check until after we’ve taken the time to really listen to what’s being said  in his post Hailing Frequencies Open. 

In this post Jim points out the difference between waiting to talk and what is called empathetic listening where we are actually listening with the intent to really try to understand the other person’s frame of reference. 

About Jim:  Jim Harris is an independent consultant, speaker, writer and blogger with over 15 years of professional services and application development experience in data quality.  Obsessive-Compulsive Data Quality is an independent blog offering a vendor-neutral perspective on data quality.

While we are on the topic of communication Steve Sarsfield recommends some data governance dialog for CEO’s in his post, 9 Questions CEOs Should Ask About Data Governance.

 In this post Steve points out that the executive team is responsible to lower risk and gain control through their influence on data governance.  The following are a few of the question Steve suggestions:

1. Do we have a data management strategy?
2. Are we in compliance with all laws regarding our governance of data?
3. Do you have the access to data you need?

About Steve: Steve Sarsfield is a data quality evangelist and author of the book the Data Governance Imperative. 

Finally in Sweden meets United States a post from Henrik Liliendahl Sørensen on data matching and how character sets, address formats and naming conventions are just a few of the complexities when the data originates in a different language.  Maybe, as he suggests, centralized reference data is a step in the right direction to solving some of these issues?

About Henrik: Henrik is a Data Quality and Master Data Management professional also doing Data Architecture.  You can check out what is on his mind at Liliendahl on Data Quality.

I hope you enjoyed this month’s edition of the blog carnival!  I want to thank all those who’ve submitted postings and encourage those who have not done so yet to participate in this opportunity.

Howdy Folks!  I’d like to sing you a tune about matching customer data if you don’t mind?  It’s called the “data quality two step”. 

Heck! You can even grab your partner if the mood strikes you?

Feels good!

I’ve recently cleansed some customer data for a great bunch of folks that I love calling my client. 

We had a consolidation ratio of 1:6 or around 17%.  Which equated to roughly a million duplicates removed.  That’s a lot savings on postage stamps for the marketing department so they were psyched!  We validated over 90% of the addresses and built reports to identify those that did not meet the requirements for a valid address.  Not too shabby if I don’t say so myself! 

Now that we’ve deployed the data into User Acceptance (UAT) I find myself in a familiar place; the business logic. 

You missed something

You can spend all the time you need, or even care to, on rules for consolidation but it usually is not until the data hits the screen that the ramifications are easily understood by the average business user.

Case in point, I recently received an email from a stakeholder asking me to look over some data with him.  I was curious what I’d find when I reached his office as I analyzed this data and the processing code more than a few times by now.  On my walk over I went through many possible scenarios in my head.

Was it something I missed?  Surely not.  I’ve performed several test runs in order to validate the business logic.  With my curiosity peaked I rounded the corner and into his office I went.

"Good point!"

After a little chit-chat, like I said I love this client, we got down to business.  He proceeded to type a few parameters in the search utility and I waited with anticipation.

However after a second, maybe less,  my anticipation was replaced with relief and more than a little disbelief.  I’d been over this a time or two which is why I was in a state of shock.  Not to mention my client was not someone who needed “Data for Dummies”. 

With identities masked to protect the innocent, below is a sample of the records he was concerned about and wanted me to see.

Duplicate?

So if you’ve been wondering about this two step thing, here it comes. 

In order to positively identify a non-unique individual you need to pair their name with an additional piece of identifying information, usually an address. 

In other words, it is a two part match on name and address that can, with a realtively high confidence level, identify a true duplicate. 

If we only used a match on name to identify duplicate, we’d consolidate all the John Smith’s in the dataset to one customer.  Talk about lost opportunity!  This approach could turn millions of customers into thousands in an instant.

One brief glance in the local phone directory will be enough to demonstrate how non-unique names really are.  

They've been through this before!

Go a step further and ask your local DBA to run some counts on first-last name combinations and you’ll be surprised at the results.

Just in case this little story wasn’t sufficient enough to remind you here is that tune I promised you:

 

The two step matching ditty goes a little like this …

 

Grab your partner’s name and twirl it around

Make sure the nickname’s proper equal is found

Then grab you their address and scrub with the care

Make sure their mail can be delivered there

Don’t get rid of your partner until you are sure

That you’ve got a match on more

Than the name or the door

lyrics by Data Pickins

music by YouToo?