Bruce Perry, Salesforce Admin/Developer

Manchester, NH 2019@NEDreamin #NEDreamin

With special thanks to our Platinum sponsors

Dealing With DuplicatesBruce Perry, Salesforce

Admin/Developer



Quiz Time

If you don’t have any duplicates,

A. Maybe you have a user adoption problem?B. Org-wide defaults set to Read-Only?C. You’re the only one with access to the database?D. Or maybe you’re just a really, really good admin?


Duplicates are a data quality issue

They contribute to

● Inaccurate reports/estimates● Increased costs (time wasted, duplicated calls and mailings)● Makes your organization seem disorganized to customers/clients● User dissatisfaction

Bottom line: Duplicates can make Salesforce (and you) look bad.


Where do duplicates come from?

● Users○ Lack of training (Search before you click new!)○ Bad processes

● Bulk data loads○ Import Wizard will check for duplicates(can load up to 50,000 records)○ Data Loader won’t check for duplicates○ Overlapping data sets○ Inaccuracy/inconsistency in initial data set

● Integrations/External apps that communicate with Salesforce● Attempts by alien civilizations to communicate with us via popular cloud apps


Best Practice Alert!

Make a backup before you start - and put it somewhere safe.

Options:

● Back up everything (Salesforce export)● Use a report* to export a more targeted selection of fields/records● 3rd party backup tool (if you happen to have one available)

*If you use a report for this purpose, do not forget to include any relevant IDs (eg. account ID if backing up contacts)


Best Practice Alert #2!

We interrupt the previous interruption to inform you that it is a best practice to have a unique identifier* for every batch of data that you upload.

It is A LOT easier to remove a batch of uploaded data if all members share a common key.

Deleting a batch of uploaded data is a very fast way to get rid of duplicates.

*doesn’t need to be fancy, your initials, the date, and maybe a number is enough.


Where do we find duplicates?

● Contacts● Leads● Accounts (and sub accounts and household accounts)● Affiliations● Custom objects


Steps to a duplicate-free database

1. Train your users2. Turn on duplicate detection/warning/reporting (reporting is a must)3. Develop reports on duplicates4. Make a backup!5. Merge critical duplicates manually6. Develop exact match scenarios.7. Merge duplicates from the top of your object hierarchy down.8. Develop fuzzy match scenarios.9. Merge duplicates from the top of your object hierarchy down.


Preventing (or flagging) new duplicates

● Activate a standard matching rule● Or Create and Activate a custom matching rule● Activate standard duplicate rule● Or Create and Activate a custom duplicate rule

These are located under Setup

● Data○ Duplicate management

■ Matching rules


Matching Rules and Duplicate Rules

Matching Rules (Standard or custom) tell Salesforce what qualifies as a duplicate.

Duplicate Rules tell Salesforce to handle any duplicates that are found.

Pro Tip - Don’t forget!!!Both types of rules need to be turned on.

If you turn on a matching rule without turning on its duplicate rule, Salesforce will do the comparisons, but take no action. Ask me how I know this.


Matching Rules, part 1

Matching rules can get complicated*, You can have:

● Exact matching● Up to 10 conditions● Filter logic (AND/OR)● Fuzzy matching (multiple flavors!)

Before getting into the weeds, ask yourself, “What combination of matching fields in common use gives me a 95% chance of finding a duplicate?”

*Jaro-Winkler/Metaphone 3/Kullback Liebler complicated!



Fuzzy Matching

What fields have the biggest potential benefit from fuzzy matching?

● Phone number - (123) 456 7890 is the same as 123-456-7890● First name - Bob/Rob/Robert might be the same person

Also, possibly email, but Salesforce doesn’t have this option. BTW, did you know that

[email protected] = [email protected]

It’s true!

mailto:[email protected]

mailto:[email protected]


Duplicate Rules, Part 1

Duplicate rules are simpler than matching rules. The major decision is to allow or block the duplicate. You can do that both when a record is being created AND when it is being edited. You can also alert the user and flag the duplicate for reporting purposes.

You can choose to enforce sharing rules - or not. If you’re getting duplicates because users can’t see records they are duplicating, consider turning this off.

You also get the option to add filter logic. You could have a duplicate rule that applied to certain account names only or contacts created after a certain date.


New Flash!!!

If you want to do cross-object comparisons, the place you do them is the

Duplicate Rule



Turn on Duplicate Management gradually

You can turn duplicate reporting on and nobody will even know you’re doing it.

You can turn on both warn and report. Users can still work as they did previously, but they will be told when they are creating a potential duplicate.


How are duplicates tracked?

Short version: background objects

Why is this useful to know?

Because you can use these objects for reports!

● Salesforce uses Duplicate Record Sets/Duplicate Record Items● DupeBlocker uses DupeBlocker Potential Duplicate● Duplicate Check uses Duplicate Check Group● Ringlead Cleanse does not keep info on matches in Salesforce


Report Types for Duplicate Reporting

Generic duplicate reporting - across all object types

● Duplicate Record Set○ Duplicate Record Item

Specific object duplicate reporting:

● <Object Name>○ Duplicate record Item


Report tricks for duplicate reporting, Part 1

Use a row level formula* to build a key field composed of multiple fields that you want to compare. You can then sort by the key field and see important values all in one place.

Example - <Full Name>-<City>-<State>-<Zip Code>

Maybe someday we’ll be able to group by row level formulas to make duplicates really stand out here. Not all fields available (eg. email and phone)

No added fields in your objects!

*you may need to enable these in your org first.


Report Tricks, Example 1



Group by* columns you don’t usually use for groups (eg. email or cell phone)

The records that end up in groups might be duplicates

*Sort by works too, but Group by makes the matches stand out more


Report Tricks, Example 2



Group by* columns you don’t usually use for groups (eg. email or cell phone)

The records that end up in groups might be duplicates

*Sort by works too, but Group by makes the matches stand out more


Merging individual duplicates, Part 1

Classic

● Account - Accounts Tab | Tools | Merge Accounts● Contact - Individual Account | Contacts Section | Merge Contacts

In Salesforce Classic, accounts must have similar names to be merged

Lightning

● Account - Potential duplicates shown on account record● Contact - Potential duplicates shown on contact record



Lightning Limitation Alert!In Lightning, the Potential Duplicates component is only available for accounts, contacts, and leads. For any other object, you get the warning that you’re about to create a duplicate. After that warning, you have to find the duplicate record yourself.

Potential duplicates are probably based on your current rules. There’s no “Search” feature as in Classic.



NPSP

● Account (also Household) - Potential Duplicates pane on account (or switch to classic)

● Contact - App Menu | Merge Contacts

Remember, NPSP uses household accounts and relates contacts to accounts via primary affiliation, not a direct lookup. Households can be split and merged.

Contacts that are not on the same account can be merged (unlike Classic).



Merge higher level objects first, then related lower level objects. Merging the higher level objects may mean there are now duplicated lower level objects

For example, first Accounts, then Contacts.


Before you merge, know your data

Are two contacts with these characteristics duplicates?

● same address, ● same birthday, ● same home phone, ● same last name, ● very similar first names


Not necessarily

Meet the twins!

Frank and Frankie Smith**real world example, last name changed

Do you have a data guru? Find one before you start to de-duplicate.



Quiz Time!

Are you done with duplicates after merging two accounts and any resulting contact duplicates?


Not necessarily

You might be using NPSP and might have duplicated affiliations

You might have custom objects related to the contact. These can become duplicates after merging contacts.


Before Account and Contact Merge

Account

Contact

Affiliation

After

Account

Contact

Affiliation

Account

Contact

AffiliationAffiliation


Finding and handling existing duplicates

● Reports on flagged duplicates (report type)● Basic reports● Salesforce Merge tools - Salesforce account and

contact merges, NPSP merges● Third party apps (next slide)


Spotting/Fixing Duplicates in Bulk

Salesforce duplicate jobs (Performance and unlimited Editions only)

Third party tools:

● Duplicate Check● Cloudingo● Dupe Blocker● Demand Tools● Ringlead Cleanse● CleanCRM● ZaapIT DeDupe Manager


Budget Alert!

It has come to my attention that some of these tools cost money. If getting money for these kinds of add-in tools is an issue for you, keep track of how long it takes to correct bulk duplicates by:

● Exporting data● Matching the duplicates in Excel● Deciding which fields to keep● Uploading updated data to the correct record● Deleting the duplicate record

The cost of these tools should be easy to justify in terms of your time saved (for cool new projects!)


Spotting/Fixing Duplicates in Bulk

Don’t forget the backup!


Required Features for Bulk Duplicate Handling

● Can save your search/scenario/job for re-use● Fuzzy matching of at least first name and phone● Flags duplicates for later merging● Auto-merging (use carefully!)● User-defined rule based picking of winners/losers● oldest/newest wins rules

Custom object matching/merging and cross object matching are also useful, but you might not need them


Process for bulk merging

Do a first pass using exact matching followed by one or more fuzzy passes. Merge parent records first, child objects second (eg. merge accounts, then contacts). Save your searches/jobs/scenarios (you’ll need them again)

● Parent - exact pass 1● Child - exact pass 1● Parent - fuzzy pass 2● Child - fuzzy pass 2

Note: you may well have more contact duplicates after you merge accounts.

Why? Duplicate checking looks for duplicate contacts within accounts


Tools for duplicate management - details

Switch to Google sheet here – URL for sheet is in resources


Resources – Page 1Link to “Duplicate Tools” Google sheet:

https://bit.ly/30zTfcd

Manage Duplicates in Salesforce (Summer 2019 version) - 44 pages full of all the details on duplicate management including exactly how all the different flavors of fuzzy compares work.

http://resources.docs.salesforce.com/220/13/en-us/sfdc/pdf/salesforce_data_quality_duplicate_prevention.pdf

Youtube playlist – matching rules, duplicate rules, and duplicate reporting (Jay Delaune, Force501.org)

https://www.youtube.com/playlist?list=PLVkg_BRPOj3z_oCzzGi39JgO6SVQGP90C

https://bit.ly/30zTfcd

http://resources.docs.salesforce.com/220/13/en-us/sfdc/pdf/salesforce_data_quality_duplicate_prevention.pdf

https://www.youtube.com/playlist?list=PLVkg_BRPOj3z_oCzzGi39JgO6SVQGP90C


Resources – Page 2

NPSP merge/split households

https://powerofus.force.com/s/article/NPSP-Merge-or-Split-Households

Include Classic-style merging in Lightning (untested, may be unsuitable for NPSP)

https://forcepanda.wordpress.com/2019/07/30/merge-accounts-and-contact-in-lightning-like-classic/

https://powerofus.force.com/s/article/NPSP-Merge-or-Split-Households

https://forcepanda.wordpress.com/2019/07/30/merge-accounts-and-contact-in-lightning-like-classic/


With special thanks to our Platinum sponsors!

Q & A

Documents

Bruce Perry, Salesforce Admin/Developer