How to Automatically Remove Duplicates in Marketing Automation Systems?

Don’t fear, no duplicates are here!

Nowadays, most companies use multiple systems to manage customer relations. CRM, Customer Service Software and Marketing Automation Systems are merely the tips of the iceberg. While each system on its own might be a well-functioning ecosystem, the interconnection of systems usually stirs up trouble. In terms of data quality, one of the most common problems we have identified in system integrations is the creation of duplicates. These duplicates might be created by one system pushing lead or contact data into a receiving system — no matter if the receiving system holds the exact same data already. Consequently, several contacts with the same contact information exist in the receiving system.

Duplicates Cause Major Problems

Duplicates usually originate from the duplicate handling process of the involved systems. Oftentimes, the rules for duplicate detection are set up in a way that doesn’t meet the business needs. This results in duplicates not being detected at all by the receiving system.

This comes with a number of implications for Marketing Managers:

• The size of audiences for sendouts is unclear since several duplicates of each contact might be included.
• The deliverability quality of sendouts might decline because the email system might block several emails going out to the same email address.
• If the email system doesn’t block duplicate email addresses and the emails actually go out several times to the same person, the performance KPIs of the emails like open and click rate might be very low. Additionally, it goes without saying that one contact receiving the same communication multiple times does not reflect a desirable or acceptable customer experience.
• The profile information of existing contacts might not be updated anymore, which leads to:
  • Wrong information in dynamic and personalised emails, for example, false information about the last purchase.
  • Wrong information about communication consent (opt-ins and opt-outs), which might breach GDPR and could result in severe fines.
• When contacts exist multiple times, automated processes might fail because it is unclear which contact should progress through a flow.
• In many cases, the cost of a Marketing Automation system licence is influenced by the amount of contacts stored in the database. Therefore, amassing duplicates will not only clutter the database but might also lead to increased licence fees.

Even though many CRM systems are able to address deduplication processes to some extent, the correct synchronisation of data between multiple systems comes with a level of complexity that cannot be solved using CRM system capabilities only.

Automated Deduplication

We have experienced companies trying to get their data under control by using manual deduplication. Some even hire “deduplication agencies” who provide the manpower that is required to remove the vast amount of steadily incoming duplicates.

This solution of course comes with additional costs, inefficiency, lack of documentation of the merging history and human errors, resulting in loss of customer information.

The disadvantages can be mitigated by using automated processes, allowing for saving time, money, and valuable data about customers.

In this article, we aim to introduce an automated custom solution which we developed for clients struggling with duplicate record creation when integrating and running multiple systems:

Example: Duplicates in Marketo

In the following, we will introduce the Avaus Deduplicator application while referring to the example of the interconnection between Marketo and Salesforce CRM (Salesforce Service Cloud or Salesforce Sales Cloud), because it is a common setup. Salesforce CRM data as the dominant data set is sent to Marketo where the data is used for all kinds of marketing purposes.

Exhibit1: Typical setup of CRM and Marketing Automation System

A dedicated connector between both systems is available on the market. However, depending on the setup of the systems, the connector comes with a flaw. As the data flows from Salesforce to Marketo, the leads may be duplicated, resulting in multiple rows describing the same person. This is where the problem occurs, because Marketo is not able to merge duplicate rows.

While there are tools on the market to tackle the issue, together with our clients we came to realise that these tools don’t pose the best fit to their needs. Thus, we decided to create a flexible solution to solve the issue of deduplication in the connection between Marketing Automation Systems and CRM.

The solution: Deduplication application

The Avaus Deduplicator removes duplicates using a program consisting of nine components. The first three components of the application are basic setup tasks that allow accessing the Marketo dataset and exporting it into the deduplication application. The importance of data security should not be underestimated in this part of the setup. Recommendations for safe and reliable hosting solutions are part of our standardised setup process.

After the data has been made available the deduplication takes place. Data has to be pre-processed in order to fit the application. It will then be structured so that duplicates can be detected. Based on a set of individual and customisable rules, the duplicate rows are finally merged within the deduplication app. 

As a next step, the data is sent back to Marketo: Duplicate records are removed from Marketo and only one “master” record remains. This master record is enriched with any information that was stored on the duplicate records.

Exhibit 2: The nine components of the Avaus Deduplication solution

The flexibility of the solution is enabled by the adaptable components, as displayed in the green-colored boxes in the flow chart above. These green components can be described as templates that will be adapted according to client needs. The cost-effectiveness of the solution can be achieved due to well-tested, standardised components, displayed in the blue boxes.

Setup: Process of implementing the deduplication solution

The first step in creating a solution to avoid duplicate records in Marketo is to understand the client’s individual needs. The logic behind the deduplication rules needs to be aligned and defined first. As a next step, the information is “translated” into the code templates according to custom logic requirements. Looking at it from a technical perspective, the program is running on a Docker container along with features that allow logging the program’s activity, scheduling the program’s execution, and running the program in a cloud or local environment.

Only when these preparations have been completed, first local tests can be executed and optimisations get implemented. The deduplication results of the tests need to be reviewed thoroughly jointly with the client to ensure satisfying results.

After review of test results and subsequent approval, the solution is ready for deployment on a Docker container in any cloud environment. The solution is running automatically and only requires monitoring. If you are interested in further details, please reach out to us.

Exhibit 3: Implementing the deduplication solution in six steps

Case Study: Deduplication results

A client struggled with a number of +15.000 duplicates in their Marketing Automation system. As the integration with their CRM produced new duplicates on a daily basis, manual deduplication efforts by a deduplication agency didn’t show long-term results. After running the Avaus Deduplicator, around 98% of duplicates could be resolved. We are confident that even a rate of 99.9% could have been achieved. However in this client case, the remaining 2% of duplicates are required to stay in the system due to the unique business rules the client had to set up as part of their compliance requirements.