Deduplicator

The Deduplicator is a specialized transformation object used to identify and remove duplicated records from data during ETL (Extract, Transform, Load) processes. It ensures that the final dataset is clean, accurate, and free of redundancy, which is essential for maintaining data quality and integrity in analytical or operational systems.

Properties

To change Deduplicator properties double click on the object. Tick Deduplicate for the fields you wish to deduplicate, only ticked fields are passed to the next object.

The Deduplicator is essential for maintaining high-quality datasets, particularly in scenarios involving data integration, migration, or preparation for analytics.