0015 Skip duplicated values

Skip duplicate values during transformation

This example shows how to skip repeated values while processing a file. It is useful when the source keeps sending repeated rows or grouped records, and the target only needs the first occurrence. Duplicates are not always a disaster, but they are rarely invited.

Data example

The sample data contains repeated values. The transformation compares the current value with the previous value and skips the duplicate when the value has already been processed.

Run the example

• Download and install Advanced ETL Processor [Link]
• Download and unzip the example [Link]
• Create a new transformation and open the .ats file.

• Double-click the Reader object and amend the source file path.
• Double-click the Writer object and amend the target file path.
• Run the transformation by pressing the green arrow.

Transformation logic

The transformation uses previous-value logic to decide whether the current record should continue to the target or be skipped.

Previous Value transformation function

There are several ways to remove duplicates. In this example, the Previous Value transformation function is used because it can compare the current row against the row processed immediately before it.

When this method fits

This approach works best when duplicate values arrive next to each other or after the data has been sorted. If duplicates can appear anywhere in the file, sort the data first or use a lookup-style check instead.