This tutorial shows how to avoid double processing of data files in Advanced ETL Processor. Use it when incoming files must be processed once, moved or marked, and then ignored on the next run. Duplicate imports are like relatives at Christmas: once is fine, twice starts costing money.
What You Will Learn
- Identify which source files have already been processed.
- Use file names, directories, or marker files to control repeat runs.
- Iterate through source files with loop variables where needed.
- Avoid reloading the same file after a scheduled workflow restarts.
Before You Start
Use a test repository or safe sample data while following the tutorial. Production jobs are a poor place to experiment.
File Processing Docs
This tutorial covers the practical side of avoiding duplicate file loads. In most workflows, that means moving files after processing, writing a marker, checking a processed folder, or generating predictable names for control files. The important rule is simple: leave evidence that the file has already been handled.
The filename and directory documentation covers working with filenames and directories, including dynamic names based on system dates and supported functions. That matters when processed files, archive folders, or marker files include dates in their paths.
When the workflow processes multiple files, the Files Loop Action documentation covers file selection parameters, the N Value parameter, resuming loop execution after failure, loop variables, and execution logs.
Need Advanced ETL Processor First?
Download the Advanced ETL Processor trial before following the tutorial, or review Enterprise if you need scheduled unattended execution.