How Directory Monitor Works
Directory Monitor watches folders for file events, checks whether files are ready, and then writes queue records for a separate execution service to run your ETL. In practice, it is a listener plus a traffic controller, not the engine that runs transformations.
What Directory Monitor actually does
It tracks create, delete, modify, and rename events in local folders or network shares. When an event matches your monitor settings, it inserts a record into a queue table. A separate service then executes the ETL workflow.
This split is deliberate. Monitoring and execution are different jobs. Mixing them is how pipelines become fragile and, eventually, dramatic.
If you are comparing this with a scheduled approach, see monitoring directories for new files and cloud storage automation patterns.
The thread architecture in plain English
There is a management thread that creates, updates, and removes monitor threads based on settings. Each monitor thread watches one directory target. User changes update settings without restarting the service, which is useful when operations teams need live adjustments.
Under load, monitor threads hand off readiness checks to lightweight tasks instead of creating one heavyweight thread per file. Rule of thumb: many small tasks scale better than a thread explosion that makes your server sound like it is auditioning for a jet engine role.
This design aligns with how file notification systems are commonly implemented in Windows environments (see Microsoft documentation for FileSystemWatcher and directory change notifications).
Why file readiness checks are essential
File creation events often arrive before the file is fully copied. If you execute immediately, you risk reading incomplete data. Directory Monitor solves this by running file usage checks that test whether the file can be opened safely and retry until timeout.
Nine times out of ten, this is where teams think ETL is broken when the real issue is timing. The software is fine. The file is still arriving in pieces (a bit like my old playlists, which are also missing critical parts).
For adjacent workflow patterns, see Excel workflow automation and what ETL is and why it matters.
Setup rules that avoid duplicate processing
- Use file filters so only expected files trigger jobs.
- Set sensible wait and timeout values for large network copies.
- For large files, copy into a separate staging directory first, then move into the monitored folder when the copy is complete.
- Keep raw files untouched and process copies when possible.
- Run on sample folders first, then scale to production paths.
- Use queue-state checks and file checksums to prevent the same file being processed twice.
When not to use this: if files arrive once a month and timing is irrelevant, a simple scheduled task can be enough. No heroics required.