Working with a large lists of values
Advanced ETL Processor’s “Is In list” validation function is used to check if a value exists in the database or file.
But In some situations, it is not very efficient.
- The list of values can be too big and it would not possible to fit it into the memory.
- Pulling the entire lookup may take too much time
- We may want to process just 10 records but our lookup has millions of records
In this case, there is better alternative “Is value in database” and “If value in database” transformation functions
The way it works is very simple: during execution <value> is replaced with actual value If the number of records returned is more than zero it returns success otherwise failure
Note: This function’s performance is negatively impacted by large record counts, making it unsuitable for such use cases.
For more technologies supported by our ETL Software see Advanced ETL Processor Versions
Confused? Ask question on our ETL Forum