We did spend a lot of time optimizing our ETL software and here is what we were able to achieve so far:
Loading data into Oracle database via direct path loading from a textfile
Version 126.96.36.199: 28000 rec/second
Version 188.8.131.52: 46000 rec/second
Version 184.108.40.206: 84000 rec/second
Version 220.127.116.11: 94000 rec/second
Version 18.104.22.168: 101000 rec/second
There a lot of ways to make our software better, we will continue working on improving performance and will publish our results here.
In latest version we have introduced support for Global Variables
They are stored in the registry, this allows to use different settings on different computers.
Please be very careful with this option it is very use to make mistake and run transformation agains wrong database
Advanced ETl Processor's "Is In list" validation function is used to check if value exists in the database or file.
But In some situations it is not very efficient.
In this case there is better alternative "Is value in database" and "If value in database" transformation functions
The way it works is very simple: during execution <value> is replaced with actual value If number of records returned is more than zero it returns success otherwise failure