NOTE! This site uses cookies and similar technologies.

If you not change browser settings, you agree to it. Learn more

I understand

One day we will be able to process one million records per second

We did spend a lot of time optimizing our ETL software and here is what we were able to achieve so far:

Loading data into Oracle database via direct path loading from a textfile

Version 8.3.6.1: 28000 rec/second

8.3.6.1

Version 8.3.6.3: 46000 rec/second

8.3.6.3

Version 8.3.6.4: 84000 rec/second

8.3.6.4

Version 8.3.6.6: 94000 rec/second

8.3.6.6

Version 8.3.6.8: 101000 rec/second

8.3.6.8

There a lot of ways to make our software better, we will continue working on improving performance and will publish our results here.

Switching from Production to development environment

In latest version we have introduced support for Global Variables
They are stored in the registry, this allows to use different settings on different computers.

Here is very simple example: two computers sharing same repository using different server for transformations
Production  Development 
  DataBaseOptions

Note:

Please be very careful with this option it is very use to make mistake and run transformation agains wrong database

Q-Eye performance boost

Today we have a reason to celebrate, our development team has managed to boost loading data from QVD files by 100 percent. Q-Eye version 3.1 can load 2.3 million records in just 16 seconds, that is 140k records per second

 

Q-EyeP

Working with large lists of values

Advanced ETl Processor's "Is In list" validation function is used to check if value exists in the database or file.

Is In List

IfIn List

But In some situations it is not very efficient.

  • The list of values can be too big and it would not possible to fit it into the memory.
  • Pulling entire lookup may take to much time
  • We may want to process just 10 records but our lookup has 10 million records

In this case there is better alternative "Is value in database" and "If value in database" transformation functions

Is value in database

If value in database

The way it works is very simple: during execution <value> is replaced with actual value If number of records returned is more than zero it returns success otherwise failure

If value in database Prop

More Articles...

"The DBSL Integration solution eliminated our data access bottle neck that previously impeded company growth. We are now able to provide solutions to long standing problems areas such as automated order processing and business reporting limitations. Additionally the solution allows for new opportunities to simply hook-on to our existing data sources. From development through testing the DBSL support  team continues to be helpful, resourceful and responsive to our company needs."

John Kil,
IT Manager

Our customers

BP

BBC

HSBC

Orange

CAT

SEPA
emc
NHS
Australian Rugby Union
xerox
Databases we work with