How to improve the reader to read more records per second in Transformations?

More
3 years 11 months ago #21991 by ashokv
In transformations, Readers read at least 190 records per second until we upgraded ETL Tool Version:6.3.10.7. In transformations after upgrading, Readers could read up to 16 records per second maximum and they would take 2 hours to complete. If I used the same upsert logic in SQL scripts, those SQL scripts took ten to fifteen minutes to run.
Attachments:

Please Log in or Create an account to join the conversation.

More
3 years 11 months ago #21992 by admin
We do not know anything about your transformation how it works and what you are trying to achieve but yet you want us to help you.

How????

Mike
ETL Architect

Please Log in or Create an account to join the conversation.

More
3 years 11 months ago #21993 by Peter.Jonson
There are multiple factors which have an impact on performance.
One of them is the performance of the source database.
If you have very complex SQL it might take a while for the server to return the data.
The easiest way to test it is to remove all transformations and write data into the text file.
Running the transformation will give you an idea of how fast the source SQL is
You can always try adding indexes or rewriting source SQL

The next thing is the transformations.
The more complex transformation the slower it is
Some of the transformations are very slow by design for example SQL lookup

Writing data can be very slow as well
Please read the following article
www.etl-tools.com/wiki/knowledgebase/issues/update-is-slow/

I hope you will find my suggestion useful

Peter Jonson
ETL Developer
The following user(s) said Thank You: DeanCovey

Please Log in or Create an account to join the conversation.

Cookies user preferences
We use cookies to ensure you to get the best experience on our website. If you decline the use of cookies, this website may not function as expected.
Accept all
Decline all
Read more
Marketing
Set of techniques which have for object the commercial strategy and in particular the market study.
Google
Accept
Decline
Analytics
Tools used to analyze the data to measure the effectiveness of a website and to understand how it works.
Google Analytics
Accept
Decline
Google Analytics
Accept
Decline
Functional
Tools used to give you more features when navigating on the website, this can include social sharing.
Advertisement
If you accept, the ads on the page will be adapted to your preferences.
Google Ad
Accept
Decline
Save