Code prefetching files

More
8 years 1 month ago #13569 by gia
Code prefetching files was created by gia
Our ETL code is reading of heavy data files, it appears that the data is pre-fetched even while we are coding and hence a double click on transformer takes minutes to come back, is there a way we could restrict the no of rows fetched for say lookups. We have about 10 lookups in the transformer. Or do you think the issue is not with prefetching of data.
we arent having lag in other jobs.

Please Log in or Create an account to join the conversation.

More
8 years 1 month ago - 8 years 1 month ago #13570 by Peter.Jonson
Replied by Peter.Jonson on topic Code prefetching files
The Lookups can be the source of the problem
I would need to speak with our developers about it

There is simple solution you can use right now
Let assume that all your lookup files are stored in separate directory
Create another folder called "small lookup" put smaller version of files there.
Before starting development rename folders so the smaller version of lookup is used.
Once development is completed rename folders back

Peter Jonson
ETL Developer
Last edit: 8 years 1 month ago by Peter.Jonson.
The following user(s) said Thank You: gia

Please Log in or Create an account to join the conversation.