Compressing large files

12 years 6 months ago #1853 by Richard
Hi there

The initial requirement is to run a complete extract for the database as at the time of cutover (earlier today) then run an extract every day automatically at 03:00 KL time.

As before the full extract generates some fairly large CSV files (21 CSV files in total) and at then in the last step the CSV files are compressed down to one Zip file

The individual exports of tables to CSV files went fine how ever the set in the package for the Zip function failed at the first compress.

The first file is a very small table – I wondered why it was taking so link until I uncovered the error screen – see following -
In the executing window you can see that the file that was being compressed was Bank.csv – following is the contents of this file –

Clicking of the OK button in the error window moved the Zip process to the next CSV file to compress – with each click of the OK the system would move to try and compress the next file but would get the same error – see following –

You will see that it have moved on through several CSV files and is now at the CheckLog.csv file – this is a large CSV file – 908MB –
The Zip file shown in the above screen shot was created by the ETL extract run and a copy of this file is attached –

I ran a separate WinZip utility after the ETL run had failed on the folder and the 21 CSV files and the compression worked fine without any errors of messages being generated. The compress zip file size was 408MB

Is it possible that the particular implementation of the compression software in the ETL processor will not handle this combination of number/size of files?

This is the largest database extract we have experienced to date and is probably 20% larger than anything run previously.


Please Log in or Create an account to join the conversation.

12 years 6 months ago #1854 by admin
Replied by admin on topic Re: Compressing large files
We will check it with development team.
To be honest I have never compressed such big files


ETL Architect

Please Log in or Create an account to join the conversation.