Array list equivalent in ETL?

More
8 years 10 months ago #11947 by orlandoc
Hi,

One of my package is using a file system to get list of files and audit each file name to the database and send out an email with the list of file names listed in the email body.
What I did inside the transformation is pass the FileName to a variable <FileNameVar> and in the Send Email function in the package level I pass that variable <FileNameVar>.
If I have 2 files in that file system, the transformation can able to audit those two successfully but the email were only sent one file name - so the ETL variable can only store one value. So is there a way to collects all the filename values during transformation in a variable array?

Please Log in or Create an account to join the conversation.

More
8 years 10 months ago #11948 by John.Paterson
This is far too complex ant not really necessary.
Better option is to copy all files you want to email into separate folder and
email all of them in one go using mask

You can also compress the first

John Paterson,
Support Engineer

Please Log in or Create an account to join the conversation.

More
8 years 10 months ago #11951 by orlandoc
Array list or holding list of records in the memory level will helps me in many different scenarios like this ...

Imagine these scenario ...
ETL has to process hundreds of CSV files - validates the contents of each file row by row.
Let's say I have to validate order number in every rows and this particular csv file contains 100+ rows, so let's say there are 3 incoming csv files and two of them contains several empty order number. So I want to compose my email body like this ...

File: test1.csv file contains invalid data
Row 1: has no order number
Row 4: has no order number
Row 7: has no order number

File: test2.csv file contains invalid data
Row 3: has no order number
Row 99: has no order number
... etc.

so are you suggesting that I have to write it to a file every time and create another transformation just to read this file?
In my transformation, the validator validates this order number and can able to capture and insert it to the database already. I wonder if I can able to create two writer inside that transformation that can insert to the database and write to a file in parallel - is this possible?

File Attachment:

File Name: twowritert...cord.zip
File Size:70 KB
Attachments:

Please Log in or Create an account to join the conversation.

More
8 years 10 months ago #11953 by John.Paterson
Sorry I've got confused about your original question.
I see what you mean now

John Paterson,
Support Engineer

Please Log in or Create an account to join the conversation.

More
8 years 10 months ago #11957 by John.Paterson
How about something like this?


John Paterson,
Support Engineer
Attachments:

Please Log in or Create an account to join the conversation.

More
8 years 10 months ago - 8 years 10 months ago #11959 by orlandoc
yes something like that. I have this captured already into the database, so how do i do that in parallel if i create another one that writes to a log file?
keep in mind that the ETL is smart enough not to send this log file again for the next run of the package if the next batch files have no errors. so there must be a flag or a way to send this log file via email ONLY if it encounter an error in point in time.

here are my another scenario where i need to collects all the files processed by the ETL on a given time ...

Attachments:
Last edit: 8 years 10 months ago by orlandoc.

Please Log in or Create an account to join the conversation.