- Posts: 290
- Thank you received: 1
Array list equivalent in ETL?
8 years 10 months ago #11947
by orlandoc
Array list equivalent in ETL? was created by orlandoc
Hi,
One of my package is using a file system to get list of files and audit each file name to the database and send out an email with the list of file names listed in the email body.
What I did inside the transformation is pass the FileName to a variable <FileNameVar> and in the Send Email function in the package level I pass that variable <FileNameVar>.
If I have 2 files in that file system, the transformation can able to audit those two successfully but the email were only sent one file name - so the ETL variable can only store one value. So is there a way to collects all the filename values during transformation in a variable array?
One of my package is using a file system to get list of files and audit each file name to the database and send out an email with the list of file names listed in the email body.
What I did inside the transformation is pass the FileName to a variable <FileNameVar> and in the Send Email function in the package level I pass that variable <FileNameVar>.
If I have 2 files in that file system, the transformation can able to audit those two successfully but the email were only sent one file name - so the ETL variable can only store one value. So is there a way to collects all the filename values during transformation in a variable array?
Please Log in or Create an account to join the conversation.
- John.Paterson
- Offline
- New Member
-
8 years 10 months ago #11948
by John.Paterson
John Paterson,
Support Engineer
Replied by John.Paterson on topic Array list equivalent in ETL?
This is far too complex ant not really necessary.
Better option is to copy all files you want to email into separate folder and
email all of them in one go using mask
You can also compress the first
Better option is to copy all files you want to email into separate folder and
email all of them in one go using mask
You can also compress the first
John Paterson,
Support Engineer
Please Log in or Create an account to join the conversation.
8 years 10 months ago #11951
by orlandoc
Replied by orlandoc on topic Array list equivalent in ETL?
Array list or holding list of records in the memory level will helps me in many different scenarios like this ...
Imagine these scenario ...
ETL has to process hundreds of CSV files - validates the contents of each file row by row.
Let's say I have to validate order number in every rows and this particular csv file contains 100+ rows, so let's say there are 3 incoming csv files and two of them contains several empty order number. So I want to compose my email body like this ...
File: test1.csv file contains invalid data
Row 1: has no order number
Row 4: has no order number
Row 7: has no order number
File: test2.csv file contains invalid data
Row 3: has no order number
Row 99: has no order number
... etc.
so are you suggesting that I have to write it to a file every time and create another transformation just to read this file?
In my transformation, the validator validates this order number and can able to capture and insert it to the database already. I wonder if I can able to create two writer inside that transformation that can insert to the database and write to a file in parallel - is this possible?
Imagine these scenario ...
ETL has to process hundreds of CSV files - validates the contents of each file row by row.
Let's say I have to validate order number in every rows and this particular csv file contains 100+ rows, so let's say there are 3 incoming csv files and two of them contains several empty order number. So I want to compose my email body like this ...
File: test1.csv file contains invalid data
Row 1: has no order number
Row 4: has no order number
Row 7: has no order number
File: test2.csv file contains invalid data
Row 3: has no order number
Row 99: has no order number
... etc.
so are you suggesting that I have to write it to a file every time and create another transformation just to read this file?
In my transformation, the validator validates this order number and can able to capture and insert it to the database already. I wonder if I can able to create two writer inside that transformation that can insert to the database and write to a file in parallel - is this possible?
Please Log in or Create an account to join the conversation.
- John.Paterson
- Offline
- New Member
-
8 years 10 months ago #11953
by John.Paterson
John Paterson,
Support Engineer
Replied by John.Paterson on topic Array list equivalent in ETL?
Sorry I've got confused about your original question.
I see what you mean now
I see what you mean now
John Paterson,
Support Engineer
Please Log in or Create an account to join the conversation.
- John.Paterson
- Offline
- New Member
-
8 years 10 months ago #11957
by John.Paterson
John Paterson,
Support Engineer
Replied by John.Paterson on topic Array list equivalent in ETL?
John Paterson,
Support Engineer
Please Log in or Create an account to join the conversation.
8 years 10 months ago - 8 years 10 months ago #11959
by orlandoc
Replied by orlandoc on topic Array list equivalent in ETL?
yes something like that. I have this captured already into the database, so how do i do that in parallel if i create another one that writes to a log file?
keep in mind that the ETL is smart enough not to send this log file again for the next run of the package if the next batch files have no errors. so there must be a flag or a way to send this log file via email ONLY if it encounter an error in point in time.
here are my another scenario where i need to collects all the files processed by the ETL on a given time ...
keep in mind that the ETL is smart enough not to send this log file again for the next run of the package if the next batch files have no errors. so there must be a flag or a way to send this log file via email ONLY if it encounter an error in point in time.
here are my another scenario where i need to collects all the files processed by the ETL on a given time ...
Last edit: 8 years 10 months ago by orlandoc.
Please Log in or Create an account to join the conversation.