Dynamic READER Line Skip

AETLE x64 v6.3.2.11

Hi, I have not seen an out of the box solution for this. But I cam across a scenario today where an inbound CSV file came in from a trading partner with an extra header line. We normally skip 3 rows to get to the header columns but in this case the file had an extra set of Carriage Line Feeds thus pushing the header row down one to line 4. Consequently the transformation failed.

I can probably introduce an OS call to the parent package to strip blank header rows from the inbound files before passing it to the transformation. But I was wondering if there was an already existing feature to do something similar.

Another approach may be for AETL to have an extra READER TEXT TAB property check box to strip Line Terminators where there is no data other than the line terminators.

The backstory here is that we have/had two CSV files coming in as email attachments that had 2 rows that needed to be ignored before the header columns which appeared in row 3. The trading partner stills sends one of these attachments but has now decided to split the two attachments into two separate emails and in addition change the second attachment by adding a blank row to the front of the file. Everything else is identical.

I could resolve this issue by coming up with a second transformation that handles this but then all of the mapping logic needs to be duplicated and then maintained in multiple places.

SO my thought was to somehow generically handle this by stripping the blank rows before the header if they exist other wise just treat it like normal. I have subsequently realised that the blank rows is not just a Carriage Line Feed combination as the file was originally an excel file that was converted to a CSV prior to AETL getting it.

Therefore, the reality is that whilst I still think being able to strip empty rows or ROWS with some type of conditioning BEFORE looking for the header columns would be a good idea I thought could probably overcome this by using a Validator and discarding the blank rows (or rows with certain values in it).

My problem is that the COLUMN HEADERS in one instance are in Row 3 and then in another are in Row 4. So I am not sure how to do this and keep all of the mapping links I have done in the transformer.

Another option is that I use an external CLI (such as SED) to strip the rows before the Transformation is called but if I can do it all inside AETL then I think it is a cleaner solution.

Sample 1 - Column Header at Row 3
Sample 2 - Column Headers at Row 4

Just wanted to say I resolved this by

1. Changing the reader to NOT SKIP any header rows
2. creating a validator that discarded rows that had column header text or blank/null values

This then allowed me to cater for the floating Column header row whilst still retaining my field mappings against reader field names

Yes, using validator is the way to sort out this kind of problems