Data validation using SQL Query

More
12 years 9 months ago #1277 by abuthangam
Hi,
Aplogies if this has been already explained. I am not able to find the info in the forum. Hence, posting it.

I am evaluating Advanced ETL Processor Ent for my Company.

I am loading Customer Master data from Excel file into MySQL database. The source file has got the following fields.
1. EMP_NO
2. COMPANY_NO
3. NAME
4. DOB
...etc

EMP_NO should be checked against EMPLOYEE table to make sure that the Employee does not exist.
COMPANY_NO should be checked against COMPANY table to make sure that the Company exists.

I am trying to implement this check in Validator component. Is this possible? If not, how these validations can be implemented?

Also, is there any option to execute parameterised SQL query, without using Lookup? For example, I would like to execute the below query with the input fields <name> and <company_no> from Excel source, to get the EMP_NO, which will be mapped to an output field

SELECT EMP_NO FROM EMPLOYEE WHERE NAME=<name> AND COMPANY_NO=<company_no>

Thanks & Regards,
Syed.

Please Log in or Create an account to join the conversation.

More
12 years 9 months ago - 12 years 9 months ago #1280 by admin
Hi

1 Use two InList functions

A ) EMP_NO should be checked against EMPLOYEE table to make sure that the Employee does not exist.
use
select distinct EMP_NO from EMPLOYEE
as a source
B ) COMPANY_NO should be checked against COMPANY table to make sure that the Company exists.
use
select distinct COMPANY_NO from COMPANY
as a source

2 It is not possible at the moment but we will add support for it to the futurure release
In most of the cases lookup is faster than running select lookup_value from lookup_table
Imaging doing it for 2M records file

Mike

Mike
ETL Architect
Last edit: 12 years 9 months ago by admin.

Please Log in or Create an account to join the conversation.