Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1
  • 2

TOPIC: No warning or error when data was dropped when loading to SQL Server Table

No warning or error when data was dropped when loading to SQL Server Table 2 months 2 weeks ago #18987

Hi,

AETLE x64 v6.3.4.4

We have a process that picks up a CSV file with a number of decimal values. In this case we had a value of the format 12345678.123456789

The staging table used to hold this data before loading had a decimal column defined as (15,5). In most cases this is more than adequate and the fix was simple.

However, in this case the row was 'transparently' ignored or dropped during the transformation/load step. I say 'transparently' because we only found out after another business unit reconciled the data and noticed the missing row. Reviewing the AETL log showed no indication of an error or even a warning that a row had been dropped.

Is there something we have missed in our Transformation Actions that would have flagged this as an issue?

Attachments:

Please Log in or Create an account to join the conversation.

No warning or error when data was dropped when loading to SQL Server Table 2 months 2 weeks ago #18988

Hi Bruce.

We will try to reproduce the problem and get back to you
Peter Jonson
Support Analist

Please Log in or Create an account to join the conversation.

No warning or error when data was dropped when loading to SQL Server Table 2 months 2 weeks ago #18991

  • admin
  • admin's Avatar
  • Offline
  • Moderator
  • Moderator
  • Deeds not Words
  • Posts: 7522
  • Karma: 33
  • Thank you received: 409
I think there is slight confusion on how it works.
The screenshot you posted is a package log. (1).
The error messages are written into the (2).



Mike
Attachments:
The following user(s) said Thank You: bruce.gibbins

Please Log in or Create an account to join the conversation.

No warning or error when data was dropped when loading to SQL Server Table 2 months 2 weeks ago #18994

ahhh.. Thanks for this and I have been caught with this before. So appreciate the reminder. However, I'd like to suggest this possibly an area that we could do with some type of feature enhancement. Specifically, that we have the option to let the parent package know of errors or perhaps even warnings generated by the child transformation.

I understand that we have ODBC in the mix here and in fact may get different feedback from different drivers but in this case we could have caught the issue in the back-end in preference to it being caught out by a manual reconciliation process had we had early warning.

As your example highlights the Package log had no indication of an error from the transformation and yet the transformation "sub-log" indicated that rows were being dropped from the source data.

We could build in some type of data count and check on actual versus expected based on the source data but in our case the effort would not necessarily be justified given the low rate of occurrence. But still something to be considered if we can get some type of alert cascading back-up to the parent.

thanks again

Please Log in or Create an account to join the conversation.

No warning or error when data was dropped when loading to SQL Server Table 2 months 1 week ago #18995

  • admin
  • admin's Avatar
  • Offline
  • Moderator
  • Moderator
  • Deeds not Words
  • Posts: 7522
  • Karma: 33
  • Thank you received: 409
Hi Bruce

I assume this is what you want?

Mike
Attachments:

Please Log in or Create an account to join the conversation.

No warning or error when data was dropped when loading to SQL Server Table 2 months 1 week ago #18996

Yes, visually that would be what I had expected. Would that then bubble up to the package as an error event that could be caught by the "On Error" handler in the package?

Please Log in or Create an account to join the conversation.

  • Page:
  • 1
  • 2

We have 205 guests and no members online