Execution Log / Log Object

I am trying to track down some issues that are occurring in a transformation I am developing. There is an ongoing issue where we do not get helpful information from the log to debug a specific record.

In this case I am getting the following error: "Error 11/11/2013 4:00:16 PM 214 Writer: {ROS Table}, [`ros`] Record/Line:0 Record Rejected: Field: Mandatory field roaccountname is missing"

So I decided to explore using the Log object and write the log out to a table in my database. I was hoping to get more detail. Obviously this did not work. Here is what I get out of the database. "Writer: {ROS Table}, [`ros`] Record/Line:0 Record Rejected: Field: Mandatory field roaccountname is missing". In addition I discovered that no data is written to the transformation log tab.

One difference that I see between the error messages is that the one in the native AETL Execution Log is that there is a Record value of 214. This value is not stored in the database.

The value of 214 is of some value to me. I have to go to the transformation itself, change the number of rows to show, then scroll through the list to see what record 214 is. I look at that row then look at an ID field that is meaningful to me. I am not able to go directly to line 214 since this is more than the 100 item limit allowed in the Transformation Editor. So I have to filter my data using the id that I looked up from the main screen. At least I can use the global variables trick to filter my data to the offending record.

This convoluted process is time consuming.

Basically there are three issues / points in this post:

1. The log object that writes to a writer does not include the same data as the Execution Log at the transformation itself.
2. While the limit of 100 source records is understandable and a good idea we do need the option to increase that to a higher limit like we can in the transformation itself.
3. We really need a way to pick a field that the Execution Log / Log Object can add to the existing data to help identify a record via an ID I know about. For example, after going through the process I now know that record 214 is actually ROID 1431. Knowing this up front from the log would have saved a lot of time going through all of the other process. It should be noted that the record number is variable depending on the source data. This means that the record number is a moving target as the source data changes.

I know you have done a lot of work in the area of logging lately. I appreciate seeing it in the places where it is evident. It is not evident that changes have been made to the execution log in the error above. I am hoping that the suggestions above can be considered and implemented. They would result in a huge time saver for us.

I know I push back a lot with ideas. I appreciate your tolerance of this. I know the enhancements you have been able to incorporate since I started using the program have been very much appreciated by me (I do not know about the entire user base though). What I am trying to say is "Thank You" for all you do to make our work easier.

While the limit of 100 source records is understandable and a good idea we do need the option to increase that to a higher limit like we can in the transformation itself.

>>>We have just removed the limit, this might however bring different issues like running out of memory in some cases

We really need a way to pick a field that the Execution Log / Log Object can add to the existing data to help identify a record via an ID I know about. For example, after going through the process I now know that record 214 is actually ROID 1431. Knowing this up front from the log would have saved a lot of time going through all of the other process. It should be noted that the record number is variable depending on the source data. This means that the record number is a moving target as the source data changes.

>>>In some situations the errors are generated by the server and it is not possible to link it back. Even when it is possible it will require enormous effort from our site. We will concentrate n different issues for now.

M

In this case I am getting the following error: "Error 11/11/2013 4:00:16 PM 214 Writer: {ROS Table}, [`ros`] Record/Line:0 Record Rejected: Field: Mandatory field roaccountname is missing"


You are constantly wasting your own time by not using validator affter this what is it for

M

I am curious how the validator would resolve this issue and would like to understand that with my data.


In the specific instance above the target field is required and the default in MySQL is ''. As we discussed in another thread you do not support the default of '' as that is not correct data design or something. The data going into that field was the result of a lookup. I tried doing "IF(LENGTH(roaccountname) > 0, roaccountname, '') AS roaccountname," in the SQL Source Data, however, the space gets stripped out. That then leaves me with the task of creating 50+ calculations where I have code that is like this"

begin
Result:=Iif(Length([F001]) = 0, ' ',[F001]);
end;

That does work.

I have tried using the validator for this and the '' seems to get stripped out at least in some cases.

In the specific instance above the target field is required and the default in MySQL is ''. As we discussed in another thread you do not support the default of '' as that is not correct data design or something.

>>>>It is not correct it is supported now

The data going into that field was the result of a lookup. I tried doing "IF(LENGTH(roaccountname) > 0, roaccountname, '') AS roaccountname," in the SQL Source Data, however, the space gets stripped out. That then leaves me with the task of creating 50+ calculations where I have code that is like this"

>>>There is function called "Is Empty" it does what you need. As i said validator is very usefull object

M

Ok - I put a validator between the transformer and writer. I get a message that "It is not reccomended to connect this object to writer object, use transform instead". If it is not recommended then how does the validator help? The validator before the transformer does not have the field that needs the validation so I cannot validate there.