After server outage, agent executed two of all missed tasks

More
5 years 2 months ago #18349 by Carl A
Greetings,

Our server had a several hour outage on Friday July 13th. When the server restarted all the missed tasks executed twice. This caused some files to process and be emailed twice, which could have been quite an issue, but luckily was not. I have included an image of the execution log, but don't have much more to help.

AETL Processor Enterprise (32 Bit) 6.3.2.13
Windows Server 2008 R2 Enterprise SP1
Repository SQL server 2014
Attachments:

Please Log in or Create an account to join the conversation.

More
5 years 2 months ago #18350 by Maria
I can see from the log that you have duplicates, for example, ID 347845 is duplicated.
That can be a result of server failure or repository corruption.
If your repository is MS Access please run compact and repair.

The log consists of 4 tables.

QUEUE, QUEUE_HISTORY, QUEUE_ACTIONS, QUEUE_ACTIONS_HISTORY

QUEUE and QUEUE_ACTIONS tables hold information about the actions which are waiting to be executed or being executed.

Once execution is completed the records are moved from QUEUE to QUEUE_HISTORY and from QUEUE_ACTIONS to QUEUE_ACTIONS_HISTORY

If the software is not able to move records you will see the duplicates.

Please try the following

Stop the agent.
run the following SQL against the repository.

delete from QUEUE_ACTIONS
delete from QUEUE

Keep us posted

Please Log in or Create an account to join the conversation.

More
5 years 2 months ago #18351 by Carl A
The issue is not the duplicates in the execution log. The issue is each process EXECUTED twice which can cause issues for our clients. Example, a process checks for a file, if present it zips and emails to the client, now because the process has two executions going at the exact same time, it creates two zips and attaches both to two separate emails. So the client has received two emails with two identical zips. Basically, I am asking your team to look at how AETL Processor handles scheduled tasks when the agent restarts after a several hour downtime.

Also, this is the first time I have seen this behavior but I have not had significant downtime in quite a while, luckily.

Thank you

Please Log in or Create an account to join the conversation.

More
5 years 2 months ago #18352 by Maria
Maybe they were executed twice.
My feedback is based on the screenshot provided, unfortunately, I cannot see the fields headers.
By looking the screenshot I can see that some records have exactly same dates and times and ID's
I guess the date time is [SUBMITED] field.
Therefore I assumed that the records were not deleted from [QUEUE] table

EG: One record on the screen is from [QUEUE] table and another is from [QUEUE_HISTORY]

SELECT * FROM [QUEUE] where [QUEUE_ID]=347845

SELECT * FROM [QUEUE_HISTORY] where [QUEUE_ID]=347845

So it looks to the end user that it was executed twice but actually it was not.

Therefore I suggested deleting records.

FYI Below is the SQL executed once package execution is completed

insert into QUEUE_HISTORY select * from QUEUE where QUEUE_ID=FQUEUE_ID
insert into QUEUE_ACTIONS_HISTORY select * from QUEUE_ACTIONS where QUEUE_ID=FQUEUE_ID
delete from QUEUE_ACTIONS where QUEUE_ID=FQUEUE_ID
delete from QUEUE where QUEUE_ID=FQUEUE_ID

Handling failures:

when the agent fails to connect to the repository it writes an error message into the log file located in the same folder.

We did a lot of agent testing: stopping the repository database, rebooting servers, disconnecting network cables and every time agent was able to recover

Please Log in or Create an account to join the conversation.