- Posts: 68
- Thank you received: 4
After server outage, agent executed two of all missed tasks
5 years 2 months ago #18349
by Carl A
Greetings,
Our server had a several hour outage on Friday July 13th. When the server restarted all the missed tasks executed twice. This caused some files to process and be emailed twice, which could have been quite an issue, but luckily was not. I have included an image of the execution log, but don't have much more to help.
AETL Processor Enterprise (32 Bit) 6.3.2.13
Windows Server 2008 R2 Enterprise SP1
Repository SQL server 2014
Our server had a several hour outage on Friday July 13th. When the server restarted all the missed tasks executed twice. This caused some files to process and be emailed twice, which could have been quite an issue, but luckily was not. I have included an image of the execution log, but don't have much more to help.
AETL Processor Enterprise (32 Bit) 6.3.2.13
Windows Server 2008 R2 Enterprise SP1
Repository SQL server 2014
Please Log in or Create an account to join the conversation.
5 years 2 months ago #18350
by Maria
Replied by Maria on topic After server outage, agent executed two of all missed tasks
I can see from the log that you have duplicates, for example, ID 347845 is duplicated.
That can be a result of server failure or repository corruption.
If your repository is MS Access please run compact and repair.
The log consists of 4 tables.
QUEUE, QUEUE_HISTORY, QUEUE_ACTIONS, QUEUE_ACTIONS_HISTORY
QUEUE and QUEUE_ACTIONS tables hold information about the actions which are waiting to be executed or being executed.
Once execution is completed the records are moved from QUEUE to QUEUE_HISTORY and from QUEUE_ACTIONS to QUEUE_ACTIONS_HISTORY
If the software is not able to move records you will see the duplicates.
Please try the following
Stop the agent.
run the following SQL against the repository.
delete from QUEUE_ACTIONS
delete from QUEUE
Keep us posted
That can be a result of server failure or repository corruption.
If your repository is MS Access please run compact and repair.
The log consists of 4 tables.
QUEUE, QUEUE_HISTORY, QUEUE_ACTIONS, QUEUE_ACTIONS_HISTORY
QUEUE and QUEUE_ACTIONS tables hold information about the actions which are waiting to be executed or being executed.
Once execution is completed the records are moved from QUEUE to QUEUE_HISTORY and from QUEUE_ACTIONS to QUEUE_ACTIONS_HISTORY
If the software is not able to move records you will see the duplicates.
Please try the following
Stop the agent.
run the following SQL against the repository.
delete from QUEUE_ACTIONS
delete from QUEUE
Keep us posted
Please Log in or Create an account to join the conversation.
5 years 2 months ago #18351
by Carl A
Replied by Carl A on topic After server outage, agent executed two of all missed tasks
The issue is not the duplicates in the execution log. The issue is each process EXECUTED twice which can cause issues for our clients. Example, a process checks for a file, if present it zips and emails to the client, now because the process has two executions going at the exact same time, it creates two zips and attaches both to two separate emails. So the client has received two emails with two identical zips. Basically, I am asking your team to look at how AETL Processor handles scheduled tasks when the agent restarts after a several hour downtime.
Also, this is the first time I have seen this behavior but I have not had significant downtime in quite a while, luckily.
Thank you
Also, this is the first time I have seen this behavior but I have not had significant downtime in quite a while, luckily.
Thank you
Please Log in or Create an account to join the conversation.
5 years 2 months ago #18352
by Maria
Replied by Maria on topic After server outage, agent executed two of all missed tasks
Maybe they were executed twice.
My feedback is based on the screenshot provided, unfortunately, I cannot see the fields headers.
By looking the screenshot I can see that some records have exactly same dates and times and ID's
I guess the date time is [SUBMITED] field.
Therefore I assumed that the records were not deleted from [QUEUE] table
EG: One record on the screen is from [QUEUE] table and another is from [QUEUE_HISTORY]
SELECT * FROM [QUEUE] where [QUEUE_ID]=347845
SELECT * FROM [QUEUE_HISTORY] where [QUEUE_ID]=347845
So it looks to the end user that it was executed twice but actually it was not.
Therefore I suggested deleting records.
FYI Below is the SQL executed once package execution is completed
insert into QUEUE_HISTORY select * from QUEUE where QUEUE_ID=FQUEUE_ID
insert into QUEUE_ACTIONS_HISTORY select * from QUEUE_ACTIONS where QUEUE_ID=FQUEUE_ID
delete from QUEUE_ACTIONS where QUEUE_ID=FQUEUE_ID
delete from QUEUE where QUEUE_ID=FQUEUE_ID
Handling failures:
when the agent fails to connect to the repository it writes an error message into the log file located in the same folder.
We did a lot of agent testing: stopping the repository database, rebooting servers, disconnecting network cables and every time agent was able to recover
My feedback is based on the screenshot provided, unfortunately, I cannot see the fields headers.
By looking the screenshot I can see that some records have exactly same dates and times and ID's
I guess the date time is [SUBMITED] field.
Therefore I assumed that the records were not deleted from [QUEUE] table
EG: One record on the screen is from [QUEUE] table and another is from [QUEUE_HISTORY]
SELECT * FROM [QUEUE] where [QUEUE_ID]=347845
SELECT * FROM [QUEUE_HISTORY] where [QUEUE_ID]=347845
So it looks to the end user that it was executed twice but actually it was not.
Therefore I suggested deleting records.
FYI Below is the SQL executed once package execution is completed
insert into QUEUE_HISTORY select * from QUEUE where QUEUE_ID=FQUEUE_ID
insert into QUEUE_ACTIONS_HISTORY select * from QUEUE_ACTIONS where QUEUE_ID=FQUEUE_ID
delete from QUEUE_ACTIONS where QUEUE_ID=FQUEUE_ID
delete from QUEUE where QUEUE_ID=FQUEUE_ID
Handling failures:
when the agent fails to connect to the repository it writes an error message into the log file located in the same folder.
We did a lot of agent testing: stopping the repository database, rebooting servers, disconnecting network cables and every time agent was able to recover
Please Log in or Create an account to join the conversation.