Faulting aetlcl.exe zlib1.dll

AETLE v6.3.10.9 x64

Hi today we had a very unstable instance of AETLE running on Windows 2019 x64 server. By unstable I mean the host itself was very laggy when using the desktop and many other application were slow to start and lacked responsiveness when they did. The CPU and memory did not seem to be over consumed. And at some point we even lost the AETLE options.ini file in that it was there but completely empty.

We rarely use the Designer on the server but when we did during recovery we noticed that the Global Variables were missing and that the log files were being written to the Root of C Drive. This was due to the empty options.ini file.

After recovering all of that and restarting various services we then found numerous Windows Application logs all indicating that ZLIB1.DLL was faulting.

We use the ZIP Package Action quite a lot and I suspect that this exasperated the issue. We did find that one of package had been set to run as a 32bit platform and it is a scheduled package that runs several times a day. We have now made sure it is configured to run as a x64 platform.

We also found at one time numerous aetlcl.exe instances running and then they disappeared. Very strange. There were also numerous "Executing" Packages some of which were related to the package mentioned above.

As a remedial move and after adjusting the package mentioned I stopped all of the services, cleared out the QUEUE table and gradually restarted the ETL Directory monitor and Agent Execution services.

I am not sure of the root cause but have attached one of the many Windows Error Reporting files we found all generated over the past 24 hours. We will continue to monitor.



kind Regards
Bruce

The computer/server may become very slow and unresponsive if there too many processes running.
If you had hundreds aetcl.exe that would be the case.
Some of our customers reported similar issues in the past.
This might happen when aetcl.exe is unable to connect to something.
For example, it cannot connect to SQL server and it waits for a timeout
Another possible cause is running too many tasks in parallel.
You said that options.ini was missing.
The "Execute one action at a time" option is stored in this file.
That could the root cause of the problem

Thanks Peter.

With this issue in mind. What would your suggestion be to manage the following.

We get numerous inbound data feeds, generally by email from numerous senders.

The time of delivery can fluctuate dramatically and in many cases there could be several emails from the same source send the same files again due to a correction.

Each file is different in layout and content.

We therefore generally setup a package to poll the mailbox on regular intervals matching on sender, subject and attachment name. We currently have a package for each unique combination. Probably close to 30 of them. We tried the IMAP4 monitor but I found it to be sometimes unstable . We have now converted all to use the new O365 reader but there is no matching monitor.

So at this stage we are stuck with scheduling each of these packages to run basically all day during business hours. We typically have a frequncy set to every 15 minutes and different initial start times.

As you can imagine we basixally have a mailbox poll package running every few minutes and.of course there could be a few at the same time. Plus there a number of directory monitors happening as well.

So it is very possible that we are starting to hit some type of resource ceiling.

Do you have any suggestions on how we can manage the execution schedule better?

Thanks again.

Hi Bruce

1. Instead of clearing QUEUE table, you could have just set "Execute one action at a time " to true.
This will prevent the agent from running so many aetlcl.exe processes.

2. Having thought about the issue you have just had I think the best option would be to prevent the agent from running too many aetlcl.exe processes. EG add a "Max number of parallel tasks" option This can be done by checking the QUEUE table before starting the package.

This however may not work if you have to run a lot of packages in parallel. In this case, adding more computers may be required.

Thanks Peter. Will keep monitoring and take a look at some of your suggestions.

Cheers