O365 Message Monitor

My guess is that this may be a timing issue if there are existing scheduled packages using the O365 Reader Action on the same folder but with different filters.

In our case we have mailbox rules that move messages out of the primary inbox folder into another subfolder based on Subject and sender. This allows us to split out junk and other miscellaneous messages from the ETL flow. We get messages from numerous sources and they all havento be handled differently and since we didn't have a monitor we have matching scheduled packages all with their own filter. If we add a monitor on the same subfolder but with a uniqiue filter one can imagine it would work.

The thing is that the message rules move these good messages all into the same subfolder and we let the packages sort out which messages they want and then after processing them they move the message to another processed folder.

So, is the "total count" per filter or is based on the total number of unfiltered messages in the monitored folder?

Added to this is the fact that there is always a lag when reading a mailbox and so if the total count is based on all unfiltered messages then the count could appear to stay static as one message came in while another was being moved out.

Now I know it is count based I will see if I can build up a test suite that replicates what we see in our instance.

So, is the "total count" per filter or is based on the total number of unfiltered messages in the monitored folder?

It is based on the total number of unfiltered messages.(totalItemCount)
learn.microsoft.com/en-us/graph/api/user...h-rest-1.0&tabs=http

The error messages should be written into the log

Thanks Mike. So here is my take on things.... No solution just some thoughts...

The scenario we have is a monitor on a specific folder in the O365 Mailbox. This folder is a common staging area for ETL bound messages that come from numerous sources, have different formats, subject lines, attachment names and ultimately different ETL outcomes. 

This folder is not the parent INBOX folder but a subfolder of a subfolder for example INBOX/{PROD/PENDING

By doing this we can also have INBOX/DEV/PENDING and simply switch environments PROD versus DEV based on Variables. 

Because we have existing scheduled packages looking in this same folder for unique messages based on filters it is possible that messages will come and go quite frequently from this folder. 

I know this is over simplified but depending on the polling used by the monitoring agent itself then would it not be possible that the following could occur?

1. There is an existing message in the folder and so TotalItemCount = 1 when the monitoring agent first runs. Since this is the first run. Nothing happens.
2. Scheduled Job happens to fire and extracts the message and performs ETL.
3. Another message arrives - just one.
4. The monitoring agent runs again and gets a totalItemCount = 1 (again) and so does not think a new message has arrived
5. The new message which is expected to be consumed by the monitor is ignored as the count matched and nothing 'new' is seen.

A following thought I had was if the initial count started at 1 and then at step (3) 2 messages come in. Does the monitor then create 1 or 2 tasks? I am guessing it takes the difference between the start and finish counts which in this case would be 2-1 = 1. If that is the case then does it take the first message, last message or.... what happens if another message comes in whilst it is deciding? My guess it takes the first message and leaves the other alone, resetting the count back to 1 and missing the new message that came in.

Of course, all of this is speculative. But I know that having the mailbox watcher predictable would make a great asset for us. As it stands now, we have a lot of scheduled packages that we have random frequencies on to try and spread the load.

We also, cannot go Big Bang on the watcher alone as we need to keep existing packages running. But having existing scheduled packages affecting the TotalItemCount is probably the root cause of the problem. Plus, it is possible that there may be messages in the folder which do not match the filters and so just keep hanging around until they are manually managed. In this time, we could have messages come and go but the monitor (again depending on polling cycle) may not notice an item count change.

Another approach I thought of was to have the monitor run a single package that then ran a parent package that called all of the existing packages in series expecting one of them to find the data it needs. But that is highly inefficient, and the execution log only shows the result of running the parent and not the child. 

The other was that the monitor properties are extended to include the filter logic.

cheers

There is another option to use unreadItemCount and if it is more than 0 fire the event.
Then it would be up to the user to process the messages inside the folder and move them to the different one.
Again this is not the ideal solution because if the user opens the folder manually it may mark the mais as read accidentally.

I am against using filters in the monitor because it will put additional pressure on the server plus we would need to reprocess all messages every time we check the folder

Hi Mike

Makes total sense. We will just have to work around this as we are. In a real sense we are still loading the mail server anyway as we run numerous scheduled packages with filters. For us, a missed message can be quite time critical. We could have an external monitor looking for stale messages... However, this results in the same load on the mail server.

I had thought of having a single "New Mail Monitor" mainline package that then calls all of the other packages that would normally be scheduled. I had a Proof-Of-Concept of this, but it was difficult to see the end result in the Execution Log and so I reverted back to just scheduling all of the packages at different time offsets to reduce the burden. Because in our case it could call 15+ separate packages all then need to interrogate the mailbox to see if the 'new message(s) were meant for it. Then when they end there is no visual FAILED element in the Execution Log as they are called as a sub-package of a parent and if one fails then we only need to 'see' that one. We do of course get mail messages etc. But when troubleshooting it is good to use the Execution log to drill down into the child logs and not knowing which parent run failed (visually) just makes that process time consuming. If that makes sense.

A different version of this is to have the mainline package generate a Queue event for each of the typically scheduled tasks. But this still ends up being a load on the mail server - either way.

The only other approach I can think of is to have a separate mailbox folder per sender and have mailbox rules push the messages to the appropriate folder. Then a separate monitor for each folder.  The problem with this is that in our case a sender may send different types of messages for different purposes. Hence, we still need to filter these messages anyway. Additionally, we have a strategy of retaining the inbound message in a 'processed' folder based on the sender and purpose. Managing mailbox rules is tedious enough as it does not have a very good interface or a way of organising the rules thus making maintenance problematic.

I like the option of using the Monitor but how to avoid new messages being missed is still an issue and, in our case, we still need to parse the message to determine who and what so we can call the required package and transformations.

Cheers

Hi All

We are letting everyone know that we have just released a new version of our ETL software.

In this version, Office365 email monitor uses unreadItemCount instead totalItemCount.
The event is fired when unreadItemCount is more than zero.
To help with the debugging it writes "unreadItemCount" value into the debug log

Please let us know if it works for you or not.