Processing Large Numbers of Email Messages
The video replaces many separate Office 365 receive packages with one workflow for processing large numbers of email messages in Advanced ETL Processor. The completed package is configured to retrieve each message through an Office 365 Email Loop, check it against several routing rules, save matching attachments into the correct network folder, and route successful extraction to Delete Message. The business result is one maintainable package instead of a small army of scheduled jobs all asking the mail server the same question every minute.
Video Walkthrough: Processing Large Numbers of Email Messages
Watch how the original one-package-per-rule design is consolidated into a single loop with message checks, attachment extraction actions, success routing, and deletion.
Watch Processing Large Numbers of Email Messages on YouTube.
How to Process Large Numbers of Email Messages
The sequence below follows the package shown in the video. The mailbox connection and the exact condition inside each message check are not opened in the recording, so those details are intentionally not invented here.
Step 1: Consolidate the repeated Office 365 jobs
The original design uses many similar packages. Each package connects to Office 365, applies one filter, extracts an attachment, and is configured to repeat every minute. Create one combined package instead so the mailbox connection is not repeatedly opened by every separate rule.
Result: the video opens one package named 003 Process Emails containing all visible routing branches.
Step 2: Start with Office 365 Email Loop
Add Office 365 Email Loop as the first action. The loop retrieves messages through Microsoft Graph API and supplies one current message to the package at a time.
Result: downstream actions can inspect the current message without reconnecting once for every routing rule.
Step 3: Check the current message against each rule
Add branches using Check Email Message. The completed canvas shows checks such as Reqnoorders, AP Age Debt, AP payment history, AR Age Debt, GLT, Order status, and Transactions APAR. The narrator gives the attachment name as one possible criterion; the recording does not open the exact expression used by each branch.
Result: a matching message follows the corresponding extraction branch, while the loop continues evaluating the package flow.
Step 4: Extract attachments into the branch folder
Connect each matching branch to Extract Email Attachments. In the inspected Reqnoorders action, Execution Result is Defined By Action, Message is #Message#, and Folder is \\fin-aptos\data upload\nep\reqs no orders. Use Attachment File Save Filter is False.
Result: the action is configured to save attachments from the current matching message into that rule’s network folder.
Step 5: Delete only after attachment extraction succeeds
The inspected extraction action routes On Success to [1]-Delete Message. The package’s successful branches converge on an action named Delete Message. Current Office 365 Email Action documentation also documents delete and move operations.
Result: a successfully handled message is configured to be removed so it is not collected again on the next scheduled run.
Step 6: Return to the loop for the next message
Connect the completed path back to Office 365 Email Loop. The loop advances to the next message and repeats the same checks and extraction routes.
Result: one package is configured to process the selected folder one message at a time. The video calls this batch processing, but it does not show a configurable batch-size or per-run message-limit setting.
003 Process Emails package. Notice the single Office 365 Email Loop on the left, rule-specific extraction paths in the centre, and the shared Delete Message action on the right.What the Completed Workflow Does
- Starts with one Office 365 Email Loop configured to retrieve messages one at a time.
- Passes the current message through multiple named routing branches.
- Routes a match to its corresponding Extract Email Attachments action.
- Configures that action to save attachments into the branch’s network folder.
- Routes successful extraction to Delete Message.
- Connects the completed path back to Office 365 Email Loop for the next message.
The video does not demonstrate a failure route, retry queue, duplicate table, message archive, or final schedule for the combined package. Those are production design decisions, not hidden features we should pretend appeared off-camera.
Designing the Workflow for a Large Mailbox
- Reduce repeated connections: one loop package replaces many one-rule packages that reconnect on separate schedules.
- Process one current message: the loop supplies each message to the same set of checks and actions in turn.
- Separate retrieval from processing: Office 365 Email Loop retrieves the message; Check Email Message routes it; Extract Email Attachments writes the files.
- Change message state after success: the demonstrated extraction action reaches Delete Message through its success output.
- Keep mailbox size and run size distinct: the mailbox may contain many messages, but the loop handles one current message at a time. Neither the video nor current loop documentation shows a batch-size limit.
- Schedule the combined package: use the Scheduler at an interval appropriate for the mailbox and downstream work. The final schedule is not shown in the video.
- Review execution information: use the Execution Monitor and action logs while testing. Detailed action logging can create large files and should not be left on casually in production.
For an IMAP mailbox, the documented IMAP4 Email Loop provides the same one-message-at-a-time pattern. The embedded video itself uses Office 365.
Common Problems When Processing Large Mailboxes
- Too many scheduled receive packages: the video’s original jobs all repeat every minute. The narrator warns that this repeated connection and disconnection work can lead to throttling or denied requests.
- No defined error branch: the inspected extraction action shows
On Error = [-1]-Not Defined. Add a review or retry path before relying on the package unattended. - Deleting before downstream success: route deletion from the successful end of attachment or data processing, not from message retrieval alone.
- No duplicate safeguard: successful deletion prevents the normal next run from seeing that message, but the video shows no independent idempotency key if deletion or downstream processing only partly completes.
- No retained original: the demonstrated package deletes successful messages and does not show an archive step. Move or save the message first if audit and replay requirements demand it.
- One malformed message stopping the run: isolate failures so the message remains visible and later messages can still be handled. This branch is not present in the recording.
Where this email-loop pattern fits
Reports and scheduled files
Match a known report or attachment name and save it into the network folder used by the next controlled process.
Supplier and order messages
Route order confirmations, supplier files, or invoice attachments through separate checks before downstream validation or database loading.
Support requests and alerts
Use sender, subject, recipient, attachment, or body rules to direct a message to the appropriate workflow branch.
Continue the Incoming-Email Workflow
Return to the Email Automation hub for receiving, extracting, sending, logging, and reply-tracking guides.
Review Advanced ETL Processor Enterprise or use the ETL Tools download page to get the fully functional 30-day trial.
Questions About Processing Large Numbers of Email Messages
How can I process a large number of email messages automatically?
Use one Office 365 Email Loop or IMAP4 Email Loop package to retrieve messages one at a time. Check each message against your rules, process matching attachments, and change the message state only after its downstream action succeeds.
Should all mailbox messages be processed in one run?
Not necessarily. The video loops through the messages in the selected folder one at a time, but it does not show a configurable message limit. Use a controlled mailbox folder, test with representative volumes, and choose a schedule that respects the mail provider and the work performed for each message.
How can I prevent the same email from being processed twice?
The demonstrated workflow routes successful attachment extraction to Delete Message, so it is configured to remove that message before the next run. The video does not show a separate duplicate key; add one before production if deletion can fail or downstream work must be idempotent.
What should happen when processing one message fails?
Route the failed message to a defined review, retry, or failure path and leave it available for investigation. The inspected Extract Email Attachments action shows On Error as Not Defined, so failure isolation still needs to be designed for production.
Can email attachments be processed automatically?
Yes. The video uses Extract Email Attachments actions to save each matching message’s attachments into a rule-specific network folder. Further file validation or database loading can follow after extraction.
Should processed messages be moved to another folder?
The video deletes successfully handled messages. The current Office 365 Email Action also supports moving a message into a folder, which is often preferable when retention, auditing, or manual review is required.
Can the workflow run on a schedule?
Yes. Advanced ETL Processor can schedule packages at regular intervals. The video replaces many separate jobs repeating every minute with one consolidated package, but it does not show the final package’s schedule.
Does Advanced ETL Processor process email on my own infrastructure?
Yes. Advanced ETL Processor is self-hosted and the workflow runs on your Windows infrastructure. An Office 365 mailbox is still accessed through Microsoft Graph API, so the mailbox remains with its configured provider.
Can I test this workflow before buying?
Yes. Download the fully functional 30-day trial with no registration required. Test the package against a controlled mailbox folder before enabling a production schedule.
Build Your Own Email Processing Workflow
Download the fully functional 30-day trial and test the workflow with a controlled mailbox folder before scheduling it.
Direct link, no registration required.