Add save_attachments action and provide enhancements to search_items action.

.gitignore

@@ -87,3 +87,6 @@ ENV/
 
 # Rope project settings
 .ropeproject
+
+# VS Code project settings
+.vscode

CHANGES.md

@@ -1,5 +1,45 @@
 # Change Log
 
+## 1.1.0
+
+* **Add** `save_attachments` action.
+    * Includes several _pack_ configuration options:
+        - `attachment_directory`: Fully-qualified server path name used to store attachments. Must be readable and writeable by Stackstorm. Defaults to "/opt/stackstorm/packs/msexchange/attachments".
+        - `attachment_folder_maximum_size`: Maximum storage space in MB (default is 50MB) alloted to `attachment_directory`. Pack maintenance process (see below) manages this.
+        - `attachment_days_to_keep`: Maximum number of days to keep saved attachments (default is 7 days). Also, managed by pack maintenance process.
+    * Uses same model as `search_items` for finding email messages for which to save attachments. Only _email_ messages and _file_ attachments are supported.
+    * Action returns a _list/array_ of dictionaries with following attributes:
+        - `email_subject` - Full subject of the email.
+        - `email_sent` - Date that email was sent.
+        - `sender_email_address` - Email address of the sender.
+        - `attachment_files` - _List/array_ of fully-qualified filenames from server of attachments saved. Example:
+        ```JSON
+        [
+            {
+                "email_subject": "ACCOUNT LIST - resend for testing",
+                "email_sent": "2021-06-25 13:54:34+00:00",
+                "sender_email_address": "someone@example.com",
+                "attachment_files": [
+                "/opt/stackstorm/packs/msexchange/attachments/Accounts_06_23_2021.xlsx"
+                ]
+            }
+        ]
+        ```
+    * If attachment filename is **not** unique in target folder, attempts to generate a unique filename for each attachment via several methods, including date sent, date _and_ time sent, and "random" 8-character string.
+    * Attachments can be saved either as BINARY (default) or TEXT format.
+    * An `attachment_directory_maintenance` sensor/trigger/rule combination has been implemented and, by default, runs once daily (polling interval of 86400 seconds) to enforce these rules through the `do_attachment_directory_maintenance` action. This action can be run manually, as well, if you need to override the pack configuration values; running manually it with no input values uses the pack configuration.
+    * Maintenance process removes files by age first and then, if necessary, deletes remaining files starting with _largest_ files until threshold is reached.
+    * Save attachment action has `replace_spaces_in_filename` enumerated value parameter (NONE [default], UNDERSCORE, OCTOTHORPE/HASH, and PIPE) to allow user to replace spaces in attachment file names, if desired. Default (NONE) is to preserve spaces.
+
+* **Enhancements** to `search_items` action.
+    * Moved search logic from `run` method in `search_items` action into `_search_items` utility method in `base/actions.py` to allow functionality to be shared by `search_items` and `save_attachments`.
+    * Added `search_start_date` parameter specifying the start date for items to search. (End date is always "today".) Date can be entered as free-form text. Most any date format is supported, as [`dateutil`](https://dateutil.readthedocs.io/) library is used to parse input to valid `datetime` value.
+    * Update `search_items` action to return additional item attributes, specific to _email_ messages, from `item_to_dict` helper method. Such attributes can be useful in filtering e-mails based on sender and/or originating domain.
+        - `sender_email_address` - Email address of sender.
+        - `email_recipient_addresses` - List/array of email recipients (from [`exchangelib`](https://ecederstrand.github.io/exchangelib/) `to_recipients` list **only**).
+    * Added optional `folder_name` parameter to `item_to_dict` helper method to include the name of the folder used in the search as attribute of returned dictionary.
+    * Update `requirements.txt` to include `python-dateutil` (see above) and `pytz`, which is needed for creating timezone-aware `exchangelib` [`EWSDateTime`](https://ecederstrand.github.io/exchangelib/exchangelib/ewsdatetime.html#exchangelib.ewsdatetime.EWSDateTime) objects for date searches.
+
 ## 1.0.0
 
 * Drop Python 2.7 support
@@ -15,7 +55,7 @@
 ## 0.1.3
 
 * Set default folder to `Inbox` for `search_items`
-* Fixed sensor bug with config object handling for non-autodiscovery systems 
+* Fixed sensor bug with config object handling for non-autodiscovery systems
 
 ## 0.1.2
 

README.md

@@ -6,14 +6,19 @@ This pack provides Microsoft Exchange integration to perform simple searches on
 Exchange Server 2010, 2013 and 2016 as well as Office 365 hosted Exchange accounts.
 
 ## Actions
+* `do_attachment_directory_maintenance` - Performance maintenance on server directory in which email file attachments are saved.
 * `get_calendar_items` - Get a list of calendar items within a date range
 * `get_folder` - Get information about a folder (mail, contact, meta)
 * `list_folders` - List all folders or subfolders within a folder
-* `search_items` - Search for items by subject within a folder (default folder Inbox)
+* `save_attachments` - Save _file_ attachments on _email_ messages to server directory.
+* `search_items` - Search for items by subject and/or date within a folder (default folder Inbox)
 * `send_email` - Send an email
 
+## Rules
+* `attachment_directory_maintenance` - Runs maintenance (storage usage) via `do_attachment_directory_maintenance` action on server directory in which file attachments are saved when triggered by associated sensor.
 
 ## Sensors
+* `attachment_directory_maintenance_sensor` - Runs maintenance  periodically (default daily).
 * `item_sensor` - Monitors the configured folder (Inbox by default) for new items and sends a `exchange_new_item` trigger when one is received
 
 ## Configuration
@@ -48,4 +53,9 @@ username: "bob@company.com"
 password: "B0bsPassword!"
 timezone: "Europe/London"
 sensor_folder: "My folder to monitor"
-```
+```
+
+### Email Attachment Configuration
+- `attachment_directory`: Fully-qualified server path name used to store attachments. Must be readable and writeable by Stackstorm. Defaults to "/opt/stackstorm/packs/msexchange/attachments".
+- `attachment_folder_maximum_size`: Maximum storage space in MB (default is 50MB) alloted to `attachment_directory`. Pack maintenance process (see below) manages this.
+- `attachment_days_to_keep`: Maximum number of days to keep saved attachments (default is 7 days). Also, managed by pack maintenance process.

actions/base/__init__.py

@@ -1,3 +1,6 @@
+from exchangelib import Mailbox, Message
+
+
 def folder_to_dict(folder):
     return {
         'id': folder.folder_id,
@@ -9,7 +12,7 @@ def folder_to_dict(folder):
     }
 
 
-def item_to_dict(item, include_body=False):
+def item_to_dict(item, include_body=False, folder_name=None):
     result = {
         'id': item.item_id,
         'changekeyid': item.changekey,
@@ -32,4 +35,16 @@ def item_to_dict(item, include_body=False):
     if not include_body:
         del result['body']
         del result['text_body']
+    if folder_name:
+        result["folder_name"] = folder_name
+    # If this is an email message, add sender and recipients.
+    if isinstance(item, Message):
+        result["sender_email_address"] = None
+        if isinstance(item.sender, Mailbox):
+            result["sender_email_address"] = str(item.sender.email_address)
+        result["email_recipient_addresses"] = list()
+        for recpt in item.to_recipients:
+            if isinstance(recpt, Mailbox):
+                result["email_recipient_addresses"].append(
+                    str(recpt.email_address))
     return result

actions/base/action.py

@@ -4,7 +4,8 @@
 from st2common.runners.base_action import Action
 from st2client.client import Client
 from st2client.models import KeyValuePair
-from exchangelib import Account, ServiceAccount, Configuration, DELEGATE, EWSTimeZone
+from exchangelib import (Account, ServiceAccount, Configuration, DELEGATE,
+                        EWSTimeZone, EWSDateTime)
 
 CacheEntry = namedtuple('CacheEntry', 'ews_url ews_auth_type primary_smtp_address')
 
@@ -54,6 +55,9 @@ def __init__(self, config):
                     access_type=DELEGATE)
             self._store_cache_configuration()
 
+        # Configure attachment parameters
+        self._attachment_configuration()
+
     def _store_cache_configuration(self):
         ews_url = self.account.protocol.service_endpoint
         ews_auth_type = self.account.protocol.auth_type
@@ -79,3 +83,126 @@ def _get_cache(self):
                 primary_smtp_address=primary_smtp_address.value)
         else:
             return None
+
+    def _attachment_configuration(self):
+        attach_dir = self.config.get("attachment_directory", None)
+        if not attach_dir:
+            try:
+                from st2common.content import utils as content_utils
+                pack_name = getattr(self.action_service._action_wrapper,
+                                    "_pack", "unknown")
+                pack_path = content_utils.get_pack_base_path(pack_name)
+                attach_dir = os.path.join(pack_path, "attachments")
+            except ImportError:
+                err_msg = (
+                    "Unable load import 'st2common.content.utils' "
+                    "library. Using pack default attachment directory of "
+                    "'/opt/stackstorm/packs/msexchange/attachments'.")
+                self.logger.error(err_msg)
+                attach_dir = "/opt/stackstorm/packs/msexchange/attachments"
+        else:
+            attach_dir = os.path.abspath(attach_dir)
+
+        # Create the folder/directory, if it doesn't exist,
+        # and make it writeable.
+        if not os.path.exists(attach_dir):
+            os.makedirs(attach_dir, exist_ok=True)
+            os.chmod(attach_dir, 0o755)
+            self.logger.info("Created directory '{dir}' and made writeable."
+                             .format(dir=attach_dir))
+
+        if not os.access(attach_dir, os.W_OK):
+            raise OSError("Unable to write to attachment directory '{dir}'."
+                          .format(dir=attach_dir))
+
+        self.attachment_directory = attach_dir
+        self.attachment_directory_maximum_size = int(self.config.get(
+            "attachment_directory_maximum_size", 50))
+        self.attachment_days_to_keep = int(self.config.get(
+            "attachment_days_to_keep", 7))
+
+    def _get_date_from_string(self, date_str=None):
+        """
+        Use dateutil library (https://dateutil.readthedocs.io/) to parse
+        unstructured date string to standard format.
+        :param date_str str: Date as string in unknown/unstructured format
+        :returns EWSDateTime object or None
+        """
+        # If date_str is not provided, we assume that this is for the *end*
+        # of the filter range, which we set to "now", using timezone from
+        # pack configuration.
+        if not date_str:
+            return EWSDateTime.now(tz=self.timezone)
+
+        try:
+            from dateutil import parser
+            import pytz
+            parsed_date = parser.parse(date_str)
+            utc_date = pytz.utc.localize(parsed_date)
+            local_date = utc_date
+            try:
+                local_date = utc_date.astimezone(self.timezone)
+            except Exception:
+                self.logger.error("Unable to convert search date to pack "
+                                  "timezone. Using UTC...")
+            start_date = EWSDateTime.from_datetime(local_date)
+            self.logger.debug("Search start date: {dt}".format(dt=start_date))
+        except ImportError:
+            self.logger.error("Unable to find/load 'dateutil' library.")
+            start_date = None
+        except ValueError:
+            self.logger.error("Invalid format for date input: {dt}"
+                              .format(dt=date_str))
+            start_date = None
+
+        return start_date
+
+    def _search_items(self, folder, subject=None, search_start_date=None):
+        """
+        Common method for searching for MS Exchange items (email messages,
+        calendar items, etc.). Used by _search_items_ and _save_attachments_.
+        """
+        folder = self.account.root.get_folder_by_name(folder)
+
+        start_date = None
+        if search_start_date:
+            start_date = self._get_date_from_string(search_start_date)
+            # For email messages, MS Exchange does not support using only a
+            # start date for searches. Instead, we must use a *range* of dates
+            # for search, so we set the *end* of range to "now".
+            # See https://stackoverflow.com/a/48742644 for details.
+            end_date = self._get_date_from_string()
+
+        if subject:
+            if start_date:
+                # First, try searching for messages...
+                try:
+                    items = folder.filter(
+                        subject__contains=subject,
+                        datetime_received__range=(start_date, end_date))
+                # Search on other items, which have regular "start" attribute.
+                except Exception:
+                    items = folder.filter(
+                        subject__contains=subject, start__gte=start_date)
+            else:
+                items = folder.filter(subject__contains=subject)
+        else:
+            if start_date:
+                try:
+                    items = folder.filter(
+                        datetime_received__range=(start_date, end_date))
+                except Exception:
+                    items = folder.filter(start__gte=start_date)
+            else:
+                items = folder.all()
+
+        return (items)
+
+    def _get_item_by_id(self, item_id, change_key):
+        """
+        Utility method to get MS Exchange item (email message, calendar item,
+        etc.) by combination of item ID and change key directly.
+        """
+
+        item_iter = self.account.fetch(ids=[(item_id, change_key)])
+        return [item for item in item_iter]