diff --git a/.github/actions/spelling/expect.txt b/.github/actions/spelling/expect.txt index de156505f..00432d98c 100644 --- a/.github/actions/spelling/expect.txt +++ b/.github/actions/spelling/expect.txt @@ -548,6 +548,7 @@ mkhomedir mmap mmencode MMIME +mnt modseq mountpoint mozilla diff --git a/source/admin_manual/importing_mailboxes.rst b/source/admin_manual/importing_mailboxes.rst index ae025b537..d2363406e 100644 --- a/source/admin_manual/importing_mailboxes.rst +++ b/source/admin_manual/importing_mailboxes.rst @@ -55,3 +55,81 @@ If you have only read privileges, you can try using in-memory indexes. doveadm import -u destuser maildir:/opt/backup/destuser/Maildir:INDEX=MEMORY "" ALL + +Merging Storages +---------------- + +In some disaster recovery cases you may end up having mails for the same user +in two different locations, and need to merge them. For example the storage +goes down and fixing it takes a long time, so during the fixing you can let the +users access their emails as an empty account, which can receive new mails. +Later on you can use ``doveadm import`` to merge the mailboxes. + +Note that there is no way to make this solution perfect: + * IMAP clients that have cached mails locally will delete their local caches + and have to re-download mails later on. + * POP3 clients that leave mails on server will notice all the mails are gone, + and delete their local UIDL caches. When old mails come back, they're + re-downloaded as new emails (duplicates). + +Also, there are 3 alternative ways of how mails can be imported into mailboxes: + * Old recovered mails are imported on top of the newly received mails. The + downside here is that mails may now be sorted in a weird order. If the IMAP + client shows the mails in the saved order, the new received emails show up + as oldest emails. Although this may not be an issue, since many IMAP clients + sort the mails by either Date: header or the received timestamp (IMAP + INTERNALDATE). + * New mails are imported on top of the old recovered mails. This avoids the + sorting problems, so it's likely the preferred method. The downside here is + that the IMAP/POP3 clients will have to re-download also the newly delivered + emails, as well as the old ones. Another issue with this is that some IMAP + clients might not show the old recovered mails without manually rebuilding + local caches, because the mails become inserted to the beginning of the + folders, which isn't allowed by the IMAP protocol. + + * Another thing to keep mind here is that IMAP clients shouldn't see + IMAP UIDs pointing to different emails before/after the merge. Otherwise + their local cache could point to a different email, which could even + cause the user to delete wrong messages. This shouldn't be an issue as + long as new mail deliveries and all user access is disabled during the + merging. The old mails have the old UIDs, and newly delivered mails + would all have higher UIDs (because the UIDNEXT value is not shrunk + during index rebuild that clears out the mailbox). + + * New recovered mails are imported under a separate ``Recovered/`` folder, + i.e. there will be ``Recovered/INBOX``, ``Recovered/Sent``, etc. The user + will need to manually merge the folders. The upside here is that POP3 + clients won't re-download any mails as duplicates, but otherwise it's not + much different from the 1st case. + +Example for the 2nd case where mail storage broke down, but a separate index +storage is ok, and index storage supports snapshots: + + * Snapshot the current index volume at the time of breakage + * Make sure ``mail_location`` setting has ``ITERINDEX`` feature enabled, so + folder listing is done using the index volume rather than the mail volume. + * Mount a new empty mail volume + + * The first time IMAP/POP3 client attempts to access an existing mail, + Dovecot rebuilds the indexes for the folder. This makes the folder look + empty. The folder structure is preserved, as long as ``ITERINDEX`` + setting is used. + + * Once the original mail volume is recovered, first disable all user access + and all new mail deliveries. + * Snapshot again the index volume + * Mount the old mail volume to the original mountpoint + * Replace the index volume with the first created snapshot. Now the storage + looks exactly like it was at the time of breakage. + * Mount the new mail volume to some temporary mountpoint + * Mount the second index snapshot to some temporary mountpoint + * Use ``doveadm import`` to recover new mails: + + .. code-block:: + + doveadm import -u user@example.com sdbox:/mnt/temp-mail-storage/user:INDEX=/mnt/temp-index-storage/user:CONTROL=/mnt/temp-index-storage/user:ITERINDEX "" all + + If your normal ``mail_location`` has other settings, you may also want to + specify them using some temporary locations. For example + ``VOLATILEDIR=/tmp/doveadm-import/user:LISTINDEX=/tmp/doveadm-import/user/dovecot.list.index`` + and after importing delete the directories.