Why

In a stitch of irony given my prior articles wrestling with a decent IDLE daemon for use with getmail, I'm faced with a new problem in figuring out the best way to migrate all my existing, locally hosted email to Gmail.

This is evidently not an uncommon problem for people, presumably for largely the same reasons I'm facing: although I like having everything locally on my own server, it only works in places where (1) I live in the same place as the server and (2) where my server won't be double-NAT'd so dynamic DNS can actually reach it.

How

My personal email has been hosted on a Dovecot IMAP server in a Maildir up till now. Our tool of choice for this migration will be the venerable OfflineIMAP utility, available on Debian-ish systems with apt-get install offlineimap.

A Foreword

I tried a lot to get this to work properly in a Maildir -> Gmail configuration, and while it's technically possible I couldn't seem to get the folder creation to play nicely with tags - OfflineIMAP wants to create them with a leading separate ('/' on Gmail) but Gmail itself doesn't recognize that as root tag. There doesn't seem to be anyway around this behavior with name translation or anything.

I suspect you could work around this by uploading to a subdirectory, and then moving everything out of the subdirectory (sub-tag?) on Gmail, but didn't try it.

Configuration file

In your home directory (I did this on my home server since 7gb of email takes a long time to upload over ADSL) you need to create an .offlineimaprc file. For an IMAP -> IMAP transfer, it has a structure something like this:

[general]
accounts = Gmail-wrouesnel

# Gmail max attachment size - you'll get errors otherwise.
maxsize = 25000000
socktimeout = 600

[Account Gmail-wrouesnel]
# Note the ordering - Gmail is the 'local' folder.
remoterepository = Local
localrepository = Gmail

[Repository Local]
type = IMAP
# This ensures we only do a 1-way transfer. If you want to do 2-way then you need a
# rule to exclude the Gmail [All Mail] folder.
readonly = True
remotehost = localhost
remoteuser = <local user>
remotepass = <local password>
ssl = yes
# I use SSL so this is needed - let it throw an error, then copy the hash back.
cert_fingerprint = 60571343279e7f43ee95000762f5fcd54ad24816
sep = .
subscribedonly = no

[Repository Gmail]
type = IMAP
ssl = yes
remotehost = imap.googlemail.com
remoteuser = <gmail user>
remotepass = <gmail password>
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
sep = /
subscribedonly = no

Running

Test the process first with offlineimap --dry-run to check that things are going to turn out roughly how you expect. Then execute offlineimap to start the process. I really recommend doing this in a byobu or screen session, or at least with the nohup utility since a connection drop will cause offlineimap to abort.

Check back on the process once every day or so to check it's still running - OR - write a shell script to re-invoke it until it succeeds (untested so I won't propose any code).

Personal thoughts

This seems to be the most painless way to upload old email to Gmail. In my case, the move is prompted by a real life move where my 24TB server won't be coming with me. I followed up some options for moving my email system, for example to a Docker image for $5 a month for 20gb, but at the end of the day had to face the fact that there was a perfectly capable free-alternative available and it would just be throwing money away. Everything already operates through my Gmail accounts anyway, so it's not like there's a security concern there and when it comes to email you either use GPG or you're doing nothing anyway.

It's worth the observation here that the same process used for the migration can also be used for a local backup, which is a system I will most definitely be using in the future. OfflineIMAP can write Maildir natively, so there's no need to use an IMAP server locally for that, and helpfully solves the "what if Gmail suddenly disappears problem" (more likely from a power failure then anything else, but my email is important to me).