A bit of history for the context
After running a Zimbra mail server in a 500Gb Virtual Machine, for about 4 years, the server started feeling a bit crowded and with #df -h reporting less than 50Gb of space left, it was time to move to a larger machine.
The version I was (and still am) running is the open source version, there are no migration tools available as part of the package, although you can find plenty of tutorials on the web forums about how to rsync stuff between the old and the new server. I was not comfortable with that. For starters, there would be down time involved, but apart from that I would have to rsync between two identical servers, meaning that the new server would still need to be zimbra 8.6 and running on an identical Centos 6 machine.
There was also the fact that some time ago, the server did a very bad shutdown (due to a power failure), and the database had been corrupted, and every so often a problem or two would crop up in the logs. This became very evident when an upgrade to 8.7.1 failed miserably, and the only thing that saved the day was the backup from the previous night ! I was afraid that most likely, an rsync migration would also transfer the problematic data, and that it would be back to haunt me down the road.
Since I was going to have to go through all the aches and pains of a server migration I wanted to end up not only with more space, but also to move to a newer O.S., and a newer release, so rsync was out of the question.
zmmailbox and cli to the rescue
If you have spent any time at all administering a Zimbra mail server, you are definitely familiar with the
zmprov command. You might have used it to reset a forgotten admin password, or to list all the accounts using a forwarding address or to really do any other thing you could do from within the web interface ! You might be a bit less familiar with zmmailbox.
Like zmprov, zmmailbox is also a very powerful command, and can be used to do something as thrivial as creating a mail folder, but it can also export whole mailboxes in “tgz” format, including any folder structure the user might have …. AWESOME !! The only downside is that each mailbox needs to be exported separately, and you cannot use “*” or any wildcards. We are running on Linux however, so we can write bash scripts to do all the heavy lifting for us :).
Preparing the new server
You can follow my guide about how to install Zimbra 8.7.1 on Centos 7
here. If you already know how to do it, just go ahead and install the new server. Basically you will need to install a server with the same settings you used for your old server, but with a different I.P. Address. Remember to setup named, and the zone file accordingly, along with the /etc/hosts file. You do not need to create all the domains inside the server. Just create your main one, we will import the others automatically. Remember that if you intend to install a “letsencrypt” certificate later on, your server name needs to be the same as your http://webmail name. Normally I like to use webmail.domain.com
Switching over to the new server
In order to have virtually zero down time, we will proceed as follows :-
- Set the DNS TTL Entries pertinent to the mail server to the shortest possible time (Ideally this is done a day before to make sure the ttl propagates accordinly)
- Prepare a fully working new server
- Import all existing domains from the old server.
- Import all existing accounts, passwords, distribution lists, and aliases from the old server
- Move all DNS Pointers and firewall port forwards to the new server (or leave the DNS Pointers as they are, and simply swap the servers’ I.P. Addresses old to new, and new to old. (More about this later)
- Make sure that new mail is arriving on the new server.
- Make sure users are able to connect and use the new server.
- Export Mailbox data from the old server, and import it to the new while the new server is running
Before you go any further …
Most of the commands executed during export, and more importantly during import, may take hours. These should be run directly from a console session. If you have absolutely no choice then to run the commands remotely, make sure you use the “screen” command, so that if the connection gets interrupted, you can connect back to your screen, without disrupting any running scripts
The export part
Before we begin the export part, we need to make sure we have enough storage space, which can be accessed from both “old” and “new” servers. The old server did not have any space left but the new server had vitually 1Tb of free space. The way to go, in my case was to remote mount an NFS share from the new server, on the old server and use it as the intermediate storage. Other ways include external usb drives, a network attached storage etc. The choice is really up to you. The steps that follow are generic, and assume the intermediate storage is located at /migration/zimbra. If your path is different you will need to modify the scripts accordingly. To keep things organised, we will create separate folders for all the different migration files. Also make sure the path is readable / writable to Zimbra user. In this case I would recommend
# chmod 777 /migration
# chmod 777 /migration/zimbra
# chown zimbra:zimbra /migration
# chown zimbra:zimbra /migration/zimbra
Step 1. Export all domains
# su - zimbra
# cd /migration/zimbra
# mkdir domains
# cd domains
# zmprov gad | tee -a domains.txt
The command should last only a few seconds, depending on the number of domains on the server. A quick cat domains.txt will show us the available domains
# cat domains.txt
domain1.com
domain2.com
Step 2. Export all accounts
Go back to /migration/zimbra
Create a new directory called accounts and export the admin accounts
# cd /migration/zimbra
# mkdir accounts
# cd accounts
# zmprov gaaa | tee -a admins.txt
Again a quick “cat admins.txt” should list the domain admins, the same ones you see during execution of the “tee – a” command.
Next we export all the users
# zmprov -l gaa | tee -a users.txt
Step 3. Export all account details
Go back to /migration/zimbra
Create a new directory called account_details, and export all account details from Zimbra.
# cd /migration/zimbra
# mkdir account_details
# cd account_details
# for user in `cat ../accounts/users.txt`; do zmprov ga $user | grep -i Name: | tee -a $user.txt ; done
Step 4. Export all account passwords
Go back to /migration/zimbra
Create a new directory called passwords, and export all shadow passwords
# cd /migration/zimbra
# mkdir passwords
# cd passwords
# for user in `cat ../accounts/users.txt`; do zmprov -l ga $user userPassword | grep userPassword: | awk '{ print $2}' | tee -a $user.shadow; done
Step 5. Export all distribution lists
Go back to /migration/zimbra
Create a new directory called distribution_lists, and export all distribution lists
# cd /migration/zimbra
# mkdir distribution_lists
# cd distribution_lists
# zmprov gadl | tee -a distribution_lists.txt
# for list in `cat distributinlist.txt`; do zmprov gdlm $list > $list.txt ;echo "$list"; done
Step 6. Export all aliases
Go back to /migration/zimbra
Create a new directory called aliases and export all mail aliases
# cd /migration/zimbra
# mkdir aliases
# cd aliases
# for user in `cat ../accounts/users.txt`; do zmprov ga $user | grep zimbraMailAlias | awk '{print $2}' | tee -a $user.txt ;echo $i ;done
This process will take some time, depending on the number of accounts. Since most accounts won’t actually have aliases, we will end up with a lot of unnecessary empty text files, which we will delete using “find . -type f -empty | xargs -n1 rm -v”
# find . -type f -empty | xargs -n1 rm -v
So far we have exported everything from our old server, except the mailboxes themselves and any filters the users might have created. We also have the following backup folder structure :-
Step 7. Restore all domains to the new server
The first step in the restoration process starts by restoring all domains. If you have already created the main domain on the new server, you can delete it from /migration/zimbra/domains/domains.txt
The next part assumes that the data we just just exported in the above section is available or copied (in the same directory structure) on the new server. If you have to copy the data, ensure that you issue a chown -R zimbra:zimbra in /migration/zimbra to make sure, user zimbra has full acess to all the files and directories.
Let me also remind you again that any interruption during the the execution of these scripts could result in database corruption, these commands should be run either from a console or via a screen session. You have been warned.
Let’s begin. Login as zimbra user and issue the following commands :-
# cd /migration/zimbra/domains
# for domain in `cat domains.txt `; do zmprov cd $domain zimbraAuthMech zimbra ;echo $domain ;done
A quick visit to https://youserver-ip:7071 should confirm that the domains have been imported successfully.
Step 8. Restore all accounts and passwords
Restoring accounts and passwords requires the use of a small script. Restoring each account will entail creating the account with a temporary password, and restoring the old password in another step.
Go to /migration/zimbra
create a directory called “scripts”, and cd to it, and open your favourite editor to create a file called restore_accounts.sh
# cd /migration/zimbra
# mkdir scripts
# cd scripts
# vim restore_accounts.sh
Paste this script to your file and save it
#!/bin/bash
PASSWDS="../passwords"
ACCOUNT_DETAILS="../account_details"
USERS="../accounts/users.txt"
for i in `cat $USERS`
do
givenName=$(grep givenName: $ACCOUNT_DETAILS/$i.txt | cut -d ":" -f2)
displayName=$(grep displayName: $ACCOUNT_DETAILS/$i.txt | cut -d ":" -f2)
shadowpass=$(cat $PASSWDS/$i.shadow)
zmprov ca $i "TeMpPa55^()" cn "$givenName" displayName "$displayName" givenName "$givenName"
zmprov ma $i userPassword "$shadowpass"
done
Make the script executable chmod 777 restore_accounts.sh, and execute it
# chmod 777 restore_accounts.sh
# ./restore_accounts.sh
The user accounts, and their relative passwords are now being imported into the new server. You might get an error every now and again saying :-
ERROR: account.ACCOUNT_EXISTS (email address already exists: admin@domain.com, at DN: uid=admin,ou=people,dc=domain,dc=com)
This basically means that the account which is currently being imported already exists. This is true for admin, anti spam and anti virus accounts (you may wish to delete these accounts from /migration/zimbra/accounts/users.txt. Also keep in mind that after the script runs, the admin password of the new server will now be the same as the admin password of the old server . The password you have created during the installation will be overwritten (unless you delete admin@domain.comfrom users.txt)
Step 9. Restore distribution lists
Go to /migration/zimbra and execute the following command to re-create the distribution lists
for lists in `cat distribution_lists/distribution_lists.txt`; do zmprov cdl distribution_lists/$lists ; echo "$lists -- done " ; done
Next we need to populate the lists, using a very short bash script
Go to /migration/zimbra/distribution_lists and copy and paste the script below to restore_dist_lists.sh
# cd /migration/zimbra/distribution_lists
# vim restore_dist_lists.sh
paste the following script, save and make it executable, and run it.
#!/bin/bash
for list in `cat distribution_lists.txt`
do
for mbmr in `grep -v '#' ./$list.txt | grep '@'`
do
zmprov adlm $list $mbmr
echo " $mbmr has been added to $list"
done
done
# chmod 777 restore_dist_lists.sh
# ./restore_dist_lists.sh
You will see members being added to the various distribution lists, as the script goes thru the lists. Depending on the number of distribution lists, and the number of members in each list, this script could take a few minutes to finish
Step 10. Restore all aliases
Next we restore the user aliases.
You guessed it go to /migration/zimbra/aliases
create a new script called restore_aliases.sh paste the script below into it, and make it executable
# cd /migration/zimbra/aliases
# vim restore_aliases.sh
#!/bin/bash
echo "Processing User accounts"
for user in `cat ../accounts/users.txt`
do
echo $user
if [ -f "./$user.txt" ]; then
for alias in `grep '@' ./$user.txt`
do
zmprov aaa $user $alias
echo "$user ALIAS $alias - Restored"
done
fi
done
echo "Processing Admin accounts"
for user in `cat ../accounts/admins.txt`
do
echo $user
if [ -f "./$user.txt" ]; then
for alias in `grep '@' ./$user.txt`
do
zmprov aaa $user $alias
echo "$user ALIAS $alias - Restored"
done
fi
done
# chmod 777 restore_aliases.sh
# ./restore_aliases.sh
Step 11. Bring the new server online, and take the old server offline
At this point, we have migrated all the user accounts, their passwords, all the distribution lists, and account aliases. For all intents and purposes, we have a fully working server.
For good measure, let’s restart zimbra services
# zmcontrol restart
Wait for the services to restart, and issue a zmcontrol status
#zmcontrol status
Host domain.com
amavis Running
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Running
memcached Running
mta Running
opendkim Running
proxy Running
service webapp Running
snmp Running
spell Running
stats Running
zimbra webapp Running
zimbraAdmin webapp Running
zimlet webapp Running
zmconfigd Running
Login to the web interface, and check that the number of accounts on the new server is the same as the number of accounts on the old server. There could be a minor discrepancy, if you have used a different domain name when you created the new server, this is due to the spam, galsync, and antivirus accounts which will lead to the number of accounts on the new server to be slightly higher than on the old one.
Once happy with what you see, login using a normal user account, and check to see if you can send emails to your domain, using the test server. Since it is running its own DNS server, and it believes it is the mail server for your domain.com, emails to any account on the server should be delivered immediately.
Check /var/log/zimbra.log for any anomalies.
If all seems well, it is time to take server live.
There are two ways to do this.
- Change all DNS records and port forwarding on your firewall to the new server I.P
- Wait for the DNS records to propagate before continuing with the migration
or
- Disconnect the new server from the network
- Change its I.P. Address to that of the old server
- Modify its /etc/hosts, /etc/named.conf and /var/your_zone_file accordingly
- Restart the zimbra services (this is necessary if you mess around with DNS records)
- Make sure the server is still able to send / receive emails to and form its own domain
- Disconnect the Old mail server from the Network
- Connect the new server to the network (which will now be live)
- Change the IP address of the Old server to an unused I.P. Address
- Modify its /etc/hosts, /etc/named.conf and /var/your_zone_file accordingly
- Restart zimbra services
- Connect it back to the network, to continue the migration of the mailbox data
The second method, seems more complicated, but it will guarantee ZERO downtime. Believe it or not, it could be the fastest method to switch over to the new server.
Once the switch over is complete, any users logging in to the server using IMAP or Webmail will be greeted with an empty mailbox, but any new emails arriving to the domain, will be delivered to the new server.
In the final few steps we will export the mailbox data from the old server, and import it to the new server.
Step 12. Migrating mailbox data
The next process will be the most time consuming of all the processes. Depending on the amount of data, exporting the data could take as long as a day if not more. Data import will take just as long.
Now that our users can login to the new server and send and receive emails, the pressure is a bit lower. Also if a very important email needs to be accessed, this can be done via the web interface of the old server, using https://old-server-i.p.
Let’s start the data export.
Login to the old server, via console or via a “Screen” session. I am assuming that the backup repository we were using in /migration/zimbra is mounted again on the “old” server using the same path
Login as Zimbra user and goto /migration/zimbra
# su - zimbra
# cd /migration/zimbra
# mkdir mailbox_data
# cd mailbox_data
# for user in `cat ../accounts/users.txt`; do echo "Exporting mailbox $user" ; zmmailbox -z -m $user getRestURL '/?fmt=tgz' > ./$user.tgz ; done
You will start seeing mailboxes being exported. This is going to take a while, so you might go endeavor in other projects.
In the event that any mailboxes report errors during export, make a note of the problematic accounts, and try to re export them later, by creating a text file :
/migration/zimbra/accounts/problematic_accounts.txt
and running the following command. The format of the file needs to be exactly like the format inside /migration/zimbra/accounts/users.txt. Use it as reference.
# for user in `cat ../accounts/problematic_accounts.txt`; do echo "Exporting mailbox $user" ; zmmailbox -z -m $user getRestURL '/?fmt=tgz' > ./$user.tgz ; done
Once the mailbox migration complete, let’s also export any user filter settings. We need a bash script for this.
Create a new directory called filters, create a file called export_filters.sh and paste the script below into it.
# mkdir /migration/zimbra/filters
# cd /migration/zimbra/filters
# vim export_filters.sh
# chmod 777 export_filters.sh
#!/bin/bash
mkdir tmp
set -x
clear
for user in `cat ../accounts/users.txt`;
do
filter=`zmprov ga $user zimbraMailSieveScript > ./tmp/$user`
sed -i -e "1d" ./tmp/$user
sed 's/zimbraMailSieveScript: //g' ./tmp/$user > ./$user;
rm ./tmp/$user
echo "Export filter for $user"
done
\rm -rf tmp
Mount, move or copy /accounts/zmigrate to the new server, and go to the /migration/zimbra/mailbox_data directory
# cd /migration/zimbra/mailbox_data
# for mailbox in `cat ../accounts/users.txt`; do zmmailbox -z -m $mailbox postRestURL "/?fmt=tgz&resolve=skip" ./$mailbox.tgz ; echo "$mailbox - done "; done
This process will also take quite a while to complete, the users may use the server freely during the import. When their turn arrives, they will start seeing their mailbox starting to get populated with the emails they left behind on the old server.
If you get any errors, you can re-export, and re-import the problematic accounts by manipulating /migration/zimbra/accounts/users.txt accordingly and re-running the export command. If you get any “broken pipe” type errors it probably means that the tar archive got corrupted during the transfer for some reason or other, and is essentially incomplete.
Finally let’s import the user filters. Go to /migration/zimbra/filters, and create a file called import_filters.sh. Paste the script below into it, save and make it executable.
# cd /migration/zimbra/filters
# vim import_filters.sh
# chmod 777 import_filters.sh
#!/bin/bash
for filter in ./*
do
if [ "$filter" == "./import_filters.sh" ] ; then
continue;
fi
if [ "$filter" == "./export_filters.sh" ] ; then
continue;
fi
if [ "$filter" == "./tmp" ] ; then
continue;
fi
Filter_String=`cat "$filter"`
Account=$filter
zmprov ma $(echo $filter | grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b") zimbraMailSieveScript '$Filter_String'
echo "Process filter $Account"
done
echo "All filter has been import successfully"
Conclusion
That’s it, your users should be happy using your newly migrated server. The only thing they will find missing are their account signatures. To date I have not yet found an automated reliable way to copy them automatically. What I did in this case was, made the old server available on a temporary sub-domain (old-webmail.domain.com), so that the users could login in to the old server, and essentially copy and paste their signature to the new server. If you feel extremely generous, you could also do it via the admin interface, but you would have to do the process account by account manually, so it could take some time.
