Zimbra Upgrade: Red Hat 5 to Red Hat 6

We have been running Zimbra at my current employer using Red Hat 5. With the coming version changes from v7 to v8, due to some dependency changes, Red Hat 5 was no longer supported and thus required us to move to Red Hat 6 along the way.

Now, Zimbra does have SOME documentation for this process but a lot of it is outdated and/or requires a lot of testing and command changes between versions. I understand the need to move to RH6 but Zimbra Support could have done a much better job at clearly documenting this process for the users. Also quite frustrating is each version had various little bugs/issues so depending on which version you are coming from will depend on what “challenges” you encounter. The Zimbra installer should also be improved to compensate for more of these issues as well.

For any upgrade of this size its obviously recommended to spin up a test instance of your production environment and do a bunch of testing. I encountered a few interesting challenges myself that delayed our upgrade and flat out caused some derailments but in the age of VMWare snapshots and Veeam sandboxes all of this was minimal effort required.

With that said, let’s look at the overall process. I am primarily looking at a “single” server deployment, a “multi-server” install basically follows the same process with some exceptions since everything is spread out among various servers. This post will look mainly at the single server situation. Please make sure you take snapshots or VM backups BEFORE making any changes!

Upgrade existing instance to latest v7 GA release
This is important especially in version 7 as many bugs/issues were fixed throughout the releases to deal with things like mailbox moves, migration, etc. If you are coming from older releases you really need to make sure you check the documentation as you could need to do “step upgrades” to certain versions which is important!

Spin up a new Red Hat 6 VM
I won’t cover how to do this, you should know already. Strive to set this up per best practice: few default services as possible, no conflicting Zimbra services installed, latest updates applied, etc. Once you have this VM to a “standard” deployment state, install the same version of Zimbra that you are running on your original instance. During the install you need to make sure you point this new server at your original LDAP Master (which in a single server environment should me your original server). See this wiki under section “Configuring The New Server” for more information:  Also don’t forget to install all the roles you have on your original server as they will need to replace them!

At this point you should have 2 Zimbra servers and should see this reflected in the Servers section of the Admin GUI. At this point I would move all system/user mailboxes over to the new mailbox server. Initially I received random errors in the web GUI so I switched to everything via the CLI, apparently my version of Zimbra had some bugs. The zmmboxmove is the newest command for doing this, check out options here. Once the mailbox moves are done you can begin migrating all services to the new server and discontinuing the old.

Configure LDAP Replication
You need to setup LDAP replication on your original master so it is replicating changes to your new RH6 VM. See this wiki for more information. Short way is just to run:
/opt/zimbra/libexec/zmldapenablereplica
Allow some time for the replications to happen and check logs to confirm!

Promote LDAP Replica to Master
This process will result in some downtime and has a few steps.
See this wiki for the process

Move additional roles to new server
This would include things like MTA, etc. The primary services needing to be moved are MTA and LDAP though!

Perform cleanup on old server and remove
At this point you should have a new server running with your mailboxes, MTA and LDAP Master running without issues. Once you’ve reached this point, do the following:
– Check for any mailboxes living on old host: zmprov searchAccounts “(zimbraMailHost=mail1.domain.com)”
– 
Once you are sure everything of value is off the server you can remove it from Zimbra with the following command: zmprov deleteServer mail1.domain.com

Proceed with Upgrade
Once you have completely removed the Red Hat 5 VM you can now upgrade your new RH6 VM to v8.0.7 (plus Patch 1!)

And there it is, not too bad, although there is a bit of downtime required. The above is not an exhaustive list of what to do, which is why I’ve included wiki links that get more specific. I have made a lot of assumptions that you know Linux, you know a bit about Zimbra and how the backend services work and the other requirements that exist like ensuring time sync on the host, SSL certs., etc. My goal was to help provide a bit better documentation than what Zimbra provides today, as you can see most of the process is split between various older version wiki articles, etc. You can also see this wiki for more about this RH5 to RH6 process.

With that said, let’s look at some of the “gotchas” I encountered and some of the issues I have seen:

– During upgrade noticed our /var/log files related to Zimbra sky-rocketed in size causing root (/) to grow to 100% usage in about 10 minutes, thus killing upgrade due to “no space” available. After some research, it appears we had a “dead” reference to our original Zimbra server in rsyslog (/etc/rsyslog.conf) that would constantly fill the log with errors, commenting/removing this old reference resolved this issue. There is a somewhat related bug entry here . Also a similar issue on the forums but related to Ubuntu. If you remove this or comment out the dead entries and restart things should go well or you can run /opt/zimbra/libexec/zmsyslogsetup to regenerate.

– Please make sure to read the “After You Upgrade” section of the 8.0.7 Release Notes:
http://files.zimbra.com/website/docs/8.0/ZCS_807_NE_ReleaseNotes_UpgradeInst.pdf
See Page 37. You need to run zmldapupgrade -b 66387  This will change the
zimbraAllowFromAddress values to grants since this can’t be done for internals anymore in v8 and beyond. If you rerun the command a 2nd time it should report no changes made so you know it was successful.

– After the upgrade went to login with a test user and received a general error, after further inspection any user using LDAP authentication was coming back with errors. Luckily I have a non-LDAP admin. user, upon logging in to admin GUI could see the install process for some reason put an additional space between a portion of my LDAP filter causing it to be invalid, after removing the space all was well. In case this happens to you and you don’t have a non-LDAP admin. user you can create one quick from the server using command: zmprov ca adminbackup@domain.com test123 zimbraIsAdminAccount TRUE

Overall the process went smooth, Zimbra could always use some improvements in the process, especially compensating for bugs/issues between versions which can cause some frustration and/or make wiki articles invalid. After the Telligent purchase it would be nice to see a renewed commitment to going into wiki.zimbra.com and updating articles, etc. Would also like to see more of the “stuff that you only can do in the CLI” make its way to the GUI, things like LDAP roles/promotion, etc. Overall the new “serenity” interface is nice and offers some great new features/enhancements.

Advertisements

Author: Travis Kensil

Director of IT. Husband and father. Michigan beachbum.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s