Veeam Office365 Backup: Mailbox Processing Errors

Have been doing some testing with Veeam’s new Office365 Backup solution, so far pretty impressed. Did run into an interesting issue out of the box, we had some random mailbox processing errors such as:
– Failed to synchronize item changes. The operation has timed out.
– Async batch export failed with timeout.
– Mail item data export failed. There is an error in XML document.

Originally I just attributed this to “issues in the cloud” with EWS since Veeam uses this but after a day or so it became clear something else was happening here; as a safety net did some packet tracing in our firewall cluster to rule out potential security services causing the timeout. I did some additional testing and also spoke with Veeam support, their recommendation was to add the following line immediately after the <Archiver> tag in config.xml located in: C:\ProgramData\Veeam\Backup365

<Source WorkerThreads=”4″ BatchSize=”10″ BatchPart=”10″ BatchTimeout=”900″ BatchMaxItemSize=”5″ />

After adding this line and re-running the job it seems to have resolved the issue as all of our mailboxes are now backing up! Currently this issue was happening on build 1.0.0.912

Advertisements

Veeam Backup: Part One

This is part one of a multi-part post on our Veeam setup, we recently deployed Veeam and have been using in production for close to a week now. Below are some initial notes/thoughts on our Veeam setup. Our Environment/SetupWe have a complete virtu…

This is part one of a multi-part post on our Veeam setup, we recently deployed Veeam and have been using in production for close to a week now. Below are some initial notes/thoughts on our Veeam setup. 

Our Environment/Setup
We have a complete virtualized environment running vSphere 4.1. We have (3) ESX 4.1 hosts connected to (2) Equallogic PS100 SANs. The SANs have around 4TBs total in their storage pool. We are running Veeam B&R on a physical 2008 host; we prefer to have our backup/restore solutions separated from the infrastructure that they are backing up. Our Veeam box is connected to around 7TB of DAS storage from an HP MSA60 (populated with 2TBx6 drives in RAID6). We are using SyncBackPro software (http://www.2brightsparks.com/syncback/sbpro.html) to copy our Veeam files to our offsite location every evening over our MPLS circuit. Our offsite location has around 24TB of storage onsite so we can archive 1+ years worth of backups.
See the below diagram:

Backupoverview

Veeam Setup
We have a very simple job setup, we only have (4) Veeam Jobs:

  • DWM (Daily/Weekly/Monthly): This is our main backup job for all our VMs, has a mix of Linux/Windows VMs. For Windows boxes, ensure you have VSS enabled and working, enable Application-aware processing and ensure you have valid credentials, once that is done things are pretty much setup on the Windows side (minus applications that are NOT VSS-aware, for those cases you will need to write some pre/post scripts via VMWare Tools to address your application-specific procedures to ensure it is in a “clean” state). Linux, due to its lack of VSS, needs a bit more configuration, mostly in the way of writing some pre/post scripts to handle various applications you may have. On future blog posts I will post some of the scripts/techniques we are using, one simple example would be for MySQL, a simple stop/start of the service would be a great way to achieve this, if you can afford the small downtime this creates.
  • Onetime: This job handles our VMs that don’t have frequently changing data and that we only run monthly.
  • VirtualCenter: This job backs up our VirtualCenter server. You need to locate the host your VC VM is running on and add that host to the Veeam console (not the VC server!) for a successful backup. The downside to this is that you will need to update this job anytime this VM changes hosts, you may want to tell DRS to leave this VM alone.
  • ProductionApp: This job backs up our primary business application every hour, so far we have noticed no performance issues with this frequency. I will write a future blog post about this job as this VM runs an IBM Informix database that has some unique requirements to get in a consistent state.

At this time we are using Network mode to backup our VMs, we currently have the capability to do SAN-level backups but chose not to due to the risks of VMFS corruption. Veeam by default disables automount in Windows but we are not comfortable with that being our only safety net, currently Equallogic does not support granting read-only permissions to a particular host, you can make the entire volume read only but this is of little use. Should automount become enabled via update or something else, there is possibility of damaging our VMFS volumes. We will wait until Equallogic adds this feature then give our Veeam host read-only to each volumes then make use of direct SAN backups. We currently have a RFE with Dell to add this feature but who knows when/if this will happen! Spoke with Dell this week who confirmed for me that this feature has been coded and scheduled to be released in next firmware update!

All our jobs are set to use Incremental mode with Synethic fulls on Saturday and one Full backup a month on 1st Saturday of the month.

The first initial backup for our environment (around 1.5TB) took approximately 18hours, 30minutes. Incrementals after that less than 2 hours!
Veeam compresses our initial backup of 1.5TB down to around 800GB which is almost half, quite impressed! Incrementals are usually around 50-200MB depending on quantity of data changed.

We are using BEST compression and WAN TARGET settings to achieve these results.

Our Data Center has an HP MSA60 with around 7TB of DAS storage, our offsite location has 24TB of storage on an HP MSA2012i.

Watch this blog for future posts on our Veeam setup.

Odyssey Access Client Backup

We use the Juniper Odyssey Access Client for our 802.1x supplicant because of how well it integrates and works with Novell for SSO. One of the issues we’ve had is how to backup/restore our user’s various networks/profiles they create, currently Ju…

We use the Juniper Odyssey Access Client for our 802.1x supplicant because of how well it integrates and works with Novell for SSO. One of the issues we’ve had is how to backup/restore our user’s various networks/profiles they create, currently Juniper does not have a utility or process to do this, so we created one in-house that has been working very well for us. Basically what we want is a backup of all the user’s various networks they have configured without any of our own in-house networks being included. Luckily Juniper has a script writer that can perform various useful tasks like adding/removing networks. Here is our process:

Write an OAC script

  • Odyssey Access Client Administrator—Script Composer
  • We use the Remove Networks option to remove all of our corporate SSIDs, leaving us just the user created ones.
  • Once done we generate the script.

Once our script is written we can move on to the “logic” piece of the puzzle. Now let’s walk through each piece of our process:

Step One: Backup Current Data
First we need to backup the user’s current networks and profiles to ensure at the end the machine can again have network access. For this process we run two commands:

reg export “HKEY_CURRENT_USERSoftwareFunk Software, Inc.Odysseyclientnetworks” C:LiveOACNet.reg
reg export “HKEY_CURRENT_USERSoftwareFunk Software, Inc.Odysseyclientprofiles” C:LiveOACProf.reg

Basically these two commands are doing a reg “dump” of the Odyssey networks and profiles to the local disk.

Step Two: Remove our various SSIDs
Once this script runs, the client will temporarily lose network (wireless) connectivity, this only lasts approximately 10 seconds which is a very reasonable time frame.
To ensure we can recover the user’s networks after each reimage cycle we need to ensure our profiles don’t carry over or exist on the machine, this is due to differences in passwords per year so old data would overwrite current data.For this process we call upon odClientAdministrator.exe to silently run the script we wrote above.

By default odClientAdministrator is located in: C:Program FilesJuniper NetworksOdyssey Access Client

We use the command: odClientAdministrator.exe /I=C:ScriptName.odyClientScript /S

This particular command is well documented in Juniper’s KB at: http://kb.juniper.net/InfoCenter/index?page=content&id=KB10744&actp=s…

Step Three: Backup User’s Data Only
Now we are ready to backup the user’s networks that they have created/added in Odyssey.To accomplish this we run this command:

reg export “HKEY_CURRENT_USERSoftwareFunk Software, Inc.Odysseyclientnetworks” C:OACNetSettings.reg

Again this basically does a reg “dump” of the networks and puts them on the local disk. We can then copy this file off the user’s machine to their network drive to ensure we have a working backup of their data.

Step Four: Reimport Production Data
Now we just reimport the backup we took from step one to ensure the client’s connectivity is restored and they have all their SSIDs back. 

regedit /s C:LiveOACNet.reg
regedit /s C:LiveOACProf.reg 

We actually run this whole process in the context of a batch file, which you can see below:

cls
@echo off
title OAC Backup
echo ************************************************
echo *           OAC Backup Process               *
echo *              Version 1.0                           *
echo *                                                         *
echo ************************************************

echo Backup Live Data
reg export “HKEY_CURRENT_USERSoftwareFunk Software, Inc.Odysseyclientnetworks” C:LiveOACNet.reg
reg export “HKEY_CURRENT_USERSoftwareFunk Software, Inc.Odysseyclientprofiles” C:LiveOACProf.reg

echo Remove APS Wifi Data
cd “C:Program FilesJuniper NetworksOdyssey Access Client”
odClientAdministrator.exe /I=C:YourScript.odyClientScript /S

echo Backup User’s Wireless Data
reg export “HKEY_CURRENT_USERSoftwareFunk Software, Inc.Odysseyclientnetworks” C:OACNetSettings.reg

echo Reimport Live Data
regedit /s C:LiveOACNet.reg
regedit /s C:LiveOACProf.reg

 

This script is not ideal, losing network connectivity for 10 seconds is definitely not preferred but is currently the best way we have to backup and restore Odyssey’s settings for our users. We do have a request for enhancement in with Juniper but who knows how long that process will take.

Goodbye Mozy: New Offsite Backup Solution

For the past 2 years I’ve been a loyal Mozy customer. I’ve backed up about 50 gigs worth of personal pictures, videos, documents, etc. In previous jobs I’ve used it as an “off-site” backup solution when the budget was tight. During “side gigs” wit…

Mozy-logo-440

For the past 2 years I’ve been a loyal Mozy customer. I’ve backed up about 50 gigs worth of personal pictures, videos, documents, etc. In previous jobs I’ve used it as an “off-site” backup solution when the budget was tight. During “side gigs” with companies or individuals I’ve recommended it to backup their data. I’ve always considered Mozy to be a reliable, secure option, EMC’s purchase of Mozy a couple years ago just validated those ideas. The price has always been a HUGE attraction to their service as well.

Today I opened my email to find the message from Mozy that pricing would be going up. At first I wasn’t too concerned but started doing the math and got a bit nervous, just my personal data would be a decent level of increase and that’s not including those I know using the service with 100s of gigs worth of content. Plus, they did email me, but I found out about the change via LifeHacker and also realized they were not “grandfathering” anyone in, ouch! I don’t mind a company changing the plan as long as I am notified in advance, the reasoning is solid and the new prices are reasonable.

SIDE RANT: If your primary business is storage and backup, why are you complaining about increase of use? We live in a digital world, files are getting bigger all the time, if you can’t hang with it, then get out of the storage business. On the Mozy Blog they admit the “concern” that things are increasing, sounds like they want their profit margins to increase as well!

With that said, I have until March 1 and then I’ll be canceling and moving to a service that understands backup storage needs to be reliable AND AFFORDABLE! With many options out there today, many from reputable companies, Mozy will have an uphill battle to convince others to stay while others are offering a better product at a similar or cheaper price.

So what are the options, for me I have narrowed in on two that seem to offer a better feature set than Mozy for a decent price. My criteria is as follows:

– Affordable (need to stay less than $20 per month total, including service fee and any/all bandwidth/storage costs plus include all my laptops/desktops)

– Reliable: Some companies such as Carbonite have had issues in the past, want to make sure they have a clean track record of uptime.

– Solid Backing: Need to find a reputable company that is providing the service, nice thing about Mozy was the backing of EMC.

– Features: Need to have Windows, Linux and mobile device support. Would be nice to have a “virtual” drive that I could use as a primary storage medium versus saving locally. Some kind of sharing access between all devices would be nice too.

The Options Thus Far:

JungleDisk
http://www.jungledisk.com/

Pro: Very flexible pricing model, can use Amazon or Rackspace storage cloud as backend. “Virtual drive” allows things to be saved to the “sharing network” versus saved locally. High-level encryption of data. JungleDisk is a subsidary of Rackspace which lends it some credibility.

Con: They have a mobile app. for iPhone but none for my Android.

SpiderOak
http://www.spideroak.com/

Pro: Cheap storage. Works across all platforms including my Droid 2. Sharing between all devices. Has de-duplication for data which is nice.

Con: Not too familiar with the company behind the product.

In the coming days I’ll begin testing both products to see which will work best. I like the SpiderOak product the best because of the Android app.