VeeamOn 2017: Recap

I just got back from my first VeeamOn event in New Orleans, LA. Was by far one of the best events I have ever attended from a technical AND personal perspective. I think the volume of people and partners in attendance (3000+) speak to how successful Veeam has been at not only creating a great product but also a great ecosystem. This event also unveiled a LOT of new features/functionality that were direct results of customer/partner feedback to Veeam. In my own experience, I have consistently seen suggested features make their way into the products so it’s clear Veeam is listening. Also as a customer the level of innovation/R&D is impressive, Veeam is constantly pushing itself and consequently the backup industry to move forward fast, some of the features presented are so impressive that literally no one in backup world is doing yet.

Some of the existing features discussed during Gostev’s v10 (Coming Soon ) presentation:

  • New Veeam Agents (Lin/Win). This is very exciting as this gives cloud/hybrid/on-prem solutions for any environment, running anywhere. We have been Veeam customers for quite a long time and I can still recall when Veeam was smaller them saying ‘we will never do agents’, glad to see they wisely reversed course on that attitude.
  • Veeam CDP (Continuous Data Protection). This one is a biggie, with the addition by VMware of the IO Filtering APIs Veeam can now grab data continuously WITHOUT snapshots! This will be huge for many of our mission critical apps that don’t respond well to snapshotting especially for the quantity of restore points we want per app.
  • Veeam and N2WS Partnership allowing backup of Amazon EC2 instances. This will help address a large gap in most public cloud providers: Backup of entire VM/instances not just in-VM agents!
  • Scale-Out Repository With Archive Tier. This was very slick, they have now added an archive tier that can go to older/slower storage and/or also out to public cloud providers for long-term retention such as AWS S3 or Glacier, Azure, etc. The best part, this is all native in the console and couple clicks activates it! I think this feature will remove excuses for anyone not to have affordable long-term retention of data.
  • NAS SMB/NFS Backups. This is another slick solution allowing Veeam to directly backup SMB/NFS shares, preserving perms/file versions, etc. Best part is it can restore to original location OR other locations which has some secondary uses for file migrations as well!
  • Veeam Cloud Connect. There were a number of vendors now providing the Veeam Cloud Connect services. Many of them have dramatically expanded the offering to allow emergency environments during a disaster, secured replica seeding and other key features. For many business’ this could allow them to get rid of their DR sites and utilize these vendors saving time/money.

Of course there are MANY, MANY more items released or coming, for a better listing see here: https://www.veeam.com/blog

The conference had a good variety of sessions to attend in addition to the General Sessions. A few of my favorites….

Ransomware Session: It is clear by how full this event was, that everyone is interested in what we can do to protect. Lot of good best practices here, some of the top:

  • Filter/limit untrusted sites, Skype file exchanges, Office exe/vbs/plugins execution.
  • Don’t rely fully on AntiVirus to stop all threats (defense in layers)
  • Use different credentials for Veeam environment
  • Use SureBackup to ensure backups work if needed
  • VLAN Segmentation to keep client devices separate from servers, helps reduce spread of issues.
  • Airgap backups, with the ability of some of these wares to easily spread its important to have copies that are not network-accessible. This is where tape, offsite Cloud Connect or even Archive to AWS/Azure would be helpful.
  • User Education to ensure they aren’t falling for attacks or providing access if threats are able to bypass security layers.
  • Keep Veeam And Backup Stores separate via different credentials, VLANs, etc.
  • Have good monitoring in place to detect anomalies.
  • Use “least privilege” model for permissions.
  • One not covered at the event but one I think is super helpful is Software Restriction Policies (a native Group Policy option) running in whitelist mode. This will basically immediately block all threats from running unless they are whitelisted or trusted Windows services. In my experience this helps stop a LOT of these ransomware attacks as they would be prevented from executing in the first place. It does take some time to setup/test the policies but it pays dividends later.

Microsoft Office365 Whats Next

This was a MUST attend for me as our use of Office365 continues to skyrocket and being able to protect data/assets is critical. Microsoft does a poor if non-existent job of this currently so the need for external backup is critical. There are some exciting things coming in the Office365 suite, in 2 separate phases. Version 1.5 which hopefully is coming in June/Aug of this year will breakout the roles (proxy/server) to allow flexibility and scalability for larger environments, add PowerShell/Restful API support and other improvements. Version 2.0 which hopefully will be before year end is the most exciting as it will include backup for Sharepoint/Onedrive. THIS IS THE CRITICAL FEATURE WE NEED MOST! If you are wondering does Veeam listen to customers, answer is a YES, check out the forums, lots of folks wanting this and it is on the list: https://forums.veeam.com/veeam-backup-for-microsoft-office-365-f47/

These improvements will make it easy to recommend we purchase this product once our 1 year free license expires.

Other Thoughts/Comments…

  • One thing I really appreciate about Veeam is how open and transparent it is. A large number of these sessions were hosted not by sales people but by Veeam engineers or managers that actually work the product so deep technical was possible as well as honest conversations about roadmap, why did you do it that way, etc. Being able to ask the Product Managers direct questions is really cool and helps when roadmap planning internally.
  • Veeam also allows customers to get up and present, saw some cool sessions from the University of British Columbia and others that were deep technical and not just sales events. Also covered some good best practices and thoughts/comments about what works and what doesn’t.
  • I saw a lot of innovation/R&D work on display at this event. Many features were direct out of customer conversations or forum posts.
  • Networking potential at this event was HUGE! There were literally people from every corner of the planet, I met some folks from Canada, Germany, Russia, Mexico and China.
  • Veeam had a strong partner ecosystem on display as well, see the floor Expo below, tons of vendors offering complementary solutions to Veeam and all the big names: Dell, HPE, Cisco, etc.
  • Tons of great swag/giveaways at this event. Even the basic conference bag that everyone got was good, combo backpack/carry bag and good build, spots for water and snacks.
  • New Orleans is a great city, lots of cool stuff to see/experience and a great location for VeeamOn! I didn’t have a ton of time or energy to do too much exploring but did look around, some pics:


  • Veeam knows how to throw a conference and a party! I was surprised by the quality and quantity of the food and parties. Almost every night there was some kind of event or thing to do, got to see a lot of NOLA places in a short period of time!





Overall was a very good conference, learned a lot, met/networked with a lot of great people and got a good feel for the Veeam 2017 roadmap, was most excited to see it aligned with the directions we need for our use cases! Next year’s VeeamOn will be in Chicago which is even closer for me so hoping, schedule allowing, I will be able to attend again.

Advertisements

Veeam Office365 Wishlist

Have been using Veeam’s Office365 backup solution now for 30 days, have to say I am very impressed with it. I do however have a couple of items that would be “nice to haves” to improve the product.

  • Support for backing up Sharepoint and OneDrive document libraries. This is a big one, especially for environments that have gone “all in” with Office365. Microsoft’s own backup/DR abilities are pretty weak, especially for the tenant side. This is heavily requested on Veeam forums:
    https://forums.veeam.com/veeam-backup-for-microsoft-office-365-f47/feature-request-backup-of-office-365-sharepoint-t36949-90.html
    Some must-haves…..

    • Ability to restore individual items, lists, folders, sites, etc.
    • Preserve permission/inheritance of restored content
    • Export data back to original location OR network share/folder locally OR perform rename function something such as restoreddoc_DATE.docx and place in original location
    • From a priority perspective it would be nice to see Sharepoint done first as this is the “business” side where most critical documents live, whereas OneDrive is more a personal repo.
  • Offer more folder exclusion options such as Sent Items. Also being able to define a set of custom folders to exclude would be helpful.
  • Enhanced retention policies similar to the options available in full Veeam BR.
    https://forums.veeam.com/veeam-backup-for-microsoft-office-365-f47/need-to-modify-retention-policy-t41359.html
  • Offer more options or those similar to Veeam BR in terms of repository storage options.
  • Personally I would like to see the separate console go away and these features be integrated natively into the Veeam Backup And Recovery console. In the future as more application-type options are added I don’t want to have separate consoles to access them, I want them all from within Veeam Backup console.

Veeam Office365 Backup: Mailbox Processing Errors

Have been doing some testing with Veeam’s new Office365 Backup solution, so far pretty impressed. Did run into an interesting issue out of the box, we had some random mailbox processing errors such as:
– Failed to synchronize item changes. The operation has timed out.
– Async batch export failed with timeout.
– Mail item data export failed. There is an error in XML document.

Originally I just attributed this to “issues in the cloud” with EWS since Veeam uses this but after a day or so it became clear something else was happening here; as a safety net did some packet tracing in our firewall cluster to rule out potential security services causing the timeout. I did some additional testing and also spoke with Veeam support, their recommendation was to add the following line immediately after the <Archiver> tag in config.xml located in: C:\ProgramData\Veeam\Backup365

<Source WorkerThreads=”4″ BatchSize=”10″ BatchPart=”10″ BatchTimeout=”900″ BatchMaxItemSize=”5″ />

After adding this line and re-running the job it seems to have resolved the issue as all of our mailboxes are now backing up! Currently this issue was happening on build 1.0.0.912

HP StoreOnce And Veeam

Our current Veeam environment is virtualized (so we can take advantage of Hot-Add) and currently our Veeam data was being stored on legacy Equallogic arrays that needed to come out of service. We began looking at options for replacement, of course there are a variety of options. Veeam themselves seems to recommend the commodity server route, there are tons of inexpensive NAS solutions and of course dedup appliances. Since we are mostly an HP shop the first stop was their solution which in this case is the StoreOnce platform. Whats great about it is that it really scales and gives you flexibility, you can buy into a software only or hardware solution and go from there. And with the recent integrations with Veeam it was a no-brainer.

HP charges a pretty good markup for the hardware StoreOnce platform; especially after talking with HP engineers its basically a G8/G9 server platform. We chose to just go out and purchase a commodity G8 server, load it with drives and memory, throw ESXi on it and run the StoreOnce VSA. Comes as a prepackaged appliance, setup in a matter of minutes, throw a large volume of storage at it, during boot it will detect and auto provision for use. Just need to go in and add a NAS share and thats about it. Really is quite simple and with the cost of the software only option VERY affordable even in the small to mid market and VERY scalable. Those who use HP’s tape loader platform will recognize the interface, not sure if thats good or bad. Even though there is the Catalyst integration we are not yet using that as there was increased cost and really couldn’t make a clear case to spend the extra money, our current performance is more than sufficient to achieve the goal.

Recommended Veeam Settings
– See this KB for Best Practice: https://www.veeam.com/kb1745

– Please make sure you are on the 3.13.3-1612.1 release, lots of issues/slowness with older builds, especially with Veeam.

– Job settings that yield best dedup results in our testing:

3

4

 

Real Environment Results
Prior to adding StoreOnce, a month of Veeam data took almost 6TB of storage. Based on current projections we will be close to a year + of retention with StoreOnce which is excellent. Environment consists of a mix of Windows/Linux VMs/Appliances/etc. Jobs are grouped primarily by data recovery needs (daily, hourly, monthly, etc.) rather than by OS type or other factors, in testing we did not notice a significant difference in terms of grouping by OS type, etc. for dedup sake. Overtime the dedup ratio continues to climb, after tweaking Veeam backup jobs to the recommended settings this went from 2:1 to now 8:1 and continues climbing. Considering the Veeam backup files are already pretty small this is a pretty good ratio I think, a lot of it depends on the data thrown at the dedup engine though. Our average throughput to the VSA is usually in the 4-500MB/s range which is quite good for a dedup appliance. I did notice extremely poor performance when trying to copy/move data from a Windows VM directly to the VSA so I let Veeam manage all file transfers internally and received much better performance; i’m sure with some tweaking this could be improved.

1

2

 

UPDATE 091216
After running our StoreOnce the past while, our dedup rate has continued to climb and is now at an awesome 12:1!

UPDATE 111816
After running our StoreOnce for a while, we have now seen our dedup rate “stabilized” at 16:1! A bit above average from what the sales folks told us so we have been pretty happy, it is a very affordable dedup solution!

Overall happy with our decision, Veeam and HP work well together. At some point we will explore the catalyst technology but to be honest things are working well and not sure extra cost could be justified. I like the scalability of the platform as well as the flexibility. I really like how I can easily move this data offsite for DR/archive with another software/hardware StoreOnce. The software price is very competitive and with the “just works” nature of StoreOnce I would choose it over a commodity server with Windows dedup situation; Microsoft’s dedup tech does not have the maturity YET of other solutions in my opinion based on use.

Chicago VMUG Review

I got to attend the Chicago VMUG yesterday, what a treat! Was definitely worth the drive and time. It was OBVIOUS to me and I’m sure others that the depth and width of VMWare’s ecosystem was on full display yesterday, I have NEVER seen a HyperV conference with this level of technology or partners all pushing the same common vision/dream. A gentlemen yesterday asked the question of “VMWare vs. HyperV” and one of the vExpert’s gave a similar answer, the ecosystem is far superior and its true (not even discussing the technology superiority!) All I can really say is that it’s a good time to be a VMWare customer!

VMWare’s full compute/storage/network virtualization strategy was the most impressive “on display” the entire event. I would have to say at least half the vendors were offering some variant of either storage, compute or network virtualization/optimization. There were a few “cloud” folks for good measure as well. A lot of the traditional vendors TRYING to catch up and “add-on” virtualization to their products, won’t name names. Also saw some “new to me” vendors that had some pretty promising stuff going out there:

– 10zig. http://10zig.com
They make the usual VDI/virtualized desktop thin clients and management software. They had some extremely compact units though. Perhaps the coolest item from them, a stripped down Linux thin client OS that you could put on any comodity hardware with management: http://10zig.com/no-touch/
We use HP Thin Clients currently today but I could easily see this as an OS upgrade path to newer Citrix support/etc. without having to purchase new hardware.

– LG USA. http://www.lg.com/us/commercial/desktop-virtualization/lg-23CAV42K
Didn’t know LG made thin clients, saw their 23CAV42K which is basically an all-in-one monitor/thin client solution, coolest piece was its all POE! HP has some all-in-ones but last time I looked at them all they had were 19in units, we have standardized on a 22in+ real estate so this item was incredibly attractive to us. Also its great because your POE switch becomes the only needed power backup/generator device for user work continuation during power events!

– Condusiv Technologies. http://www.condusiv.com/
This was an interesting one. They basically make a software layer for Windows VMs only that improves I/O performance, marketing was saying 50%+, at this time we don’t really have any I/O issues but definitely an interesting product.

Got to meet a lot of great vendors and sit in on some awesome breakouts by Veeam, Condusiv, VMWare and others. Got to finally meet Rick Vanover from Veeam, was awesome to finally meet him, awesome guy and awesome cheerleader for Veeam. Also got to touch base with the PRTG/Paessler team, was so nice to finally see them represented and get to meet some of their folks. We’ve been a loyal PRTG user for the past year and love it, most exciting news, a full syslog server has been integrated into PRTG allowing us to ditch Kiwi SysLog FINALLY! On a total side note, Veeam and Paessler are 2 SHINING examples of companies who listen to their customers and you can see that feedback reflected in future releases, which to me is HUGE and is a LARGE factor for why I continue business with these companies, I would encourage you to do the same! If your current company doesn’t listen to you, send a message with your dollars!

Veeam: Slow Processing Configuration

We are running Veeam Backup and Replication 6.5 to backup our VMWare environment and also using it to replicate our VMs to our DR site for Disaster Recovery. We had recently been noticing a slow down on some of our VMs, especially during the Processing Configuration step; most of our VMs were completing this step in 10-30 seconds, we noticed some were taking 60+ minutes to complete. After doing some investigation and reaching out to Veeam Support we discovered that you must manually map the Resource Pool and Datastores for each replica VM otherwise the long processing delay will occur. I was surprised at how easily this is overlooked in the GUI; it appears many others miss this as well. (http://blog.mwpreston.net/2013/02/07/veeam-job-taking-a-crazy-long-time-on-processing-configuration-check-your-destination) You must click the 2 blue links: Pick Resource Pool and Pick Datastore. Once you define the locations of the Resource Pool and Datastores the processing configuration step will occur much quicker.

setup

After properly mapping each of our VMs to their resource pool and datastores we noticed a substantial difference in the processing configuration step.

BEFORE

Before

AFTER

After

Remember to check in your datastores; Veeam will create a ton of empty VMNAME-replica-# folders in the datastores in which your VM does not reside and should be cleaned out for good housekeeping purposes!

Sure would be nice if they would automate this process thus allowing Veeam to determine the datastore and automatically set these settings!

Veeam Backup: Part One

This is part one of a multi-part post on our Veeam setup, we recently deployed Veeam and have been using in production for close to a week now. Below are some initial notes/thoughts on our Veeam setup. Our Environment/SetupWe have a complete virtu…

This is part one of a multi-part post on our Veeam setup, we recently deployed Veeam and have been using in production for close to a week now. Below are some initial notes/thoughts on our Veeam setup. 

Our Environment/Setup
We have a complete virtualized environment running vSphere 4.1. We have (3) ESX 4.1 hosts connected to (2) Equallogic PS100 SANs. The SANs have around 4TBs total in their storage pool. We are running Veeam B&R on a physical 2008 host; we prefer to have our backup/restore solutions separated from the infrastructure that they are backing up. Our Veeam box is connected to around 7TB of DAS storage from an HP MSA60 (populated with 2TBx6 drives in RAID6). We are using SyncBackPro software (http://www.2brightsparks.com/syncback/sbpro.html) to copy our Veeam files to our offsite location every evening over our MPLS circuit. Our offsite location has around 24TB of storage onsite so we can archive 1+ years worth of backups.
See the below diagram:

Backupoverview

Veeam Setup
We have a very simple job setup, we only have (4) Veeam Jobs:

  • DWM (Daily/Weekly/Monthly): This is our main backup job for all our VMs, has a mix of Linux/Windows VMs. For Windows boxes, ensure you have VSS enabled and working, enable Application-aware processing and ensure you have valid credentials, once that is done things are pretty much setup on the Windows side (minus applications that are NOT VSS-aware, for those cases you will need to write some pre/post scripts via VMWare Tools to address your application-specific procedures to ensure it is in a “clean” state). Linux, due to its lack of VSS, needs a bit more configuration, mostly in the way of writing some pre/post scripts to handle various applications you may have. On future blog posts I will post some of the scripts/techniques we are using, one simple example would be for MySQL, a simple stop/start of the service would be a great way to achieve this, if you can afford the small downtime this creates.
  • Onetime: This job handles our VMs that don’t have frequently changing data and that we only run monthly.
  • VirtualCenter: This job backs up our VirtualCenter server. You need to locate the host your VC VM is running on and add that host to the Veeam console (not the VC server!) for a successful backup. The downside to this is that you will need to update this job anytime this VM changes hosts, you may want to tell DRS to leave this VM alone.
  • ProductionApp: This job backs up our primary business application every hour, so far we have noticed no performance issues with this frequency. I will write a future blog post about this job as this VM runs an IBM Informix database that has some unique requirements to get in a consistent state.

At this time we are using Network mode to backup our VMs, we currently have the capability to do SAN-level backups but chose not to due to the risks of VMFS corruption. Veeam by default disables automount in Windows but we are not comfortable with that being our only safety net, currently Equallogic does not support granting read-only permissions to a particular host, you can make the entire volume read only but this is of little use. Should automount become enabled via update or something else, there is possibility of damaging our VMFS volumes. We will wait until Equallogic adds this feature then give our Veeam host read-only to each volumes then make use of direct SAN backups. We currently have a RFE with Dell to add this feature but who knows when/if this will happen! Spoke with Dell this week who confirmed for me that this feature has been coded and scheduled to be released in next firmware update!

All our jobs are set to use Incremental mode with Synethic fulls on Saturday and one Full backup a month on 1st Saturday of the month.

The first initial backup for our environment (around 1.5TB) took approximately 18hours, 30minutes. Incrementals after that less than 2 hours!
Veeam compresses our initial backup of 1.5TB down to around 800GB which is almost half, quite impressed! Incrementals are usually around 50-200MB depending on quantity of data changed.

We are using BEST compression and WAN TARGET settings to achieve these results.

Our Data Center has an HP MSA60 with around 7TB of DAS storage, our offsite location has 24TB of storage on an HP MSA2012i.

Watch this blog for future posts on our Veeam setup.