VeeamOn 2017: Recap

I just got back from my first VeeamOn event in New Orleans, LA. Was by far one of the best events I have ever attended from a technical AND personal perspective. I think the volume of people and partners in attendance (3000+) speak to how successful Veeam has been at not only creating a great product but also a great ecosystem. This event also unveiled a LOT of new features/functionality that were direct results of customer/partner feedback to Veeam. In my own experience, I have consistently seen suggested features make their way into the products so it’s clear Veeam is listening. Also as a customer the level of innovation/R&D is impressive, Veeam is constantly pushing itself and consequently the backup industry to move forward fast, some of the features presented are so impressive that literally no one in backup world is doing yet.

Some of the existing features discussed during Gostev’s v10 (Coming Soon ) presentation:

  • New Veeam Agents (Lin/Win). This is very exciting as this gives cloud/hybrid/on-prem solutions for any environment, running anywhere. We have been Veeam customers for quite a long time and I can still recall when Veeam was smaller them saying ‘we will never do agents’, glad to see they wisely reversed course on that attitude.
  • Veeam CDP (Continuous Data Protection). This one is a biggie, with the addition by VMware of the IO Filtering APIs Veeam can now grab data continuously WITHOUT snapshots! This will be huge for many of our mission critical apps that don’t respond well to snapshotting especially for the quantity of restore points we want per app.
  • Veeam and N2WS Partnership allowing backup of Amazon EC2 instances. This will help address a large gap in most public cloud providers: Backup of entire VM/instances not just in-VM agents!
  • Scale-Out Repository With Archive Tier. This was very slick, they have now added an archive tier that can go to older/slower storage and/or also out to public cloud providers for long-term retention such as AWS S3 or Glacier, Azure, etc. The best part, this is all native in the console and couple clicks activates it! I think this feature will remove excuses for anyone not to have affordable long-term retention of data.
  • NAS SMB/NFS Backups. This is another slick solution allowing Veeam to directly backup SMB/NFS shares, preserving perms/file versions, etc. Best part is it can restore to original location OR other locations which has some secondary uses for file migrations as well!
  • Veeam Cloud Connect. There were a number of vendors now providing the Veeam Cloud Connect services. Many of them have dramatically expanded the offering to allow emergency environments during a disaster, secured replica seeding and other key features. For many business’ this could allow them to get rid of their DR sites and utilize these vendors saving time/money.

Of course there are MANY, MANY more items released or coming, for a better listing see here: https://www.veeam.com/blog

The conference had a good variety of sessions to attend in addition to the General Sessions. A few of my favorites….

Ransomware Session: It is clear by how full this event was, that everyone is interested in what we can do to protect. Lot of good best practices here, some of the top:

  • Filter/limit untrusted sites, Skype file exchanges, Office exe/vbs/plugins execution.
  • Don’t rely fully on AntiVirus to stop all threats (defense in layers)
  • Use different credentials for Veeam environment
  • Use SureBackup to ensure backups work if needed
  • VLAN Segmentation to keep client devices separate from servers, helps reduce spread of issues.
  • Airgap backups, with the ability of some of these wares to easily spread its important to have copies that are not network-accessible. This is where tape, offsite Cloud Connect or even Archive to AWS/Azure would be helpful.
  • User Education to ensure they aren’t falling for attacks or providing access if threats are able to bypass security layers.
  • Keep Veeam And Backup Stores separate via different credentials, VLANs, etc.
  • Have good monitoring in place to detect anomalies.
  • Use “least privilege” model for permissions.
  • One not covered at the event but one I think is super helpful is Software Restriction Policies (a native Group Policy option) running in whitelist mode. This will basically immediately block all threats from running unless they are whitelisted or trusted Windows services. In my experience this helps stop a LOT of these ransomware attacks as they would be prevented from executing in the first place. It does take some time to setup/test the policies but it pays dividends later.

Microsoft Office365 Whats Next

This was a MUST attend for me as our use of Office365 continues to skyrocket and being able to protect data/assets is critical. Microsoft does a poor if non-existent job of this currently so the need for external backup is critical. There are some exciting things coming in the Office365 suite, in 2 separate phases. Version 1.5 which hopefully is coming in June/Aug of this year will breakout the roles (proxy/server) to allow flexibility and scalability for larger environments, add PowerShell/Restful API support and other improvements. Version 2.0 which hopefully will be before year end is the most exciting as it will include backup for Sharepoint/Onedrive. THIS IS THE CRITICAL FEATURE WE NEED MOST! If you are wondering does Veeam listen to customers, answer is a YES, check out the forums, lots of folks wanting this and it is on the list: https://forums.veeam.com/veeam-backup-for-microsoft-office-365-f47/

These improvements will make it easy to recommend we purchase this product once our 1 year free license expires.

Other Thoughts/Comments…

  • One thing I really appreciate about Veeam is how open and transparent it is. A large number of these sessions were hosted not by sales people but by Veeam engineers or managers that actually work the product so deep technical was possible as well as honest conversations about roadmap, why did you do it that way, etc. Being able to ask the Product Managers direct questions is really cool and helps when roadmap planning internally.
  • Veeam also allows customers to get up and present, saw some cool sessions from the University of British Columbia and others that were deep technical and not just sales events. Also covered some good best practices and thoughts/comments about what works and what doesn’t.
  • I saw a lot of innovation/R&D work on display at this event. Many features were direct out of customer conversations or forum posts.
  • Networking potential at this event was HUGE! There were literally people from every corner of the planet, I met some folks from Canada, Germany, Russia, Mexico and China.
  • Veeam had a strong partner ecosystem on display as well, see the floor Expo below, tons of vendors offering complementary solutions to Veeam and all the big names: Dell, HPE, Cisco, etc.
  • Tons of great swag/giveaways at this event. Even the basic conference bag that everyone got was good, combo backpack/carry bag and good build, spots for water and snacks.
  • New Orleans is a great city, lots of cool stuff to see/experience and a great location for VeeamOn! I didn’t have a ton of time or energy to do too much exploring but did look around, some pics:


  • Veeam knows how to throw a conference and a party! I was surprised by the quality and quantity of the food and parties. Almost every night there was some kind of event or thing to do, got to see a lot of NOLA places in a short period of time!





Overall was a very good conference, learned a lot, met/networked with a lot of great people and got a good feel for the Veeam 2017 roadmap, was most excited to see it aligned with the directions we need for our use cases! Next year’s VeeamOn will be in Chicago which is even closer for me so hoping, schedule allowing, I will be able to attend again.

Advertisements

My EDC Bag (Every Day Carrier)

I’ve been packing for VeeamOn this weekend and thought I’d quickly recap whats going in my “every day carrier”. I actually use 2 primary bags in my life these days, one is a “daily driver” that covers the basics and another “heavy use” bag that is much bigger, the larger bag is for data center work as I have all my adapters and cables whereas my daily driver I usually use for home to office and back. My current daily driver includes the following:

– Surface 3 Tablet with keyboard
Case Logic Case (This is meant for a 14″ laptop but perfectly fits my Surface and iPad)
– Microsoft Touch Mouse (In Veeam green, thanks Veeam!)
– iPad Air (I have 2 cases, one is a lightweight rubber flip case and the other is a bluetooth Zagg keyboard for when I need to do notes/extensive typing)
Anker 2in1 charging plug (Can charge 2 devices at once)
– 10ft Anker charging cables, 1 for Surface, 1 for iPad/iPhone (These are nice as they can reach farther from an outlet than the standard Apple cables)
– Apple Standard Headphones (i’m not an audiophile so they work fine)
– Mobile battery (Blue box, generic battery power unit I got from a vendor at a conference plus a couple of short lightning cables)
– iPhone 6s with Apple Battery Case

I find this basics kit covers most of my day to day needs. I can travel to different locations for work, back and forth to the office and have the critical elements I need. I’m also able to keep this in a nice small, lightweight carrying case. Now, the other scenario is when I’m doing “heavy lifting” type stuff: deploying new servers/hardware, site upgrades, etc. This typically requires a huge variety of cables, adapters and other gear. For this I need a rather large bag that can store all of this comfortably. This kit includes:

– Adapter breakouts for VGA and network for Surface 3
– 10ft CAT6 network cable
– 10ft CAT6 crossover network cable
– RJ45/DB9 serial cables in popular configurations and also associated changers to go between the 2 (RJ45 to DB9, RJ45 to RJ45, DB9 to DB9).
– Startech ICUSB2321F. This is a USB to Serial adapter for my Surface. I like these units as they have the FTDI chip in them which seems to work better than other units especially with older hardware.

I like to have the 2 bags as then I don’t have to carry this mismatch of equipment everyday, especially considering how infrequently I need all of this. Makes it very nice to walk into my office with a nice small, lightweight bag with just the essentials. My other bag I keep under my desk, loaded and ready if needed.

Now my ideal daily situation would be this…..
An iPad PRO that actually had a functional mouse/keyboard that I could literally carry to the office, dock it for power/bigger monitor and use all day. No cables to carry as I would have cables at office and home for power and all day battery life. Use the AirConsole and GetConsole cables/app (https://www.get-console.com/shop) for when I need to be direct connected. I could eliminate bags, cables and most importantly, my Surface 3! But alas, this is only a dream as I still need a functional mouse for the majority of my day to day and Apple does not yet understand the need to compete with Microsoft on the keyboard/mouse aspect of a “replace your laptop” type device.

Veeam Office365 Wishlist

Have been using Veeam’s Office365 backup solution now for 30 days, have to say I am very impressed with it. I do however have a couple of items that would be “nice to haves” to improve the product.

  • Support for backing up Sharepoint and OneDrive document libraries. This is a big one, especially for environments that have gone “all in” with Office365. Microsoft’s own backup/DR abilities are pretty weak, especially for the tenant side. This is heavily requested on Veeam forums:
    https://forums.veeam.com/veeam-backup-for-microsoft-office-365-f47/feature-request-backup-of-office-365-sharepoint-t36949-90.html
    Some must-haves…..

    • Ability to restore individual items, lists, folders, sites, etc.
    • Preserve permission/inheritance of restored content
    • Export data back to original location OR network share/folder locally OR perform rename function something such as restoreddoc_DATE.docx and place in original location
    • From a priority perspective it would be nice to see Sharepoint done first as this is the “business” side where most critical documents live, whereas OneDrive is more a personal repo.
  • Offer more folder exclusion options such as Sent Items. Also being able to define a set of custom folders to exclude would be helpful.
  • Enhanced retention policies similar to the options available in full Veeam BR.
    https://forums.veeam.com/veeam-backup-for-microsoft-office-365-f47/need-to-modify-retention-policy-t41359.html
  • Offer more options or those similar to Veeam BR in terms of repository storage options.
  • Personally I would like to see the separate console go away and these features be integrated natively into the Veeam Backup And Recovery console. In the future as more application-type options are added I don’t want to have separate consoles to access them, I want them all from within Veeam Backup console.

Tools Of The Trade: Moving To Office365

Getting to the cloud, specifically Office365 can be a challenge. You more than likely have hundreds if not thousands of mailboxes crammed full of user emails, network shares full of years of user documents, etc. Getting this into the cloud while respecting the business’ ability to work can be a challenge. Here are a couple of the tools I have found that are incredibly effective at helping meet these challenges. A couple of the “requirements” needed in my mind….

  • A cloud/hosted solution is ideal as this allows faster deployments instead of having to setup on-site/local resources to migrate.
  • Solution should be a “swiss army knife” in terms of out/in options for going between multiple clouds/providers/data types.
  • Affordable/scalable pricing.
  • Support to stand behind the migration product.
  • Abilities to incrementally sync data “up” so you don’t need to do everything at one time.
  • Some sort of CSV/script automation ability (who seriously wants to sit and manually do each mailbox 1 at a time?)

With those goals in mind, here are the 2 tools I use for migrations that have been very successful. One for email/mail data and the other for data/documents.

BitTitan’s Mailbox Migration
https://www.bittitan.com/products/migrationwiz/mailbox-migration

This is a great, flexible tool. You have lots of legacy/niche mail options (Lotus, Zimbra, etc.) and can go between many providers such as Gmail, O365, etc. Its cloud based so as quick as you can provide an admin account to grab the data it can begin migrating. A VERY useful component is the ability to incrementally sync the mailbox data and it has scheduling logic. This is very useful for cutover migrations, the week leading up every day you can sync mailbox data then cut MX records and sync remaining mail, no data loss and seamless process. I have also had very good accuracy in terms of no missing mail/lost items which other products I have had issues with.

Files To Go (ThinkScape)
http://www.thinkscape.com/SharePoint-Online-File-Migration-Tool

This is my goto tool when moving large quantities of network shares/user data to OneDrive/Sharepoint. My favorite feature is the pricing as its time based so you can buy it for 1-6 month increments which is very useful! It also mitigates long file names, file extension, character issues, etc. that can trip up an O365 migration. You can also do CSV file automation which really helps. It is an “on-prem” solution but the feature/functionality/cost tradeoff is worth it. Other tools can be extremely expensive whereas this is very cost effective. I believe it helps that a Microsoft O365 MVP actually wrote the product as it specifically gives the features most migration engineers care about.

Between those 2 tools, I have been able to successfully move the 2 main use cases in Office365 (email and documents) with minimal issues/cost. Hopefully that helps you as well.

Veeam Office365 Backup: Mailbox Processing Errors

Have been doing some testing with Veeam’s new Office365 Backup solution, so far pretty impressed. Did run into an interesting issue out of the box, we had some random mailbox processing errors such as:
– Failed to synchronize item changes. The operation has timed out.
– Async batch export failed with timeout.
– Mail item data export failed. There is an error in XML document.

Originally I just attributed this to “issues in the cloud” with EWS since Veeam uses this but after a day or so it became clear something else was happening here; as a safety net did some packet tracing in our firewall cluster to rule out potential security services causing the timeout. I did some additional testing and also spoke with Veeam support, their recommendation was to add the following line immediately after the <Archiver> tag in config.xml located in: C:\ProgramData\Veeam\Backup365

<Source WorkerThreads=”4″ BatchSize=”10″ BatchPart=”10″ BatchTimeout=”900″ BatchMaxItemSize=”5″ />

After adding this line and re-running the job it seems to have resolved the issue as all of our mailboxes are now backing up! Currently this issue was happening on build 1.0.0.912

My Workspace 2017

Thought I would change pace from usual deep-technical stuff and showcase my current workspace at work. I like a minimal-type approach to things so I can stay focused on my work. My 2 “daily driver” devices are a Microsoft Surface 3 on the Microsoft docking station and an HP T520 Win10 Thin Client, both of these devices I have hidden in my office cabinets so my workspace is neat and tidy. Below is my current workspace and some notes about it:

kensiltra-workspace-2017

Monitors
HP V242H 24″ monitors
The thing I like about these units is that they have VGA/HDMI/DVI inputs so I can easily toggle between different devices, I also have some video feeds of items like our monitoring wallboard (PRTG Map), security cameras, etc. Gives me a lot of flexibility in customizing my workspace without touching cables or spending a lot of time. At some point I could probably invest in a video switcher or KVM but for now this is working well. These are pretty basic LED 1080p displays, nothing fancy but they produce a nice image. I don’t do much video/graphic editing so they work great.

Monitor Mount
Mount-It! Dual Monitor
Not the fanciest mount in the world, I got it on sale for around $30 from Amazon, it has cable management and is pretty flexible. Only bummer is the monitors side-by-side are slightly off alignment due to how the arms mount, there is some “give” in the bracket. For the price though it is a decent unit and I’ve been happy with it.

Photo Frame
This is a PanDigital digital photo frame. It is cheap and runs a bunch of personal pics of the kids, wife, vacations, etc. from a USB drive. Can schedule it to turn on when I get in and shut off automatically so accomplishes the need.

iPhone Dock/iPhone
I have an iPhone 6s that I dock on a standard iPhone Lightning Dock
H
as the option to also plugin my headphones if I am on a conference call or want to listen to music during the day plus charges my phone.

Keyboard/Mouse
I don’t do a ton of “fancy” work so a simple keyboard and optical mouse are fine. I use the standard ones HP ships with their thin clients. So far working good. My mousepads are a combo collection I have had for years of a Dell one on top and a consulting house’s pad on the bottom, don’t ask me why but they have stuck with me through 3 different employers now.

Water Bottle
We don’t have filtered water at my work so I just use this guy, its a Brita Filtered Bottle. It will filter your water as you drink, does have some reduced flow though but a nice bottle overall for the work day.

That is pretty much it for my day-to-day work environment. I like to keep things simple and reliable so I can remain focused day to day. I keep dreaming of the day I can get rid of both my current devices (Surface 3 and HP Thin Client) and just use an iPad Pro but the current lack of mouse support really kills this dream, especially since Apple advertises this as a “PC Killer” type device. I got some inspiration for this post from the following sites, thought I would link them to say thanks. Keep watch for another post on my EDC (Every Day Carrier) bag and some of the items I use daily!

Mac Setups From OSXDaily

Brent Ozar: SQL Guru’s Workspace

Deploying Office365 ClickToRun Products

So Office click to run is the “new” deployment process for Office365 software. While a user could technically just “click and run” their own install they hopefully don’t have admin rights to do this and thus a more “enterprisey” deployment is needed, right?

First a few basics….

So lets begin……

  • I like to create folders in my root network share for each installation, O365 ClickToRun uses *xml configuration files to determine what and how the Suite gets installed. It is handy to separate this as you can limit installer to only do Skype for example. I usually do something like this:
    \\RandomPath\Office2016
    – Skype_64bit
    – NoOutlook_64bit
  • You need to save the Deployment tool (setup.exe) to the network path folder where it will be deployed. Example: \\RandomPath\Office2016\Skype_64bit\setup.exe
  • Now that we have it saved, we need to write a configuration.xml file that will live in the root with setup.exe that tells the installer what you want installed and what settings/options to push along. Below I’ll provide an example configuration which covers the basics:
    <Configuration>
      <Add SourcePath="\\Server\Share" OfficeClientEdition="32"  >
        <Product ID="O365ProPlusRetail">
          <Language ID="en-us" />
        </Product>
        <Product ID="VisioProRetail">
          <Language ID="en-us" />
        </Product>
      </Add>  
      <Updates Enabled="TRUE" UpdatePath="\\Server\Share" /> 
      <Display Level="None" AcceptEULA="TRUE" />  
    </Configuration>

    The sourcepath will be your UNC path to the install, client edition is either 32 or 64 depending. Product ID will be the product(s) you want to install, here things get a bit tricky. Microsoft has different “levels” of licensing in O365 that correspond to this so based on your licensing this will need to match, luckily they have a document that sort of attempts to help: https://support.microsoft.com/en-us/kb/2842297
    There was definitely some trial and error, if this setting is wrong and a user with different licensing attempts to login you will run into issues/errors so testing first is best. The other options are better described in this article: https://technet.microsoft.com/en-us/library/jj219426.aspx
    The article also mentions some Exclude parameters you can use to not install certain things, say for example no Outlook. Below I will provide some templates we use for various items….

  • Install Skype (64bit) Only For Use In A Terminal/Citrix Environment:
    <Configuration>
    <Add SourcePath=”\\RandomPath” OfficeClientEdition=”64″ >
    <Product ID=”LyncEntryRetail”>
    <Language ID=”en-us” />
    </Product>
    </Add>
    <Updates Enabled=”TRUE” UpdatePath=”\\RandomPath\Updates” />
    <Display Level=”None” AcceptEULA=”TRUE” />
    <Property Name=”SharedComputerLicensing” Value=”1″ />
    <Property Name=”PinIconsToTaskBar” Value=”FALSE” />
    </Configuration>
  • Install Visio (64bit)
    <Configuration>
    <Add SourcePath=”\\RandomPath” OfficeClientEdition=”64″ >
    <Product ID=”VisioProRetail”>
    <Language ID=”en-us” />
    </Product>
    </Add>
    <Updates Enabled=”TRUE” UpdatePath=”\\RandomPath\Updates” />
    <Display Level=”None” AcceptEULA=”TRUE” />
    <Property Name=”PinIconsToTaskBar” Value=”FALSE” />
    </Configuration>
  • Install Office 2013 (64bit), NO OUTLOOK
    <Configuration>
    <Add SourcePath=”\\RandomPath” OfficeClientEdition=”64″ >
    <Product ID=”O365ProPlusRetail”>
    <Language ID=”en-us” />
    <ExcludeApp ID=”Outlook” />
    </Product>
    <Product ID=”LyncEntryRetail”>
    <Language ID=”en-us” />
    </Product>
    </Add>
    <Updates Enabled=”TRUE” UpdatePath=”\\RandomPath\Updates” />
    <Display Level=”None” AcceptEULA=”TRUE” />
    </Configuration>

Once your xml file has been written to accommodate the needs, save the xml into the path where you saved your setup.exe file. Then run the following command:
\\RandomPath\setup.exe /download \\RandomPath\configuration.xml

The first section calls the location of setup.exe (Office Deploy Tool), next option triggers the download command which basically allows Office to go out, download the installers/packages for office and the last field points it at your configuration file so it knows what to configure/install.

Once this has been done, you can simply use same command but with the /configure command to install:

\\RandomPath\setup.exe /configure \\RandomPath\configuration.xml

That simple script above will deploy Office on the desired machine using your desired configuration data. Use Group Policy or 3rd party software deployment of choice at this point to mass deploy. 

A few gotchas/comments….

  • If you are deploying this on RDS/Terminal servers, you MUST have one of the following “desktop virtualization” entitlement licenses: ProPlus, Ent E3, Ent E4, Gov E3, Gov E4. You also MUST include Shared Computer Activation in the xml configuration file (see my example for RDS above)
    <Display Level=”None” AcceptEULA=”True” />
    <Property Name=”SharedComputerLicensing” Value=”1″ />
    Some of the Office suite sticks the licensing tokens in “not roaming profile friendly” locations so be prepared to test, Skype is a good example. You would think Microsoft’s own stuff would work well with roaming profiles in an RDS world but not so much.
  • Like most things Microsoft licensing, ProPlus and Shared Computer Activation is not pleasant. It will store a token in the user’s folders, have seen issues with it “expiring” or becoming corrupt causing licensing/activation issues with certain users. Keep in mind you have 5 “licenses” to use on various machines/PCs, Shared Activation should not count against this however. The server will need constant internet access to maintain licensing/activation as well, would be wise to ensure your internet gateway is allowing most IP/URLs that O365 uses as there are quite a few: https://support.office.com/en-us/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2
    Office checks activation every 30 days so users need to have “phoned home” at least once in that time period to prevent Office going into “reduced functionality” mode.
  • If you are planning to work Office into an image, don’t sign in to the Office 365 portal to install Office 365 ProPlus from the software page. If you do, activation occurs automatically. After the installation is complete, don’t open any Office programs. If you open an Office program, you are prompted to sign-in and activate. Even if you don’t sign in and you close the Activate Office dialog box, a temporary product key is installed. You don’t want any kind of Office 365 ProPlus product key installed in your operating system image.
  • Should just plan on bookmarking this page as you will have some kind of “goofiness” over time with Office licensing, especially in a shared activation world: https://technet.microsoft.com/en-us/library/dn782859.aspx
  • We saw constant ongoing activation/login prompts on Skype For Business deployed in an RDS/Citrix environment. Appears some of the licensing tokens are kept in Local App Data instead of roaming so we did a “quick fix”. At logoff we basically copy the appdata in local to their roaming profile, then on login we copy it back so Skype “works”, not pretty but it works.
    #Copy Skype App Data To Roaming Profile
    xcopy “%LOCALAPPDATA%\Microsoft\Office\15.0” “%APPDATA%\SkypeConfig” /E /C /Y /I#Copy Skype Roaming Profile Data
    xcopy “%APPDATA%\SkypeConfig” “%LOCALAPPDATA%\Microsoft\Office\15.0” /E /C /Y /I
  • Had some situations where users would get “This feature has been disabled by your administrator”. This comes down to Online Content being disabled. Quick fix to restore:
    Windows Registry Editor Version 5.00
    [HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Common\Internet]
    “UseOnlineContent”=dword:00000002
    [HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Common\SignIn]
    “SignInOptions”=dword:00000000
    More info: https://support.microsoft.com/en-us/kb/3039000

Overall the Click To Run technology is pretty nice, Microsoft could do things to further simplify the process but it is not too bad. The biggest challenges we have seen in a production setting have been Click To Run ProPlus licensing used in a Citrix/Terminal Server environment as you encounter licensing/activation with users. You also need to remain vigilant that a user is provided the “correct” licensed version of Office to use with the features they are licensed for!