Error: Call IscsiManager.QueryBoundVnics for object…

Ran into a situation where an environment had forcefully (meaning literally ripped the cards out of the server) removed NICs bound to iSCSI in vSphere 6. When attempting to manage iSCSI settings in the iSCSI adapter you would get the following error:
“Call IscsiManager.QueryBoundVnics for object iscsiManager on vCenter Server failed”

Appears this caused some corruption in the iSCSI stack on the host due to the cards suddenly “disappearing” from the system. Pretty easy fix to this one luckily….

1) vMotion everything off this host and put in maintenance mode
2) Copy IQN/Data settings for iSCSI config as will need to re-create
3) Go into iSCSI adapter settings and hit the Disable option
4) Delete contents (3 files) of this directory: /etc/vmware/vmkiscsid/
5) Reboot host
6) After reboot, add a new iSCSI adapter and re-add existing config IQN/Data settings. Also dont forget to bind the adapters as needed/required by your config!

Thats is, pretty simple process and gets things working again. Best bet, don’t rip and pull NICs unless config data has been removed!

Comcast Fiber Internet Routing

Ran into an interesting project while deploying a couple of sites using Comcast Fiber Internet and Sonicwall firewalls. Most providers like AT&T usually give all usable IPs in 1 block, Comcast breaks this up into a routing block (/30) and then usable IPs (/29) separate. In this scenario you essentially setup the /30 like you normally would on the Sonicwall WAN interface and then use static ARP to handle the /29 block.

Currently Sonicwall does not allow assigning multiple IPs to the primary/secondary WAN interface. The “secret sauce” to making this work is to add an ARP address on your WAN interface for each of the /29 addresses you want to use. In short these are the steps:

1) Setup /30 address on WAN interface as usual
2) Create the /29 range as an address object/s
3) Add an ARP address on WAN interface for each of the /29 addresses you want to use
4) Create a routing policy to route the /29 addresses outbound of the WAN interface
5) Create usual NAT/firewall/address objects/rules for the /29 addresses

A more detailed walk-through here:

HP StoreOnce And Veeam

Our current Veeam environment is virtualized (so we can take advantage of Hot-Add) and currently our Veeam data was being stored on legacy Equallogic arrays that needed to come out of service. We began looking at options for replacement, of course there are a variety of options. Veeam themselves seems to recommend the commodity server route, there are tons of inexpensive NAS solutions and of course dedup appliances. Since we are mostly an HP shop the first stop was their solution which in this case is the StoreOnce platform. Whats great about it is that it really scales and gives you flexibility, you can buy into a software only or hardware solution and go from there. And with the recent integrations with Veeam it was a no-brainer.

HP charges a pretty good markup for the hardware StoreOnce platform; especially after talking with HP engineers its basically a G8/G9 server platform. We chose to just go out and purchase a commodity G8 server, load it with drives and memory, throw ESXi on it and run the StoreOnce VSA. Comes as a prepackaged appliance, setup in a matter of minutes, throw a large volume of storage at it, during boot it will detect and auto provision for use. Just need to go in and add a NAS share and thats about it. Really is quite simple and with the cost of the software only option VERY affordable even in the small to mid market and VERY scalable. Those who use HP’s tape loader platform will recognize the interface, not sure if thats good or bad. Even though there is the Catalyst integration we are not yet using that as there was increased cost and really couldn’t make a clear case to spend the extra money, our current performance is more than sufficient to achieve the goal.

Recommended Veeam Settings
– See this KB for Best Practice:

– Please make sure you are on the 3.13.3-1612.1 release, lots of issues/slowness with older builds, especially with Veeam.

– Job settings that yield best dedup results in our testing:




Real Environment Results
Prior to adding StoreOnce, a month of Veeam data took almost 6TB of storage. Based on current projections we will be close to a year + of retention with StoreOnce which is excellent. Environment consists of a mix of Windows/Linux VMs/Appliances/etc. Jobs are grouped primarily by data recovery needs (daily, hourly, monthly, etc.) rather than by OS type or other factors, in testing we did not notice a significant difference in terms of grouping by OS type, etc. for dedup sake. Overtime the dedup ratio continues to climb, after tweaking Veeam backup jobs to the recommended settings this went from 2:1 to now 8:1 and continues climbing. Considering the Veeam backup files are already pretty small this is a pretty good ratio I think, a lot of it depends on the data thrown at the dedup engine though. Our average throughput to the VSA is usually in the 4-500MB/s range which is quite good for a dedup appliance. I did notice extremely poor performance when trying to copy/move data from a Windows VM directly to the VSA so I let Veeam manage all file transfers internally and received much better performance; i’m sure with some tweaking this could be improved.




UPDATE 091216
After running our StoreOnce the past while, our dedup rate has continued to climb and is now at an awesome 12:1!

UPDATE 111816
After running our StoreOnce for a while, we have now seen our dedup rate “stabilized” at 16:1! A bit above average from what the sales folks told us so we have been pretty happy, it is a very affordable dedup solution!

Overall happy with our decision, Veeam and HP work well together. At some point we will explore the catalyst technology but to be honest things are working well and not sure extra cost could be justified. I like the scalability of the platform as well as the flexibility. I really like how I can easily move this data offsite for DR/archive with another software/hardware StoreOnce. The software price is very competitive and with the “just works” nature of StoreOnce I would choose it over a commodity server with Windows dedup situation; Microsoft’s dedup tech does not have the maturity YET of other solutions in my opinion based on use.

Issue: Cannot Create Ramdisk Of Size

Working on a vSphere 5.5 to 6 upgrade and when attempting to re-mediate the host to 6 using HP’s custom ESXi version, received a failure due to “Cannot Create A Ramdisk of size 385MB to store upgrade image”. After doing some investigation, appears some of the VMware default partitioning does not leave enough space for the upgrade image due to the image’s size especially in HP’s case. Environment was all HP G8 servers running HP’s custom ESXi base. A quick check of the partitions confirmed the available space was smaller than the image needed, couple of different ways to attack this, I chose the following.  Easiest method is to find a datastore we can move the default scratch space to that has more room which is the method I chose.

  1. Need to identify the ID of the datastore you want to use, easiest way is either via GUI in Manage—>Storage or CLI on host run: esxcli storage vmfs extent list
    I chose to create a Scratch directory under the datastore to keep things clean.
  2. Log in to vCenter Server using the vSphere Web Client/GUI, click Hosts and Clusters, then select the specific host. Click the Manage tab and then click Settings. Click System and then click Advanced System Settings.
    Locate ScratchConfig.ConfiguredScratchLocation.
    Click Edit and add the path to the scratch directory. The path will look something like this, replace red ID with the datastore ID from step 1.
  3. Once you have this set, reboot the host for the changes to take effect.
  4. Once host is back online should be able to re-mediate the host without any further issues.

I have so far seen this on HP’s custom images but would imagine this would happen with any custom ESXi install and/or plugins that exceed VMware default sizes.



Back To Blogging In 2016

So considering my last post was from 2014 I’ve been away a while. Of course the usual excuses of “busy at work”, family, etc. apply. I have missed the blogging world and also missed sharing ideas/comments with folks. I am trying to set a realistic goal for myself and post 1 new item every month so a total of 12 posts per year, hopefully will be able to increase that but thats my initial goal. I am going to be going through and approving pending comments and also update/refresh the blog. Most of my posts will center on current technology I am working on or with but I may also post some travel items or other tidbits as well. Just as a basic list, these are some blog posts I have planned/thinking about based on current items I am working on:

  • Aruba Central
    This is Aruba’s newest cloud management suite, we are using it primarily for guest access but also very convenient in an Instant world to be able to mass deploy settings for consistent settings between our sites without having site controllers.
  • Zimbra To Office365
    After many years of being a Zimbra customer and working with it in different companies we have finally decided that Zimbra’s lack of support and poor feature deployments have caused us to move to Office365. I will review some items we ran into and also cover our migration using BitTitan. I will also cover mass deploying Office2013/16 as provided by O365 licensing.
  • Informational Kiosks
    I seem to be deploying a lot of business dashboard/data kiosks lately and thought I would shoot out a quick blog post about the hardware/software that makes it work well.
  • HP StoreOnce
    We are currently in the evaluation process for some dedup backup storage for our Veeam jobs and based on our existing HP-centric world would be a natural fit, especially as we look toward primary storage replacement in the coming years.
  • Veeam CloudConnect
    We are currently evaluating Veeam’s cloud DR solutions. It is very enticing to be able to get rid of the cost of a “hot” DR site and replace with an “on demand” model.
  • 3cx IP PBX
    After years of living in the shadows of a Mitel/Intertel multi-site traditional PBX, we have since decided to move to 3cx for cost savings. It is quite a fully featured product and has an attractive licensing model as compared to Cisco, Avaya and others.

I am hoping to add to that list but for now that is a “sneak peak” of whats coming. Since we are almost at month 6 (June), each of those topics will cover me until the end of 2016 🙂

Please stop back in the next 30 days and I should have an updated post going!

Chicago VMUG 2014: Quick Recap

Unless you’ve been living under a rock or stuck in the trenches of the server room, the Chicago VMUG happened yesterday. First of all, what a great event, the networking, vendors, sessions, food, etc. I have said this before and I will say it again, having been to many events over the years, some over in the Microsoft camp, the VMWare “ecosystem” is amazing, the amount of vendors writing plugins/APIs/code around VMWare’s stack is impressive, plus with VMWare’s SDDC push, they really are one of the only vendors out there with a complete stack solution now. (storage, compute, network) I’ll use this post to highlight some of the main “take aways” from yesterday, some VMWare specific, others industry specific. The VMUG is always a great reminder of the constant in our industry: CHANGE. It was on display in full form yesterday from VDI to SDN to Cloud, etc. There are numerous new technologies hitting the market, some quite disruptive, some barely remembered from a year ago. Some key “industry” points:

  • Back To The Business
    Scott Lowe gave a great keynote yesterday, my favorite topic was the idea of business-IT alignment. Basically getting back to IT being a supportive vehicle of the business, caring less about SANs/Servers/Storage and more about identifying challenges/issues and then creating solutions that address them. Saying “yes” more often then you say “no” and being an enabler to the business. Great message, great concept! For IT as a group this culture needs to be at the front of everything we do. Many of the “symptoms” of poor IT departments like “shadow-IT” have been created by us for our failure to “enable the business” and work at the speed of business, think about that for a minute!
  • IT As A Service Organization
    Somewhat related to my first point, the IT group is becoming a service organization that is more focused on creating the needed applications/services than worrying about the backend technology. Skills like risk assessment, service management, project deployment/management and business alignment strategy will become crucial. Services will exist in on-site, off-site servers/clouds, sometimes provided by 3rd parties, sometimes managed in-house but all managed under a single umbrella, the IT Department.
  • Passionate Technology Professionals
    Nick Marhsall gave an awesome presentation on passion, the driving force behind our personal and professional career success. Our jobs and careers should be driven and motivated by an inner passion. Passion is contagious and spreads. As IT professionals, we need to identify our passions and then invest in them, for our success and the success of our employers.

Some ongoing trends as seen at VMUG:

  • Software-Defined Anything
    Of course the software-defined anything was on full display. There is a fair amount of hype surrounding these technologies and a ton of “infant” stage stuff out there. I attended a session from HP/VMWare on SDN, while I saw some cool abilities coming, I also saw a solution that was in its infancy, had missing integration/pieces (the HP engineer even admitted that HP was trying to get out ahead of the SDN movement and was still building) and was a ways from a production-ready concept. The engineer surveyed the audience and everyone who raised their hand for having heard of SDN, immediately put their hands down when asked who would be deploying it in the next 6-12 months. While I think there are some great potentials coming, the technology needs to bake for a while more before its ready; there also needs to be some better use-case/ROI justifications created for the non-AWS folks of the world, much more practical needs required, easier deployment, etc. The Compute/Storage side of the house is much further along I believe. In the vendor hall I saw a myriad of “software-defined” storage/compute solutions, much consolidation will take place among the “niche” folks in the next few years I believe, especially now that the giants like VMWare, Dell, HP are entering these spaces with solutions; in an industry dominated by relationships, these vendors will use them extensively to their advantage moving forward.
  • Cloud
    Of course the Cloud is still out there and going strong. I have seen a conversation shift happen over the past year however, it used to be it was all about public cloud services, now the conversation is becoming more private cloud or hybrid type scenarios. I think as the hype cycles have died down, folks start using the Cloud, seeing pricing/benefits/uptime/etc. they have filtered through the hype and directed their resource needs at the appropriate mediums (in-house/cloud/etc.). There has also been some serious security/uptime concerns from the “big guys” this past year that has helped shape the discussion (think Snowden revelations, long downtime, etc.). The future looks a lot more hybrid/private than public I believe long term. I also find it interesting the lack of transparency by most major cloud providers as to their financials/success in the market so far and long term.
  • VDI
    The VDI landscape was alive and well at VMUG, especially in the vendor hall. The market really is a race between Citrix and VMWare as the primary market leaders. A whole ecosystem of management and endpoint services has evolved around these services though making management and provisioning much easier. Expect to see further growth as more Desktop As A Service providers launch and provide more/easier services to consume.
  • IT Optimization/Automation/Simplification
    Over the years the trend has been building to automate, simplify and optimize large portions of the IT lifecycle/processes. In my opinion this is HUGELY needed. We have for too long been “creating snowflakes” as Scott Lowe said yesterday. Now part of the problem though, I personally do not believe the concept of scripting/coding is our way forward, there are many getting wrapped up in Python, PowerShell and other ways to “script” making things happen, to me this is more complexity and less automation. Ideally the tools should be written to incorporate these automations into the product directly then can be accessed via the standard GUI interfaces. Example: Instead of writing a script to automate a process, the developer should provide a GUI front-end to do this automatically. VMWare especially should take note and incorporate more time-saving/automation features to reduce/eliminate scripting processes. Also it is important to consider the range of converged infrastructure offerings now out on the market, reduce cabling/complexity in the data center, etc.

Those are some of the larger industry/VMWare trends that are happening now as I saw at VMUG yesterday. There are of course many other trends like BYOD/Mobility/etc. but these particular ones most impact the IT group as a whole.

There were a lot of vendors on display at the VMUG yesterday, a large portion were service/consulting providers, storage platforms, SDAnything platforms, etc. Most of which I have seen countless times before, I did however encounter a couple of “cool” companies out there which I have not seen previously that I thought I would mention:

Cloud based monitoring, they have almost 1000 vendors MIBs imported and ready in their system to use. Pretty simple setup/workings. In-house proxy application installed and then sends all monitoring data to their cloud. Pricing is in packages of devices per month, with a minimum 25 device start point.

Linux-based thin client operating system. Installs on commodity hardware to turn your old desktops into “new” thin clients that are compatible with Citrix, VMWare, etc. They even have a VMWare management appliance for easy deployment/configuration. Per device licensing.

Thanks to all the sponsors the VMUG was a free event, considering the amount of technical sessions, peer networking and vendors that you get access to it is an amazing deal. If you have not attended a VMUG yet I would highly encourage you to do so, it is worth the time; I doubt you will ever regret investing in yourself and investing in others, not only does it “feel good” but also can help boost your career! Head over to the VMUG’s website to find one in your area and get connected.

Customizing Google Chrome for the Enterprise

I guess you could say this is a followup or addition to a series I started a while ago on customization of “other” (read non-IE) browsers in an enterprise environment. Today’s focus will be on Google Chrome. We originally offered IE9 and FireFox in our environment, mostly to compensate for IE’s decreasing performance with newer and newer web technologies/sites, BUT after working with FireFox for a good period of time we have discovered a few things:

1) Mozilla is not really “enterprise-aware” or is at least not making decent efforts to get there. One of the biggest frustrations is either lacking of documentation or sudden discontinuation of features needed to control the user experience. They also have a variety of random files, locations and techniques needed to customize FireFox appropriately.

2) FireFox has some serious memory leaks/issues. Throughout the course of the update cycle we have seen improvements then steps back in terms of memory consumption/leaks. We have had frequent high memory use where there shouldn’t be in our Citrix environment that ironically IE doesn’t have on the same sites.

3) Mozilla has also been making some unfriendly decisions in terms of what features they will or won’t endorse that are needed in an enterprise environment.

With that data in hand we began looking at Google Chrome. Now Google Chrome isn’t perfect and is guilty of a couple items on their own, most notably the lack of understanding of what IT groups need to be able to set/control for the user experience. Now we aren’t talking about “controlling” users in the “mean IT” way, we are talking about setting options and things that ensure they have a good experience. Take an example, a simple configuration option can ensure plugins are automatically run instead of prompting which if the user doesn’t see will cause their site to not work properly, etc.

If you are not at all familar with Chrome in a “business” setting, see this article to get yourself up to speed:

So with that said, let’s look at Google Chrome.

– First basic piece, get Chrome installed. Obviously using AD or a 3rd party automatic deployment is the “right answer” here. In our situation, this will be getting installed in a Citrix environment. Make sure you download the “business” installer and not the consumer one from here:

– Now that Chrome is installed, we need to do some “customizations” so this works properly for our users. Now if you read the above KB article you will see there is a variety of ways to do this. In our situation a global policy for all users works fine but there are ways to do things a bit more specific if needed. There are ways you can auto deploy Chrome Extensions but we have all of these features disabled and use no extensions currently, just one more item to troubleshoot, cause issues, etc.
Note: If you are a Citrix or virtual desktop situation, there are some command line arguments that you will want to use when publishing the app. to avoid some issues. Citrix has an article here:
Also TechRepublic has a good article, look for the “Avoid The Pothole” section:

The customizations we have been using are a mix of Google Best Practice and our own personal deployment preferences. I STRONGLY dislike 3rd party templates in my AD environment so I am NOT doing it that way, what I did was stage the desired settings, cleanup any unique settings and export a production-ready configuration that we apply at the HKLM registry level. This then applies to all users on all of our Citrix servers. Our configuration will not be changing pretty much at all which is why we chose this path, obviously if you are frequently updating things you may want to use the Google ADM templates. Thankfully Google Chrome has a slightly less “burdensome” process for customization than FireFox, although not by much.

Remove Desktop/Start Menu Shortcuts
In various versions of the browser installer there are ways to pass “dont create these shortcuts” but in my testing could not make this work. There seemed to be a lot of discussion on the Google Product Forums about this not working fully or correctly. After testing I found this 2 step to fix. The primary reason you want to remove these shortcuts is because you want to publish or create a shortcut for your users that passes the needed command line arguments to Chrome to avoid issues, see section in red above.

1) Create a master_preferences file in C:\Program Files (x86)\Google\Chrome\Application

The file should contain the following information:

{“distribution”:{“msi”:true,”system_level”:true,”verbose_logging”:true,”do_not_create_desktop_shortcut”: true, “do_not_create_quick_launch_shortcut”: true,”create_all_shortcuts”:false}}

This will help stop SOME of the shortcuts but not all, different versions of Chrome seemed to work and others didn’t.

2) Create a Group Policy logon script that manually removes the shortcuts on login, this also protects against updates recreating them consistently. The basics would be:

rmdir /S /Q “%USERPROFILE%\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Google Chrome”

del /q /f /s “%USERPROFILE%\Desktop\Google Chrome.lnk”

del /q /f /s “%USERPROFILE%\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\User Pinned\Taskbar\Google Chrome.lnk”

Is this ideal, NO, but of course I’m sure it will be fixed in the future but this keeps us moving at least!

Deploy Some Google Defaults

If you load the ADM template into a test machine and configure the desired parameters, you can then export HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Chrome
Make sure you clean out any “unique” attributes, you need to keep it “generalized”. Most of the settings are pretty self explanatory based on their names. If you reference these names on the Google Chrome site you can get more detail, see here:

A lot of these items do things like reduce cache sizes which aren’t good for roaming profiles. We also shut off a lot of the Google “cloud” features like printing, sync, etc. We allow outdated plugins and always authorize plugins so the user’s site they are visiting if it needs a plugin will automatically load and work thus reducing helpdesk calls, the security trade off is minimal as we have other mechanisms to protect against vulnerability, virus, etc. We also define the user data directory to a roaming profile directory so the user’s data roams with them. Note some of the cloud services can’t really “effectively” be disabled via these methods so what we have done is to block these services at the web/content filter level, things like,, etc. Some of this will depend on your organization’s tolerances for cloud use, etc.


Windows Registry Editor Version 5.00

“UserDataDir”=”${roaming_app_data}\\Google\\Google Chrome\\”


That is all we are doing today to customize/control the Google Chrome experience. We do allow auto-updates and have not really seen any negatives (yet) in doing this. There are ways to disable auto-update for organizations that prefer that level of control.

Also be advised as of August 2014 Chrome is now 64bit capable, see here: