Moving my blog to GitHub pages

I have moved my blog to GitHub pages after taking down my last vps host as it was starting to cost too much money for just running a wordpress blog. All of my posts that were previously hosted on my old server will eventually updated and moved into this blogroll. I look forward to spending less time on maintenace of the servers outside of my office hours and just focus on my personal things when I am away from the office. The technical nature of the blog shouldn't change too much as I have moved on to much more interesting things in the last few months since my last post.

Recently my wife and I got a new dog, Cooper. He is a mutt with mostly Wheaton Terrier and Wiredhaired Pointing Griffon. We are enjoying our time with him tremendously now.

Unlocking Locked VM's in ESXi

So one of your virtual machines either won't lock or got locked in ESXi and you need to fix it quickly? Well VMware's KB can be pretty useful when it's direct and to the point. However I found it difficult to figure out after a bit of digging around in my ESXi system. So I thought I would clarify this for everyone here. Often times you'll probably see one of these errors:

  • A virtual machine cannot power on.
  • Powering on a virtual machine fails.
  • Unable to power on a virtual machine.
  • Adding an existing disk (VMDK) to a virtual machine that is already powered on fails with the error:
Failed to add disk scsi0:1. Failed to power on scsi0:1  
  • When Powering on the virtual machine you see one of these errors:

Unable to open Swap File
Unable to access a file since it is locked
Unable to access a file since it is locked
Unable to access Virtual machine configuration

  • In the /var/log/vmkernel log file, you see entries similar to:
WARNING: World: VM xxxx: xxx: Failed to open swap file <path>: Lock was not free  

WARNING: World: VM xxxx: xxx: Failed to initialize swap file <path>  
  • When opening a console to the virtual machine, you receive this error:
Error connecting to <path><virtual machine>, vmx because the VMX is not started  
  • Powering on the virtual machine results in the power on task remaining at 95% indefinitely.
  • Cannot power on the virtual machine after deploying it from a template
  • The virtual machine reports conflicting power states between VMware vCenter Server and the ESXi host console.
  • Attempting to view or open the .vmx file using the cat or vi command reports the error:
cat: can't open '[name of vm].vmx': Invalid argument  

Any one of the above error statements that is output by your ESXi server there is an easy fix for it: The virtual machine files are commonly locked for run time use (i.e. when the host is running the machine) are usually any of the files in the directory. The easiest way to identify the locked file is to attempt to power on the VM and you will see an error message that will give you the full path to the VM's directory plus the locked file. So the work around is as follows:

  1. Login to the ESXi host with SSH client (I use PuTTY).
  2. Confirm that the virtual machine is registered on the server and obtain the full path to the VM. Run the command:
# vmware-cmd -l
  1. The out put returns a list of VM's registered to the host server. Should look something like this:
[<datastore>] <VMDIR>/<VMNAME>.vmx
  1. You need to move to the VM's working directory:
# cd /vmfs/volumes/<datastore>/<VMDIR>
  1. You can use any linux text editor to view the vmware.log file, near the end it will have more verbose output to your error.
  2. The command is really easy to fix the whole unlock/lock issue for the whole directory:
touch *  

You should now be able to start up your VM without issues.

Get Dell Service Tag & Express Service Code From Linux & Windows

First off I wanted to mention that I stole this article from this guy as it is totally worth getting this information out there to the masses who support Dell hardware and have to reach out to Dell Support every now and then. This is an awesome trick for those times when you need to get in touch with Dell's technical support or to get drivers and other documentation from Dell Support. I mainly use the Windows piece but will add the Linux after the jump.

1. Get DELL Service Tag on remote Windows System via RDP/VNC etc.

C:\wmic bios get serialnumber SerialNumber ABCDEF1  

The following command can give you more information such as the make, model, and service tag. C:\wmic csproduct get vendor,name,identifyingnumber IdentifyingNumber Name Vendor ABCDEF1 PowerEdge 2950 Dell Inc.

2. Get DELL Service Tag on remote Windows System

If VNC or Remote Desktop isn't available, you can execute the following command from your local machine to get the service tag of the remote machine.

C:\wmic /user:administrator /note:remote-host bios get serialnumber  
SerialNumber ABCDEF1  

[Note: replace remote-host with the machine name of your remote host.]

3. Get DELL Service Tag on remote Linux System

Login to your Linux system using SSH. Use dmidecode on Linux to get service tag as shown.

[remote-host]# dmidecode -s system-serial-number
ABCDEF1  

4. Get DELL Express Service Code from your Service Tag

Service Tag is a base-36 integer. Once you have your service tag, you can calculate your Express Service Code yourself. Express Service Code is a base-10 integer. Dell mainly uses the Service Code for support based in-call routing. When you call Dell support, it may ask you for your express service code which you can easily enter in your telephone in the dial pad, as it is a bunch of numbers. Use the following tools to find express service code from service tag and vice-versa.

  1. Creativyst Dell-Number Widget
  2. Dell PC service tag/code converter from PowerDog industries

I have also tried this method on some of my other vendor equipment so far I have seen it work on the following: 1. IBM
2. HP Blades
3. HP Servers
4. Supermicro Servers

ESXi VM need to manually change MAC Address

Today, I had to deal with a FlexLM Licensing server issue which resulted from a VM which had it's NIC configured for the SAN's VLAN rather than for the production VLAN which caused the server to have some considerable amounts of downtime. I will talk about the setting up my SAN on a different VLAN from the rest of my production network in another post. In VMWare ESXi vSphere console for the VM host server which my guest VM was hosted on I attempted to change the MAC address of a specific NIC so the license server would work correctly. Well I was able to force this in ESXi in a previous version of ESXi 3.5 which allowed me to use any MAC address that I wanted to use. Well in ESXi 4.1 you can't do that anymore as it forces you to only modify the last 6 digits of the MAC address rather than the whole set. My work around evolved below:

As you notice in the above figure, the MAC Address is a random address given by my ESXi server. I need to bypass this and force a MAC Address from a NIC that I am not using anymore on an old server that I needed to move my FlexLM Licensing from. To accomplish this I went into the NIC's Advanced configuration in the driver settings:

This allowed me to set the MAC Address for the NIC and forces this address over the one set by ESXi.

NOTE: I have confirmed that this setup works with FlexLM on both Windows Server 2003 and Server 2008. I have not yet tested on Windows Server 2012.

Also make note that: You can never fire up that NIC ever again on your old server. Even if you plan to re-purpose it unless you are able to change it's MAC Address manually or install a new NIC as firing up that NIC will cause the above setup to fail miserably as you will have tons of trouble connecting to the new server from the old one.

Why you should change your default passwords

So you bought that shiny new wireless router or your ISP gave you a new one after signing on to a new contract with them. The very first things many people do is personalize the SSIDs on the device so that they can distinguish their wireless network from their neighbor's wireless but forget to overlook a major security concern. In the article presented by Sophos, over 4.5 million DSL subsribers who got new modems from the dominant ISP in Brazil got duped because they didn't bother to change the default password on the device, however Sophos goes further in-depth in the situation by detailing the process the attackers used to change the DNS servers to malicious DNS servers which then points to malicious sites even though the site is legitimate. In the article it details that the DNS record for google in Brazil is google.com.br which is the legitimate domain name for the Brazilian version of Google. The hackers changed the DNS records by simply gaining access to all of the modems and the website looks legitimate when visiting it, however it asked users to download a file which the real Google never does. Moral of the story is to change your network device passwords as soon as you set them up because if they get modified without your knowledge you can be on the hook for something much more serious than simply changing your DNS to malicious DNS. Story from Sophos: source

Update Dell PowerConnect Firmware via SSH and SFTP

You can use PuTTY to telnet into your Dell PowerConnect 62xx switches to configure them and install new firmware. The CLI is much more powerful than the web interface of the 62xx series switches. This setup is not limited to stacking modules but in my example I am using multiple switches in a stack configuration so it only shows up as 1 switch in my configuration panel.

Download the Dell Switch Firmware from http://support.dell.com and enter the service tag of your switch to get the specific firmware for the model.

Download TFTP64server from here: http://tftpd32.jounin.net/tftpd32_download.html

Setup TFTP64server

Create a directory in C:\ root of your computer then make the following directories:

C:\switches\firmware  

Copy the dell firmware files to the firmware directory.
Commands for Telnet on 62xx switches:

Open Putty in Telnet mode to IP address of your Power Connect switch

THIS SYSTEM IS PROPERTY OF XXXXX COMPANY XXXXX INFORMATION TECHNOLOGY DEPARTMENT. UNAUTHORIZED USE OF THIS DEVICE IS NOT PERMITTED UNDER THE TERMS OF THE ACCEPTABLE USE POLICY.Press 'y' to continue (within 30 seconds) (y/n) y  
User:admin  
Password:*********  
SW-ENG>en  
SW-ENG#show version  
Image Descriptions  
image1 : default image  
image2 :

Images currently available on Flash  
--------------------------------------------------------------------
unit image1 image2 current-active next-active  
--------------------------------------------------------------------
1 3.2.0.7 3.2.1.3 image1 image1  
2 3.2.0.7 3.2.1.3 image1 image1  
SW-ENG#copy tftp://<yourTFTPserverIP>/<firmwareName>.stk imageMode...........................................  
TFTPSet TFTP Server IP............................. <yourTFTPserverIP>TFTP Path...................................... ./  
TFTP Filename.................................. <firmwareName>.stk  
Data Type...................................... Code  
Destination Filename........................... image

Management access will be blocked for the duration of the transferAre you sure you want to start? (y/n) y

TFTP code transfer starting9718796 bytes transferredFile reception completeVerifying file...File contents are valid.

Distributing the code to the members of the stack!File transfer operation completed successfully.

SW-ENG#show version

Image Descriptions  
image1 : default image  
image2 :

Images currently available on Flash  
--------------------------------------------------------------------
unit image1 image2 current-active next-active  
--------------------------------------------------------------------
1 3.2.0.7 3.3.4.1 image1 image1  
2 3.2.0.7 3.3.4.1 image1 image1

SW-ENG#boot system image2  
Activating image image2 ..  
SW-ENG#update bootcode  
Update bootcode and reset (Y/N)? Y

Issuing boot code update command...  

After hitting the 'Y' key on the Update bootcode and reset, the switch should be rebooted into the latest firmware. To confirm you can show version

Renewing SSL Certificates for Exchange

While there are a number of great guides designed to teach a budding Exchange Administrator how to add a brand new SSL certificate to your Exchange system, there seems to be a severe lack of guides to renew said SSL cert. As most of us administrators will spend more time renewing these pesky annual certificates rather than doing the first install I figured I needed to save this little gem so I can easily refer back to it year after year...

Some items of note before we get started. I've only tested this on my configuration, your systems may vary. This was tested on a fairly standard Exchange 2007 SP1 system running on Windows 2003 R2. Also this is not for the self-signed internal Exchange certificate (that process can be found here), this process is for your external facing SSL certificate (which I would assume is a purchased valid certificate from a 3rd Party root CA such as Verisign, Thwate, or even say GoDaddy if you are seeking a discount wildcard SSL cert).

So without further delay away we go, Administrators familiar with say IIS based renewals will likely be tripped up by the very first step, to begin the renewal process in Exchange 2007 you actually don't do a "renewal" per-se, instead you'll want to issue a new certificate request - there are two options here. Both of which are launched from the Exchange Management Shell:

Option 1:

Issue a SAN (Subject Alternative Names) Certificate - also called a wildcard certificate. These certs are very helpful for exchange environments as it can be used to replace the self-signed Exchange certificate as well as work for the multiple exchange sites such as autodiscover, owa.company.com, mail.company.com, etc.

New-Exchangecertificate -domainname mail.company.com, company.com, company.local, autodiscover.company.com, autodiscover.company.local, servername01.company.local, servername01 -Friendlyname companyExchCrt -generaterequest:$true -keysize 1024 -path c:\cerReq.csr -privatekeyexportable:$true -subjectname "c=US, s=Texas, l=Dallas, o=Company Name, ou=IT, cn=mail.company.com"  

Option 2:

Issue a Standard single domain certificate - useful for OWA only or environments that want to keep the self-signed exchange certificate - also MUCH cheaper.

New-ExchangeCertificate -GenerateRequest:$true -keysize 1024 -path c:\cerReq.csr -privatekeyexportable:$true -SubjectName "c=US, s=Texas, l=Dallas, o=Company Name, ou=IT, cn=mail.company.com"  

Under SubjectName you'll want to fill your values in appropriately.

c= Your 2 digit Country  
s= Your State (I tend to spell it out, though some just use the standard 2 character state code) l= Your Locality (City, Region, it's up to you)  
o= Your company name  
ou= Your Organizational Unit (common here is IT/IS Department or just omit if not required)  
cn= the common name of the certificate.  

This is important as it is the name the certificate will be issued on - if you are using this Certificate for OWA or even secure SMTP/POP access you will want this to match your MX DNS records (such as mail.company.com or owa.company.com - not as critical with a SAN cert).

Now the fun begins, you go to your certificate provider and go though the renewal process, when prompted attach or copy the text from your CSR file (c:\cerReq.csr if you followed the example above) and wait for your new certificate to be emailed back to you (or retrieved from the company's website).

Save this new certificate in a file called exchcert.cer on the root of your c:\

Note: if you get it from a web page you may need to copy everything from

-----BEGIN CERTIFICATE-----
          to   
-----END CERTIFICATE-----

into a text file (using notepad) and save it as c:\exchcert.cer

Open the Exchange Management Shell Issue the following command

cGet-ExchangeCertificate | fl | out-file -filePath c:\CurrentCerts.txt  

This will export all the certificates on the server to a text file, when opened it looks like this:

AccessRules : {System.Security.AccessControl.CryptoKeyAccessRule, System.Security.AccessControl.CryptoKeyAccessRule, System.Security.AccessControl.CryptoKeyAccessRule}  
CertificateDomains : {mail.company.com}  
HasPrivateKey : True  
IsSelfSigned : False  
Issuer : OU=[www.verisign.com/CPS Incorp.by](http://www.verisign.com/CPS%20Incorp.by) Ref. LIABILITY LTD.(c)97 VeriSign, OU=VeriSign International Server CA - Class 3,  
OU="VeriSign, Inc.", O=VeriSign Trust Network  
NotAfter : 4/18/2009 6:59:59 PM  
NotBefore : 3/26/2008 7:00:00 PM  
PublicKeySize : 1024  
RootCAType : ThirdParty  
SerialNumber : 29BBFFD5AAAB541AB9BB2A73139AG335  
Services : IMAP, POP, IIS, SMTP  
Status : Valid  
Subject : CN=mail.company.com, OU=Terms of use at verisign.com/rpa (c)05, OU=IT, o=company name, L=Dallas, S=Texas, C=US  
Thumbprint : C21DE04E3123210AD28E430742C21D0194F46421  

Two things to note here,

Services: - these two are important to the next step - depending on what you use the current certificate for you will want to make sure the renewal cert does the same thing (in the example above this certificate does IMAP, POP, IIS, and SMTP)

Thumbprint: - this is used to delete the old certificate which is our next step.

Remove-ExchangeCertificate -thumbprint [old certificate thumbprint]  

from the example above:

Remove-ExchangeCertificate -thumbprint C21DE04E3123210AD28E430742C21D0194F46421  

You will get a confirmation to remove the old certificate, you'll want to type Y to remove it. Now you are almost done, we just need to add the newly renewed certificate back into the system.

Import-ExchangeCertificate -path c:\exchcert.cer | Enable-ExchangeCertificate -Services [your services from above]  

Using our example you will want to do the following to get the certificate working on all services (except Unified Messaging - UM).

Import-ExchangeCertificate -path c:\exchcert.cer | Enable-ExchangeCertificate -Services IMAP, POP, IIS, SMTP  

Now just to confirm all is working as expected, you will want to issue the following command

Get-ExchangeCertificate  

This will display something similar to this:

Thumbprint Services Subject ---------- -------- ------- C32DE24E1214210AD28E030742C21D0194F060D1 IP.WS CN=mail.company.com  

Dell PowerConnect Switches Packet Loss Issues

  I upgraded many Dell 6248 switches to the latest and greatest firmware over the weekend.  I have been experiencing more packet loss than I would ever want to see on an internal LAN.  Connecting from the servers on the same switch would yield a periodic lost packet for no apparent reason.  There were some clues on the switch with logged “spanning tree topology changes” in the log file.  During this log event, I would drop packets not only on the local switch, but other connecting switches as well.  All of these switches are configured with Rapid STP, LAG groups between them, and two VLANs. Reading up on the dell site, I saw some good advice entailing turning on “Port Fast” on every port that isn’t an edge link between switches, namely ports connected to switches and servers.  This advice appears to be valid.  With the latest firmware I could go to Global STP settings and simply enable Port Fast.  It was smart enough to not turn it on for the LAG groups and switch interconnects with multi-vlans on them.  So far so good…over the past few hours I haven’t had any dropped packets.