Monday, November 11, 2013

Integrating an automated iOS build with a Windows Team Foundation Server build environment

My current job is a "Microsoft shop" - our source control is Microsoft's Team Foundation Server, we primarily write applications for windows using C++ and C#, and all the developers use windows workstation PC's.

We'd like to develop an iOS app, but we'd like to set it up with a repeatable, automatable build process. Simply asking the guy with the mac on his desk to compile a new build and hope it works doesn't cut it.
All our existing build processes are using Team Foundation Server's build agents and build scripts, which have some very nice properties, and we'd like to be able to queue and publish iOS builds in the same way.

The problem is that we must build iOS apps on a Mac using Xcode, but the Team Foundation Server build agent only runs on Windows

How do we bridge this gap? Using SSH from a windows build agent to a remote mac.

  • The TFS build script uses putty (specifically the command line plink application) to tell the mac to remotely do things.
  • We queue a TFS build onto one of our windows TFS build agent PC's. It basically just runs plink, captures the output and waits for it to exit.
  • We get the files from the TFS repository onto the mac via SCP - the windows TFS build agent checks everything out, then runs pscp to copy it all over to the mac
  • The mac then uses xcodebuild and xcrun to compile and package the iOS app for ad-hoc distribution
  • The windows build agent then uses pscp to copy the packaged iOS app off the mac back onto the windows build agent, which then copies it into the appropriate drop folder alongside all the other windows builds.
  • We host a basic web server to serve the iOS application's .ipa package over HTTP, and everyone installs/upgrades our app by accessing this webpage using iOS' over the air beta testing facilities.

There are a lot of pieces to this, particularly given that we do not want to put secure data such as user account passwords in source control or have them hardcoded into build scripts. Here are the details, in the order that (for me) makes the most sense.

Setting up the Mac


Install Xcode

Obviously we need to install Xcode on the mac. The easiest way is just via the Mac app store, but you can also download a DMG from the apple developer centre for offline installs.

Install Team Explorer Everywhere

I did this by unzipping microsoft's zip file into /usr/local/bin. There is likely a better/more modular way to manage this than putting everything directly in /usr/local/bin, but this works fine on my isolated build machine.

User account

In the interests of keeping the build environment clean, and repeatable (so we can set up other mac build machines easily in the future), we want all of the builds to be performed by a dedicated user account. We do not ever want to use this account to do normal day to day development. Ideally we should never log into it interactively at all, only via SSH. This helps ensure that our builds really are clean and repeatable, and that we don't end up with accidental dependencies on random files/data in a normal person's user account.

Enabling passwordless SSH logins

We need the windows build agent PC to SSH into the mac, but we don't want the TFS build scripts to have the password of the mac's build account, we want to enable passwordless logins. To do this, I followed the instructions here: http://www.tonido.com/blog/index.php/2009/02/20/ssh-without-password-using-putty/

Once this has been set up, on the mac the /Users/buildaccount/.ssh/authorized_keys file will contain the details of the ssh key, and on the windows PC there will be the matching .ppk file. We can use plink to get the mac to run various scripts, as follows

plink -ssh -batch -l buildaccount -i buildaccountkey.ppk mac01.domain.local /Users/buildaccount/build_script.sh

Enabling our non-admin account to login using SSH

Apple's default SSH security settings are such that only users with Admin rights can login via SSH (At least this is the case on OSX 10.9, which I am using). We want our build account to be a non-admin user, so we'll have to edit the access control list that controls who can access ssh, and add our build user to it.

sudo dseditgroup -o edit -n . -a buildaccount -t user com.apple.access_ssh

Setting up the build process


Certificates and Provisioning Profiles

As we're setting up for Ad-hoc deployment of our app, we need an iOS Distribution Certificate, and an Ad-hoc provisioning profile. You generate both of these via the Apple Developer website's Member Centre. When building through the Xcode UI, it handles most of this for you, but as we want to be building via the command line, and via SSH into a clean user account, we have a few more hoops to jump through.

Putting the certificates in a custom keychain

By default Xcode and it's command line tools will look for your distribution certificate in the user default keychain (the login keychain). As we want our build account to be "clean", we don't want to have to put them in the build account's login keychain. To resolve this, we can create a custom keychain, and put the certificates/provisioning profile keys in there.

We can then decide to either check this custom keychain file into source control, or put it in some other special location.

I'm assuming that you've already created and loaded/installed a distribution certificate, and an ad-hoc provisioning profile.

  1. From your normal user account, run Keychain Access
  2. Right-click on the list of keychains in the top-left, and create a new keychain. Give it a meaningful filename and store it somewhere you can find later.
  3. From your login keychain, hold the option key (to copy) and drag your iPhone Distribution certificate, and the Ad-hoc distribution private key.

Using this custom keychain in the build script

To get the Xcode command line tools to use this custom keychain file, add the following before you invoke xcodebuild

keychain_file="$SCRIPTPATH/iOSCustomBuild.keychain"
keychain_pass=secretpassword
security list-keychains -s "$keychain_file" ~/Library/Keychains/login.keychain  # put the iOS keychain in ahead of the login keychain
security unlock-keychain -p "$keychain_pass" "$keychain_file"

The security list-keychains -s command sets the search order, telling the system first to look in our custom keychain, followed by the default login keychain. This command is persistent - I haven't worried about resetting it after our script completes, but you may want to

The security unlock-keychain command is required because keychains are locked by default when code is run in a remote SSH session. We must issue security unlock-keychain or else xcodebuild will not be able to read the keychain. This is another reason to use a custom keychain file - if we were using the user's login keychain, we would be required to hard code the user account's password into our build scripts.

Compiling and packaging the app

To build our iOS app, we'll need to run xcodebuild, which is xcode's command line build tool. This requires a number of flags, which we will create variables for and then re-use. Here's mine:

# hack to get the full path to the current directory
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd`
popd > /dev/null

application_name=MyiOSApp # our app name - should correspond to MyiOSApp.xcodeproj
sdk="iphoneos7.0" # which version of iOS are we targeting

# Name of the distribution certificate. Get this by downloading the certificate from 
# the apple developer center, installing it, and then viewing it's name in keychain
codesign="iPhone Distribution: My Name (AAA12345ZZ)"

# build the project (must sign as xcode requires signing for all non-simulator bulds)
# CONFIGURATION_BUILD_DIR is only required for cordova/phonegap projects.
xcodebuild -project "$SCRIPTPATH/$application_name.xcodeproj" -target $application_name -configuration Release -sdk $sdk clean build CODE_SIGN_IDENTITY="$codesign" CONFIGURATION_BUILD_DIR="$SCRIPTPATH/build"

This only gets us halfway - xcodebuild will produce a .app package (a directory). This doesn't have the provisioning profile embedded in it, and we also need to package it up as a .ipa file to distribute it via the ad-hoc mechanisms. To do this step, we'll need to run xcrun. Here's my part of the script to do that:

# Get this by downloading it from the apple developer center
provisioning_profile_file="$SCRIPTPATH/My_Ad_Hoc_Provisioning_Profile.mobileprovision"
provisioning_profile_name="My Ad Hoc Provisioning Profile" # get this from the apple developer center when you download the file

# install the provisioning profile so xcode can use it 
provisioning_profile_uuid=`grep UUID -A1 -a "$provisioning_profile_file" | grep -o "[-A-Z0-9]\{36\}"`
cp "$provisioning_profile_file" ~/Library/MobileDevice/Provisioning\ Profiles/$provisioning_profile_uuid.mobileprovision
 
# this is where xcode will drop it's output, as a .app package
build_dir="$SCRIPTPATH/build"

# this is where we want the created .ipa file to be put
drop_dir="$SCRIPTPATH"

# package it and embed the provisioning profile (must re-sign as packaging alters the app)
/usr/bin/xcrun -sdk $sdk PackageApplication -v "${build_dir}/${application_name}.app" -o "${drop_dir}/${application_name}.ipa" --sign "$codesign" --embed "$provisioning_profile_file"

I chose to simply check my provisioning profile file into source control.

Also note that we have the extra step of "installing" the provisioning profile. This is because we're trying to run under a clean account. If we were running as a normal user account where someone had manually run Xcode, Xcode would have put the provisioning profile into the ~/Library/MobileDevice/Provisioning Profiles directory already for us and we wouldn't need to do this step.

The full build script


keychain_file="$SCRIPTPATH/iOSCustomBuild.keychain"
keychain_pass=secretpassword
security list-keychains -s "$keychain_file" ~/Library/Keychains/login.keychain  # put the iOS keychain in ahead of the login keychain
security unlock-keychain -p "$keychain_pass" "$keychain_file"

# hack to get the full path to the current directory
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd`
popd > /dev/null

application_name=MyiOSApp # our app name - should correspond to MyiOSApp.xcodeproj
sdk="iphoneos7.0" # which version of iOS are we targeting

# Name of the distribution certificate. Get this by downloading the certificate from 
# the apple developer center, installing it, and then viewing it's name in keychain
codesign="iPhone Distribution: My Name (AAA12345ZZ)"

# build the project (must sign as xcode requires signing for all non-simulator bulds)
# CONFIGURATION_BUILD_DIR is only required for cordova/phonegap projects.
xcodebuild -project "$SCRIPTPATH/$application_name.xcodeproj" -target $application_name -configuration Release -sdk $sdk clean build CODE_SIGN_IDENTITY="$codesign" CONFIGURATION_BUILD_DIR="$SCRIPTPATH/build"

# Get this by downloading it from the apple developer center
provisioning_profile_file="$SCRIPTPATH/My_Ad_Hoc_Provisioning_Profile.mobileprovision"
provisioning_profile_name="My Ad Hoc Provisioning Profile" # get this from the apple developer center when you download the file

# install the provisioning profile so xcode can use it 
provisioning_profile_uuid=`grep UUID -A1 -a "$provisioning_profile_file" | grep -o "[-A-Z0-9]\{36\}"`
cp "$provisioning_profile_file" ~/Library/MobileDevice/Provisioning\ Profiles/$provisioning_profile_uuid.mobileprovision
 
# this is where xcode will drop it's output, as a .app package
build_dir="$SCRIPTPATH/build"

# this is where we want the created .ipa file to be put
drop_dir="$SCRIPTPATH"

# package it and embed the provisioning profile (must re-sign as packaging alters the app)
/usr/bin/xcrun -sdk $sdk PackageApplication -v "${build_dir}/${application_name}.app" -o "${drop_dir}/${application_name}.ipa" --sign "$codesign" --embed "$provisioning_profile_file"

Putting it all together

All we need to do now is create TFS build script which Exec's plink and pscp and asks the mac to run the various .sh scripts which do all the work. Here's some snippets which you could use


<PropertyGroup>
    <!-- general -->
    <IpaFileName>MyiOSApp.ipa</IpaFileName>
    
    <!--File locations on the windows build machine-->
    <Plink>pathtoPLINK.EXE</Plink>
    <Pscp>pathtoPSCP.EXE</Pscp>
    
    <!--SSH connection info-->
    <SshHost>mac.domain.local</SshHost>
    <SshKey>buildaccountkey.ppk</SshKey>
    <SshUser>buildaccount</SshUser>

    <!--File locations on the mac-->
    <RemoteiOSBuildPath>/Users/buildaccount/iOSBuild</RemoteiOSBuildPath>
    <RemoteBuildScriptPath>$(RemoteiOSBuildPath)/build.sh</RemoteBuildScriptPath>
    <RemoteIpaFilePath>$(RemoteiOSBuildPath)/$(IpaFileName)</RemoteIpaFilePath>
</PropertyGroup>

...

<!--this "echo y | exit" causes plink/putty to cache the remote host key in the registry for subsequent operations.-->
<Exec command="echo y | &quot;$(Plink)&quot; -ssh -l $(SshUser) -i $(SshKey) $(SshHost) exit"/>

<!--Wipe any old files-->
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) rm -rf $(RemoteiOSBuildPath)" />
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) mkdir -p $(RemoteiOSBuildPath)" />

<!-- copy the source over to the mac -->
<Exec command="&quot;$(Pscp)&quot; -sftp -batch -r -i $(SshKey) -l $(SshUser) &quot;$(SolutionRoot)*&quot; $(SshHost):$(RemoteiOSBuildPath)" />

<!--run the build script-->
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) chmod +x $(RemoteBuildScriptPath)"/>
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) $(RemoteBuildScriptPath)"/>

<!-- copy the packaged ipa file back -->
<Exec command="&quot;$(Pscp)&quot; -sftp -batch -i $(SshKey) -l $(SshUser) $(SshHost):$(RemoteIpaFilePath) ." />

Wednesday, October 02, 2013

Programmatically controlling Hyper-V Server 2012 R2 virtual machines from C#

Recently I've wanted to remote control some virtual machines on a Hyper-V server in aid of automated testing.
I've got a server set up running Hyper-V Server 2012 R2, so the code samples in this are likely to work against Server 2012 non-R2, but may not work against Server 2008.

All the virtual machines start off at a known clean snapshot, and have a startup script which runs on boot that goes and asks a central database for a testing job to execute. So, for our system to work, we need to accomplish the following:
  1. Roll back a VM to the last known snapshot / checkpoint
  2. Power it on
There are several other things we need to do for this to work at all, which are:
  1. Connect to the Hyper-V server (this includes authentication)
  2. List out or otherwise ask the server about it's VM's so that we can rollback the correct one

To achieve thisk I'm using Hyper-V's WMI provider (V2). I don't really know much about WMI, so these code samples are just things I've hacked together. They are NOT production quality and they don't handle errors well or anything else. Please use them as guidance, not for copy/pasting.

Pre-requisites

Using WMI from C# is done by the classes in the System.Management namespace. To access this you'll want using System.Management at the top of your C# file, as well as a reference to the System.Management assembly.

Connecting and Logging on


To do this we need to create a ManagementScope object set to use the appropriate server and using the Hyper-V virtualization v2 namespace, and we also need to supply the username and password of a windows account that has privileges to administer the Hyper-V server. I did it like this:

var connectionOptions = new ConnectionOptions(
    @"en-US",
    @"domain\user",
    @"password",
    null,
    ImpersonationLevel.Impersonate, // I don't know if this is correct, but it worked for me
    AuthenticationLevel.Default,
    false,
    null,
    TimeSpan.FromSeconds(5);

var scope = new ManagementScope(new ManagementPath { 
    Server = "hostnameOrIpAddress", 
    NamespacePath = @"root\virtualization\v2" }, connectionOptions);
scope.Connect();

Note: Most of the other documentation or sample code I found refers to the root\virtualization namespace. This didn't work at all for me in Server 2012 R2, and I had to use dotPeek to decompile the Hyper-V powershell commandlets to figure out to put \v2 on the end. Perhaps the non-v2 one is for Server 2008?

Note: If your PC and the target server are on the same domain, and your windows user account has privileges to administer the remote server, You don't need the ConnectionOptions object at all. WMI will use windows authentication, and it will magically work:

Listing out the VM's and finding the one we want

Virtual machines in Hyper-V get exposed via the Msvm_ComputerSystem WMI class. In order to list them out, we simply ask WMI to give us all the objects of that class in the virtualization namespace. To do this, I'm going to create two helper extension methods, that we'll use from hereon:

public static class WmiExtensionMethods
{
    public static ManagementObject GetObject(this ManagementScope scope, string serviceName)
    {
        return GetObjects(scope, serviceName).FirstOrDefault();
    }
    
    public static IEnumerable<ManagementObject> GetObjects(this ManagementScope scope, string serviceName)
    {
        return new ManagementClass(scope, new ManagementPath(serviceName), null)
            .GetInstances()
            .OfType<ManagementObject>();
    }
}

It turns out that the Host PC is also exposed via Msvm_ComputerSystem, so we need to filter out things that are not virtual machines. This helper method will return a list of all the virtual machines. Note: I like extension methods, so I'm using them a lot here:

public static IEnumerable<ManagementObject> GetVirtualMachines(this ManagementScope scope)
{
    return scope.GetObjects("Msvm_ComputerSystem").Where(x => "Virtual Machine" == (string)x["Caption"]);
}

You can use it to get a specific virtual machine as follows:

var vm = scope.GetVirtualMachines().First(vm => vm["ElementName"] as string == "myvmname");

Rolling the VM back to it's latest snapshot / checkpoint

In order to roll a Hyper-V VM back to a snapshot, we need to get a reference to the snapshot object itself.

There are ways to simply list all the snapshots for each VM, but there is also a special Msvm_MostCurrentSnapshotInBranch class which represents the "latest" snapshot. We can use it to create a helper method as follows:

public static ManagementObject GetLastSnapshot(this ManagementObject virtualMachine)
{
    return virtualMachine.GetRelated(
        "Msvm_VirtualSystemSettingData",
        "Msvm_MostCurrentSnapshotInBranch",
        null,
        null,
        "Dependent",
        "Antecedent",
        false,
        null).OfType<ManagementObject>().FirstOrDefault();
}

And use it like this:

var snapshot = vm.GetLastSnapshot();

Now, to actually roll the VM back, we need to call the ApplySnapshot method on the Msvm_VirtualSystemSnapshotService class.
Note: Logically I thought that the snapshot methods should be on the Virtual Machine object, but Hyper-V puts them all in their own service for some reason. There is also only one global Snapshot service - not a service per VM. I've no idea why they've designed it this way.

We can create a helper method:

public static uint ApplySnapshot(this ManagementScope scope, ManagementObject snapshot)
{
    var snapshotService = scope.GetObject("Msvm_VirtualSystemSnapshotService");

    var inParameters = snapshotService.GetMethodParameters("ApplySnapshot");
    inParameters["Snapshot"] = snapshot.Path.Path;
    var outParameters = snapshotService.InvokeMethod("ApplySnapshot", inParameters, null);
    return (uint)outParameters["ReturnValue"];
}

And use it like this:

scope.ApplySnapshot(snapshot);

When I execute this with valid parameters, the VM applied it's snapshot, and I always got a return value of 4096. According to the documentation, this indicates that there's a Job in progress to asynchronously track the actual snapshot applying and determine the final success or failure. We could fetch the job out of outParameters["Job"] and use it to determine when the apply completes, but I'm not going to worry about that here. Refer to the ApplySnapshot MSDN page for other possible return codes.

Note: If the VM is not powered off, you are likely to find the ApplySnapshot call fails. You must power the VM off (and wait for the Power Off job to complete) first.

Powering on the VM to boot it up


Turning on the VM is done by the RequestStateChange method on the Msvm_ComputerSystem class. We already have the Msvm_ComputerSystem object representing the virtual machine we found earlier, so we can create a helper method and enumeration like this:

public enum VmRequestedState : ushort
{
    Other = 1,
    Running = 2,
    Off = 3,
    Saved = 6,
    Paused = 9,
    Starting = 10,
    Reset = 11,
    Saving = 32773,
    Pausing = 32776,
    Resuming = 32777,
    FastSaved = 32779,
    FastSaving = 32780,
}

public static uint RequestStateChange(this ManagementObject virtualMachine, VmRequestedState targetState)
{
    var managementService = virtualMachine.Scope.GetObject("Msvm_VirtualSystemManagementService");

    var inParameters = managementService.GetMethodParameters("RequestStateChange");
    inParameters["RequestedState"] = (object)targetState;
    var outParameters = virtualMachine.InvokeMethod("RequestStateChange", inParameters, null);
    return (uint)outParameters["ReturnValue"];
}

And use it like this:

vm.RequestStateChange(VmRequestedState.Running);

As for ApplySnapshot, when things work, the return value is usually 4096, indicating there is a Job to asynchronously track the progress of the operation.

Note: Although we invoke the method on the Msvm_ComputerSystem object, we need to get the method parameters by asking the Msvm_VirtualSystemManagementService object (which represents the host server) instead. I've no idea why.

Final fixups


ApplySnapshot will fail if the virtual machine is running. To turn it off, we can simply call vm.RequestStateChange(VmRequestedState.Off); and wait a bit for Job to complete.

RequestStateChange will fail if the state doesn't make sense. For example, if you try and turn the vm Off when it's already Off, the method will fail. You can check the current state of a Vm by reading it's EnabledState property. Valid values for EnabledState are documented with the Msvm_ComputerSystem class. I created an enum, and used it as follows:

public enum VmState : ushort
{
    Unknown = 0,
    Other,
    Running, // Enabled
    Off, // Disabled
    ShuttingDown,
    NotApplicable,
    OnButOffline,
    InTest,
    Deferred,
    Quiesce,
    Starting
}

if ((VmState)vm["EnabledState"] != VmState.Off)
{
    vm.RequestStateChange(VmRequestedState.Off); // needs to be off to apply snapshot
    Thread.Sleep(2000); // todo wait for the state change properly
}

You can do many more things by using other methods and classes from the Hyper-V WMI API. Hopefully this gives you a decent starting point.

P.S. The Hyper-V powershell commandlets are all implemented on top of this WMI Api. Using a tool like .NET reflector or dotPeek is an interesting way to see how Microsoft calls the WMI API.