Wednesday, March 25, 2015

Using ReactiveCocoa from Swift

We're trying out swift at work, and have an existing app which makes quite a bit of use of ReactiveCocoa.
Looking online, I found MVVM, Swift and ReactiveCocoa - It's all good! from Colin Eberhardt. It's a good intro, but I found I had trouble when trying to use the tricker subscribe overloads.

Without further ado, here's my own take on the extensions.
Licensing wise I declare it to be public domain, do with it as you will.
(Note: Implemented on Xcode 6.2 with Swift 1.1)


import Foundation

extension RACSignal {

    // Subscribe with an on-next block
    // Equivalent of - (RACDisposable *)subscribeNext:(void (^)(id x))nextBlock {
    func subscribe(next nextClosure:(T)->Void) -> RACDisposable {
        return self.subscribeNext { (x: AnyObject!) -> () in nextClosure(x as T) }
    }
    
    // Subscribe with an on-next and on-completed block
    // Equivalent of - (RACDisposable *)subscribeNext:(void (^)(id x))nextBlock completed:(void (^)(void))completedBlock;
    func subscribe(next nextClosure:(T)->Void, completed completedClosure:()->Void) -> RACDisposable {
        return self.subscribeNext(
            { (x: AnyObject!) -> () in nextClosure(x as T) },
            completed:{ () in completedClosure() })
    }
    
    // Subscribe with an on-next and on-error block
    // Equivalent of - (RACDisposable *)subscribeNext:(void (^)(id x))nextBlock error:(void (^)(NSError *error))errorBlock {
    func subscribe(next nextClosure:(T)->Void, error errorClosure:(NSError)->Void) -> RACDisposable {
        return self.subscribeNext(
            { (x: AnyObject!) -> () in nextClosure(x as T) },
            error:{ (x:NSError!) in errorClosure(x) })
    }
    
    // Subscribe with an on-next, on-error and on-completed block
    // Equivalent of - (RACDisposable *)subscribeNext:(void (^)(id x))nextBlock error:(void (^)(NSError *error))errorBlock completed:(void (^)(void))completedBlock;
    func subscribe(next nextClosure:(T)->Void, error errorClosure:(NSError)->Void, completed completedClosure:()->Void) -> RACDisposable {
        return self.subscribeNext(
            { (x: AnyObject!) -> () in nextClosure(x as T) },
            error:{ (x:NSError!) in errorClosure(x) },
            completed:{ () in completedClosure() })
    }
    
    // Subscribe with an on-error block
    // Equivalent of - (RACDisposable *)subscribeError:(void (^)(NSError *error))errorBlock;
    func subscribe(error errorClosure:(NSError)->Void) -> RACDisposable {
        return self.subscribeError { (x: NSError!) -> () in errorClosure(x) }
    }
    
    // Subscribe with an on-error and on-completed block
    // Equivalent of - (RACDisposable *)subscribeError:(void (^)(NSError *error))errorBlock completed:(void (^)(void))completedBlock;
    func subscribe(error errorClosure:(NSError)->Void, completed completedClosure:()->Void) -> RACDisposable {
        return self.subscribeError(
            { (x: NSError!) -> () in errorClosure(x) },
            completed:{ () in completedClosure() })
    }
}

Now, how to use it (this can be a bit unclear sometimes in swift)

To subscribe for on-next notifications


someSignal.subscribe { (arg:Type) in .... }

e.g. with an RACSignal publishing strings it would be
someSignal.subscribe { (x:String) in doSomething(x) }
or
someSignal.subscribe(
    next:{ (x:String) in 
        doSomething(x) 
    })

To subscribe for on-error notifications


someSignal.subscribe(error:{ (e) in .... })
or
someSignal.subscribe(
    error:{ (e) in 
        logError(e) 
    })


e.g. with an RACSignal publishing strings it would be
someSignal.subscribe(error:{ (e) in logError(e) }

You don't need to specify (error:NSError) as the compiler is smart enough to infer that, however you do need to specify the explicit error: parameter name, so the compiler knows this is an error handler, not an on-next handler

To subscribe for on-next and on-completed notifications


someSignal.subscribe({ (arg:Type) in .... }, completed:{ ... })

e.g. with an RACSignal publishing strings it would be
someSignal.subscribe({ (x:String) in doSomething(x) }, completed:{ allDone() })
or
someSignal.subscribe(
    next:{ (x:String) in 
        doSomething(x) 
    }, 
    completed:{ 
       allDone() 
    })


Warning If the code in your completed block (e.g. the allDone function) doesn't return void, the compiler will infer it as the return type of the completed block, and you'll get a confusing-looking Extra argument 'completed' in call error. The fix is to insert an explicit return, e.g.

someSignal.subscribe(
    next:{ (x:String) in 
        doSomething(x) 
    }, 
    completed:{ 
       allDone() 
       return
    })


To subscribe for on-next and on-error notifications


someSignal.subscribe({ (arg:Type) in .... }, error:{ (e) in ... })

e.g. with an RACSignal publishing strings it would be
someSignal.subscribe({ (x:String) in doSomething(x) }, error:{ (e) in logError(e) })
or
someSignal.subscribe(
    next:{ (x:String) in 
        doSomething(x) 
    }, 
    error:{ (e) in
       logError(e) 
    })


To subscribe for all 3 - on-next, on-error and on-completed notifications


someSignal.subscribe({ (arg:Type) in .... }, error:{ (e) in ... }, completed:{ ... })

e.g. with an RACSignal publishing strings it would be
someSignal.subscribe({ (x:String) in doSomething(x) }, error:{ (e) in logError(e) }, completed:{ allDone() })
or
someSignal.subscribe(
    next:{ (x:String) in 
        doSomething(x) 
    }, 
    error:{ (e) in
       logError(e) 
    },
    completed:{
        allDone()
    })


The warning above (sometimes you might need an explicit return in the completed block) applies here too.

To subscribe for on-error and on-completed notifications


someSignal.subscribe(error:{ (e) in ... }, completed:{ ... })

e.g. with an RACSignal publishing strings it would be
someSignal.subscribe({ error:{ (e) in logError(e) }, completed:{ allDone() })
or
someSignal.subscribe(
    error:{ (e) in
       logError(e) 
    },
    completed: {
        allDone()
    })


The warning above (sometimes you might need an explicit return in the completed block) applies here too.

Wrap-up

I hope this helps!

P.S. - if you're wondering where the "on-completed only" version is - there isn't one, because ReactiveCocoa doesn't seem to provide one for some reason. If you want that behaviour, you can subscribe with an empty on-next or on-error callback.

Monday, May 05, 2014

Codemania 2014 Presentation

I gave a presentation at Codemania 2014 titled "Understanding C++ Templates".

The abstract is as follows (posted here for posterity, as codemania will take all the 2014 details down when the 2015 conference ramps up)

Understanding C++ templates

Templates are a key feature of C++. They enable you to write safer code with less duplication and better performance. Almost all C++ programmers have encountered them, but few know them well. Orion will show you how to think about and put to use some of the more advanced things you can do with templates, and show that they're simpler than you might think.

Even if you're not a C++ developer, understanding these concepts will give you a better perspective for programming in other languages.

The source code/examples I used in the presentation are available here on Github

The slides are available on here on iCloud

Monday, November 11, 2013

Integrating an automated iOS build with a Windows Team Foundation Server build environment

My current job is a "Microsoft shop" - our source control is Microsoft's Team Foundation Server, we primarily write applications for windows using C++ and C#, and all the developers use windows workstation PC's.

We'd like to develop an iOS app, but we'd like to set it up with a repeatable, automatable build process. Simply asking the guy with the mac on his desk to compile a new build and hope it works doesn't cut it.
All our existing build processes are using Team Foundation Server's build agents and build scripts, which have some very nice properties, and we'd like to be able to queue and publish iOS builds in the same way.

The problem is that we must build iOS apps on a Mac using Xcode, but the Team Foundation Server build agent only runs on Windows

How do we bridge this gap? Using SSH from a windows build agent to a remote mac.

  • The TFS build script uses putty (specifically the command line plink application) to tell the mac to remotely do things.
  • We queue a TFS build onto one of our windows TFS build agent PC's. It basically just runs plink, captures the output and waits for it to exit.
  • We get the files from the TFS repository onto the mac via SCP - the windows TFS build agent checks everything out, then runs pscp to copy it all over to the mac
  • The mac then uses xcodebuild and xcrun to compile and package the iOS app for ad-hoc distribution
  • The windows build agent then uses pscp to copy the packaged iOS app off the mac back onto the windows build agent, which then copies it into the appropriate drop folder alongside all the other windows builds.
  • We host a basic web server to serve the iOS application's .ipa package over HTTP, and everyone installs/upgrades our app by accessing this webpage using iOS' over the air beta testing facilities.

There are a lot of pieces to this, particularly given that we do not want to put secure data such as user account passwords in source control or have them hardcoded into build scripts. Here are the details, in the order that (for me) makes the most sense.

Setting up the Mac


Install Xcode

Obviously we need to install Xcode on the mac. The easiest way is just via the Mac app store, but you can also download a DMG from the apple developer centre for offline installs.

Install Team Explorer Everywhere

I did this by unzipping microsoft's zip file into /usr/local/bin. There is likely a better/more modular way to manage this than putting everything directly in /usr/local/bin, but this works fine on my isolated build machine.

User account

In the interests of keeping the build environment clean, and repeatable (so we can set up other mac build machines easily in the future), we want all of the builds to be performed by a dedicated user account. We do not ever want to use this account to do normal day to day development. Ideally we should never log into it interactively at all, only via SSH. This helps ensure that our builds really are clean and repeatable, and that we don't end up with accidental dependencies on random files/data in a normal person's user account.

Enabling passwordless SSH logins

We need the windows build agent PC to SSH into the mac, but we don't want the TFS build scripts to have the password of the mac's build account, we want to enable passwordless logins. To do this, I followed the instructions here: http://www.tonido.com/blog/index.php/2009/02/20/ssh-without-password-using-putty/

Once this has been set up, on the mac the /Users/buildaccount/.ssh/authorized_keys file will contain the details of the ssh key, and on the windows PC there will be the matching .ppk file. We can use plink to get the mac to run various scripts, as follows

plink -ssh -batch -l buildaccount -i buildaccountkey.ppk mac01.domain.local /Users/buildaccount/build_script.sh

Enabling our non-admin account to login using SSH

Apple's default SSH security settings are such that only users with Admin rights can login via SSH (At least this is the case on OSX 10.9, which I am using). We want our build account to be a non-admin user, so we'll have to edit the access control list that controls who can access ssh, and add our build user to it.

sudo dseditgroup -o edit -n . -a buildaccount -t user com.apple.access_ssh

Setting up the build process


Certificates and Provisioning Profiles

As we're setting up for Ad-hoc deployment of our app, we need an iOS Distribution Certificate, and an Ad-hoc provisioning profile. You generate both of these via the Apple Developer website's Member Centre. When building through the Xcode UI, it handles most of this for you, but as we want to be building via the command line, and via SSH into a clean user account, we have a few more hoops to jump through.

Putting the certificates in a custom keychain

By default Xcode and it's command line tools will look for your distribution certificate in the user default keychain (the login keychain). As we want our build account to be "clean", we don't want to have to put them in the build account's login keychain. To resolve this, we can create a custom keychain, and put the certificates/provisioning profile keys in there.

We can then decide to either check this custom keychain file into source control, or put it in some other special location.

I'm assuming that you've already created and loaded/installed a distribution certificate, and an ad-hoc provisioning profile.

  1. From your normal user account, run Keychain Access
  2. Right-click on the list of keychains in the top-left, and create a new keychain. Give it a meaningful filename and store it somewhere you can find later.
  3. From your login keychain, hold the option key (to copy) and drag your iPhone Distribution certificate, and the Ad-hoc distribution private key.

Using this custom keychain in the build script

To get the Xcode command line tools to use this custom keychain file, add the following before you invoke xcodebuild

keychain_file="$SCRIPTPATH/iOSCustomBuild.keychain"
keychain_pass=secretpassword
security list-keychains -s "$keychain_file" ~/Library/Keychains/login.keychain  # put the iOS keychain in ahead of the login keychain
security unlock-keychain -p "$keychain_pass" "$keychain_file"

The security list-keychains -s command sets the search order, telling the system first to look in our custom keychain, followed by the default login keychain. This command is persistent - I haven't worried about resetting it after our script completes, but you may want to

The security unlock-keychain command is required because keychains are locked by default when code is run in a remote SSH session. We must issue security unlock-keychain or else xcodebuild will not be able to read the keychain. This is another reason to use a custom keychain file - if we were using the user's login keychain, we would be required to hard code the user account's password into our build scripts.

Compiling and packaging the app

To build our iOS app, we'll need to run xcodebuild, which is xcode's command line build tool. This requires a number of flags, which we will create variables for and then re-use. Here's mine:

# hack to get the full path to the current directory
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd`
popd > /dev/null

application_name=MyiOSApp # our app name - should correspond to MyiOSApp.xcodeproj
sdk="iphoneos7.0" # which version of iOS are we targeting

# Name of the distribution certificate. Get this by downloading the certificate from 
# the apple developer center, installing it, and then viewing it's name in keychain
codesign="iPhone Distribution: My Name (AAA12345ZZ)"

# build the project (must sign as xcode requires signing for all non-simulator bulds)
# CONFIGURATION_BUILD_DIR is only required for cordova/phonegap projects.
xcodebuild -project "$SCRIPTPATH/$application_name.xcodeproj" -target $application_name -configuration Release -sdk $sdk clean build CODE_SIGN_IDENTITY="$codesign" CONFIGURATION_BUILD_DIR="$SCRIPTPATH/build"

This only gets us halfway - xcodebuild will produce a .app package (a directory). This doesn't have the provisioning profile embedded in it, and we also need to package it up as a .ipa file to distribute it via the ad-hoc mechanisms. To do this step, we'll need to run xcrun. Here's my part of the script to do that:

# Get this by downloading it from the apple developer center
provisioning_profile_file="$SCRIPTPATH/My_Ad_Hoc_Provisioning_Profile.mobileprovision"
provisioning_profile_name="My Ad Hoc Provisioning Profile" # get this from the apple developer center when you download the file

# install the provisioning profile so xcode can use it 
provisioning_profile_uuid=`grep UUID -A1 -a "$provisioning_profile_file" | grep -o "[-A-Z0-9]\{36\}"`
cp "$provisioning_profile_file" ~/Library/MobileDevice/Provisioning\ Profiles/$provisioning_profile_uuid.mobileprovision
 
# this is where xcode will drop it's output, as a .app package
build_dir="$SCRIPTPATH/build"

# this is where we want the created .ipa file to be put
drop_dir="$SCRIPTPATH"

# package it and embed the provisioning profile (must re-sign as packaging alters the app)
/usr/bin/xcrun -sdk $sdk PackageApplication -v "${build_dir}/${application_name}.app" -o "${drop_dir}/${application_name}.ipa" --sign "$codesign" --embed "$provisioning_profile_file"

I chose to simply check my provisioning profile file into source control.

Also note that we have the extra step of "installing" the provisioning profile. This is because we're trying to run under a clean account. If we were running as a normal user account where someone had manually run Xcode, Xcode would have put the provisioning profile into the ~/Library/MobileDevice/Provisioning Profiles directory already for us and we wouldn't need to do this step.

The full build script


keychain_file="$SCRIPTPATH/iOSCustomBuild.keychain"
keychain_pass=secretpassword
security list-keychains -s "$keychain_file" ~/Library/Keychains/login.keychain  # put the iOS keychain in ahead of the login keychain
security unlock-keychain -p "$keychain_pass" "$keychain_file"

# hack to get the full path to the current directory
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd`
popd > /dev/null

application_name=MyiOSApp # our app name - should correspond to MyiOSApp.xcodeproj
sdk="iphoneos7.0" # which version of iOS are we targeting

# Name of the distribution certificate. Get this by downloading the certificate from 
# the apple developer center, installing it, and then viewing it's name in keychain
codesign="iPhone Distribution: My Name (AAA12345ZZ)"

# build the project (must sign as xcode requires signing for all non-simulator bulds)
# CONFIGURATION_BUILD_DIR is only required for cordova/phonegap projects.
xcodebuild -project "$SCRIPTPATH/$application_name.xcodeproj" -target $application_name -configuration Release -sdk $sdk clean build CODE_SIGN_IDENTITY="$codesign" CONFIGURATION_BUILD_DIR="$SCRIPTPATH/build"

# Get this by downloading it from the apple developer center
provisioning_profile_file="$SCRIPTPATH/My_Ad_Hoc_Provisioning_Profile.mobileprovision"
provisioning_profile_name="My Ad Hoc Provisioning Profile" # get this from the apple developer center when you download the file

# install the provisioning profile so xcode can use it 
provisioning_profile_uuid=`grep UUID -A1 -a "$provisioning_profile_file" | grep -o "[-A-Z0-9]\{36\}"`
cp "$provisioning_profile_file" ~/Library/MobileDevice/Provisioning\ Profiles/$provisioning_profile_uuid.mobileprovision
 
# this is where xcode will drop it's output, as a .app package
build_dir="$SCRIPTPATH/build"

# this is where we want the created .ipa file to be put
drop_dir="$SCRIPTPATH"

# package it and embed the provisioning profile (must re-sign as packaging alters the app)
/usr/bin/xcrun -sdk $sdk PackageApplication -v "${build_dir}/${application_name}.app" -o "${drop_dir}/${application_name}.ipa" --sign "$codesign" --embed "$provisioning_profile_file"

Putting it all together

All we need to do now is create TFS build script which Exec's plink and pscp and asks the mac to run the various .sh scripts which do all the work. Here's some snippets which you could use


<PropertyGroup>
    <!-- general -->
    <IpaFileName>MyiOSApp.ipa</IpaFileName>
    
    <!--File locations on the windows build machine-->
    <Plink>pathtoPLINK.EXE</Plink>
    <Pscp>pathtoPSCP.EXE</Pscp>
    
    <!--SSH connection info-->
    <SshHost>mac.domain.local</SshHost>
    <SshKey>buildaccountkey.ppk</SshKey>
    <SshUser>buildaccount</SshUser>

    <!--File locations on the mac-->
    <RemoteiOSBuildPath>/Users/buildaccount/iOSBuild</RemoteiOSBuildPath>
    <RemoteBuildScriptPath>$(RemoteiOSBuildPath)/build.sh</RemoteBuildScriptPath>
    <RemoteIpaFilePath>$(RemoteiOSBuildPath)/$(IpaFileName)</RemoteIpaFilePath>
</PropertyGroup>

...

<!--this "echo y | exit" causes plink/putty to cache the remote host key in the registry for subsequent operations.-->
<Exec command="echo y | &quot;$(Plink)&quot; -ssh -l $(SshUser) -i $(SshKey) $(SshHost) exit"/>

<!--Wipe any old files-->
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) rm -rf $(RemoteiOSBuildPath)" />
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) mkdir -p $(RemoteiOSBuildPath)" />

<!-- copy the source over to the mac -->
<Exec command="&quot;$(Pscp)&quot; -sftp -batch -r -i $(SshKey) -l $(SshUser) &quot;$(SolutionRoot)*&quot; $(SshHost):$(RemoteiOSBuildPath)" />

<!--run the build script-->
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) chmod +x $(RemoteBuildScriptPath)"/>
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) $(RemoteBuildScriptPath)"/>

<!-- copy the packaged ipa file back -->
<Exec command="&quot;$(Pscp)&quot; -sftp -batch -i $(SshKey) -l $(SshUser) $(SshHost):$(RemoteIpaFilePath) ." />

Wednesday, October 02, 2013

Programmatically controlling Hyper-V Server 2012 R2 virtual machines from C#

Recently I've wanted to remote control some virtual machines on a Hyper-V server in aid of automated testing.
I've got a server set up running Hyper-V Server 2012 R2, so the code samples in this are likely to work against Server 2012 non-R2, but may not work against Server 2008.

All the virtual machines start off at a known clean snapshot, and have a startup script which runs on boot that goes and asks a central database for a testing job to execute. So, for our system to work, we need to accomplish the following:
  1. Roll back a VM to the last known snapshot / checkpoint
  2. Power it on
There are several other things we need to do for this to work at all, which are:
  1. Connect to the Hyper-V server (this includes authentication)
  2. List out or otherwise ask the server about it's VM's so that we can rollback the correct one

To achieve thisk I'm using Hyper-V's WMI provider (V2). I don't really know much about WMI, so these code samples are just things I've hacked together. They are NOT production quality and they don't handle errors well or anything else. Please use them as guidance, not for copy/pasting.

Pre-requisites

Using WMI from C# is done by the classes in the System.Management namespace. To access this you'll want using System.Management at the top of your C# file, as well as a reference to the System.Management assembly.

Connecting and Logging on


To do this we need to create a ManagementScope object set to use the appropriate server and using the Hyper-V virtualization v2 namespace, and we also need to supply the username and password of a windows account that has privileges to administer the Hyper-V server. I did it like this:

var connectionOptions = new ConnectionOptions(
    @"en-US",
    @"domain\user",
    @"password",
    null,
    ImpersonationLevel.Impersonate, // I don't know if this is correct, but it worked for me
    AuthenticationLevel.Default,
    false,
    null,
    TimeSpan.FromSeconds(5);

var scope = new ManagementScope(new ManagementPath { 
    Server = "hostnameOrIpAddress", 
    NamespacePath = @"root\virtualization\v2" }, connectionOptions);
scope.Connect();

Note: Most of the other documentation or sample code I found refers to the root\virtualization namespace. This didn't work at all for me in Server 2012 R2, and I had to use dotPeek to decompile the Hyper-V powershell commandlets to figure out to put \v2 on the end. Perhaps the non-v2 one is for Server 2008?

Note: If your PC and the target server are on the same domain, and your windows user account has privileges to administer the remote server, You don't need the ConnectionOptions object at all. WMI will use windows authentication, and it will magically work:

Listing out the VM's and finding the one we want

Virtual machines in Hyper-V get exposed via the Msvm_ComputerSystem WMI class. In order to list them out, we simply ask WMI to give us all the objects of that class in the virtualization namespace. To do this, I'm going to create two helper extension methods, that we'll use from hereon:

public static class WmiExtensionMethods
{
    public static ManagementObject GetObject(this ManagementScope scope, string serviceName)
    {
        return GetObjects(scope, serviceName).FirstOrDefault();
    }
    
    public static IEnumerable<ManagementObject> GetObjects(this ManagementScope scope, string serviceName)
    {
        return new ManagementClass(scope, new ManagementPath(serviceName), null)
            .GetInstances()
            .OfType<ManagementObject>();
    }
}

It turns out that the Host PC is also exposed via Msvm_ComputerSystem, so we need to filter out things that are not virtual machines. This helper method will return a list of all the virtual machines. Note: I like extension methods, so I'm using them a lot here:

public static IEnumerable<ManagementObject> GetVirtualMachines(this ManagementScope scope)
{
    return scope.GetObjects("Msvm_ComputerSystem").Where(x => "Virtual Machine" == (string)x["Caption"]);
}

You can use it to get a specific virtual machine as follows:

var vm = scope.GetVirtualMachines().First(vm => vm["ElementName"] as string == "myvmname");

Rolling the VM back to it's latest snapshot / checkpoint

In order to roll a Hyper-V VM back to a snapshot, we need to get a reference to the snapshot object itself.

There are ways to simply list all the snapshots for each VM, but there is also a special Msvm_MostCurrentSnapshotInBranch class which represents the "latest" snapshot. We can use it to create a helper method as follows:

public static ManagementObject GetLastSnapshot(this ManagementObject virtualMachine)
{
    return virtualMachine.GetRelated(
        "Msvm_VirtualSystemSettingData",
        "Msvm_MostCurrentSnapshotInBranch",
        null,
        null,
        "Dependent",
        "Antecedent",
        false,
        null).OfType<ManagementObject>().FirstOrDefault();
}

And use it like this:

var snapshot = vm.GetLastSnapshot();

Now, to actually roll the VM back, we need to call the ApplySnapshot method on the Msvm_VirtualSystemSnapshotService class.
Note: Logically I thought that the snapshot methods should be on the Virtual Machine object, but Hyper-V puts them all in their own service for some reason. There is also only one global Snapshot service - not a service per VM. I've no idea why they've designed it this way.

We can create a helper method:

public static uint ApplySnapshot(this ManagementScope scope, ManagementObject snapshot)
{
    var snapshotService = scope.GetObject("Msvm_VirtualSystemSnapshotService");

    var inParameters = snapshotService.GetMethodParameters("ApplySnapshot");
    inParameters["Snapshot"] = snapshot.Path.Path;
    var outParameters = snapshotService.InvokeMethod("ApplySnapshot", inParameters, null);
    return (uint)outParameters["ReturnValue"];
}

And use it like this:

scope.ApplySnapshot(snapshot);

When I execute this with valid parameters, the VM applied it's snapshot, and I always got a return value of 4096. According to the documentation, this indicates that there's a Job in progress to asynchronously track the actual snapshot applying and determine the final success or failure. We could fetch the job out of outParameters["Job"] and use it to determine when the apply completes, but I'm not going to worry about that here. Refer to the ApplySnapshot MSDN page for other possible return codes.

Note: If the VM is not powered off, you are likely to find the ApplySnapshot call fails. You must power the VM off (and wait for the Power Off job to complete) first.

Powering on the VM to boot it up


Turning on the VM is done by the RequestStateChange method on the Msvm_ComputerSystem class. We already have the Msvm_ComputerSystem object representing the virtual machine we found earlier, so we can create a helper method and enumeration like this:

public enum VmRequestedState : ushort
{
    Other = 1,
    Running = 2,
    Off = 3,
    Saved = 6,
    Paused = 9,
    Starting = 10,
    Reset = 11,
    Saving = 32773,
    Pausing = 32776,
    Resuming = 32777,
    FastSaved = 32779,
    FastSaving = 32780,
}

public static uint RequestStateChange(this ManagementObject virtualMachine, VmRequestedState targetState)
{
    var managementService = virtualMachine.Scope.GetObject("Msvm_VirtualSystemManagementService");

    var inParameters = managementService.GetMethodParameters("RequestStateChange");
    inParameters["RequestedState"] = (object)targetState;
    var outParameters = virtualMachine.InvokeMethod("RequestStateChange", inParameters, null);
    return (uint)outParameters["ReturnValue"];
}

And use it like this:

vm.RequestStateChange(VmRequestedState.Running);

As for ApplySnapshot, when things work, the return value is usually 4096, indicating there is a Job to asynchronously track the progress of the operation.

Note: Although we invoke the method on the Msvm_ComputerSystem object, we need to get the method parameters by asking the Msvm_VirtualSystemManagementService object (which represents the host server) instead. I've no idea why.

Final fixups


ApplySnapshot will fail if the virtual machine is running. To turn it off, we can simply call vm.RequestStateChange(VmRequestedState.Off); and wait a bit for Job to complete.

RequestStateChange will fail if the state doesn't make sense. For example, if you try and turn the vm Off when it's already Off, the method will fail. You can check the current state of a Vm by reading it's EnabledState property. Valid values for EnabledState are documented with the Msvm_ComputerSystem class. I created an enum, and used it as follows:

public enum VmState : ushort
{
    Unknown = 0,
    Other,
    Running, // Enabled
    Off, // Disabled
    ShuttingDown,
    NotApplicable,
    OnButOffline,
    InTest,
    Deferred,
    Quiesce,
    Starting
}

if ((VmState)vm["EnabledState"] != VmState.Off)
{
    vm.RequestStateChange(VmRequestedState.Off); // needs to be off to apply snapshot
    Thread.Sleep(2000); // todo wait for the state change properly
}

You can do many more things by using other methods and classes from the Hyper-V WMI API. Hopefully this gives you a decent starting point.

P.S. The Hyper-V powershell commandlets are all implemented on top of this WMI Api. Using a tool like .NET reflector or dotPeek is an interesting way to see how Microsoft calls the WMI API.

Monday, October 08, 2012

TechEd 2012 background 3 - Memory Barriers

Note: This goal of this post is to provide more in-depth information and reference for attendees about a presentation I'm giving at Tech Ed New Zealand 2012.
The session is DEV402 - Multithreaded programming in .NET from the ground up.

This third post shows one the effect of the CPU re-ordering reads and writes to memory in a C# program, and explains how to fix it using Thread.MemoryBarrier

This sample program basically fires 2 functions at the threadpool over and over. In theory we can't tell which will run first, it's effectively random - This program just tries to provide an empirical measurement.

class Program
{
    static volatile int a = 0;
    static volatile int b = 0;
    static volatile bool a_ranFirst, b_ranFirst;

    static object gate = new object();

    static void ThreadA()
    {
        a = 1;
        //Thread.MemoryBarrier();
        if (b == 1)
            b_ranFirst = true;
    }

    static void ThreadB()
    {
        b = 1;
        //Thread.MemoryBarrier();
        if (a == 1)
            a_ranFirst = true;
    }

    static void Main(string[] args)
    {
        int aFirst = 0, bFirst = 0;
        int together = 0;
        int WTF = 0;

        var sw = new Stopwatch(); sw.Start();

        const int Iterations = 500000;

        for (int i = 0; i < Iterations; i++)
        {
            a = b = 0;
            a_ranFirst = b_ranFirst = false;
            Parallel.Invoke(ThreadA, ThreadB);

            if (a_ranFirst && !b_ranFirst)
                aFirst++;
            else if (!a_ranFirst && b_ranFirst)
                bFirst++;
            else if (a_ranFirst && b_ranFirst)
                together++;
            else
                WTF++; //
        }
        Console.WriteLine("Done in {0}ms", sw.ElapsedMilliseconds);

        Console.WriteLine("Thread A ran first: {0} times ({1} percent)", aFirst, ((double)aFirst / (double)Iterations) * 100);
        Console.WriteLine("Thread B ran first: {0} times ({1} percent)", bFirst, ((double)bFirst / (double)Iterations) * 100);
        Console.WriteLine("Ran together: {0} times ({1} percent)", together, ((double)together / (double)Iterations) * 100);
        Console.WriteLine("WTF: {0} times ({1} percent)", WTF, ((double)WTF / (double)Iterations) * 100);
    }
}

If I run this as-is, I get something like this:
Done in 1571ms
Thread A ran first: 379388 times (75.8776 percent)
Thread B ran first: 31140 times (6.228 percent)
Ran together: 59 times (0.0118 percent)
WTF: 89413 times (17.8826 percent)
76% of the time, A runs first (this makes sense given that it's the first argument to Parallel.Invoke) - 6% of the time B runs first, and a very small percentage of the time they run together in lockstep. However, 18% of the time, something else is happening. What's going on? To break this down, the important operations that happen on each thread are:
  • Store the value 1 to the memory location A (or B for thread B)
  • Load the value from the other memory Location
  • Figure out whether this thread was first on second based on what we read
  • Set a flag to indicate whether this thread was first or second
The first 2 steps are the key to explaining the bug. Here's what we might expect to be happening
  • When "A runs first", thread A performs both the store to A and load from B before thread B gets started.
  • When "B runs first", thread B performs both the store to B and load from A before thread A gets started.
  • When they run together, thread A stores, thread B stores, then they both load

So, what about the Fourth, "WTF" scenario? We've already marked the a and b variables as volatile, so this should tell the compiler to leave them alone. Something else is happening, but what?

Let's have a look at the assembly that's getting generated:
a = 1;
00000000  mov         dword ptr ds:[0111328Ch],1 
if (b == 1)
0000000a  cmp         dword ptr ds:[01113290h],1 
00000011  jne         0000001A 
b_ranFirst = true;
00000013  mov         byte ptr ds:[01113295h],1 
0000001a  ret 

A quick run through:
  • The first mov operation is writing 1 to memory location 0111328C (where the compiler has stored a)
  • The next cmp/jne instructions read 01113290 (where the compiler has stored b), and jump to the end of the function if it's not set to 1.
  • If it doesn't get jumped over, the final mov operation is writing 1 to location 01113295 (where b_ranFirst lives)

This all adds up - the instructions we'd logically expect to happen are indeed happening, and it's all in the order we'd expect. Clearly neither the C# or JIT compiler is at fault here... so what is?

Well, as it turns out there are actually 3 more scenarios that are occurring.

  • Thread A performs load from B; thread B then does it's normal store/load; finally, thread A performs store to A
  • Thread B performs load from A; thread A then does it's normal store/load; finally thread B performs store to B
  • thread A loads, thread B loads, then they both store

In spite of the fact that our code very explicitly always stores before it loads, the operations are being performed in the wrong order.

This is Out-of-Order Execution in action. All Intel CPU's (except the Atom) since 1995 do this, as do pretty much all AMD processors. I've got a Core i5 here, which certainly does this reordering.

Why reordering?

It takes many hundreds of CPU cycles to do a read or write to memory. The number varies quite a lot depending on what CPU you have, but it's always a significant amount. To try compensate for this, out-of-order CPU's will try and rearrange reads and writes such that they can carry on doing work while waiting for the data to arrive or the write to complete. Writes can also be batched in some situations, so the CPU may hold off doing a write until a second one arrives to issue a single batch.

When is reordering a factor?

For the majority of code, reordering isn't an issue.
While it is executing code, a CPU keeps track of whatever re-ordering it does, and makes sure that all operations appear to execute as you'd expect. If it moves something before or after another operation, then it will make sure to fix up the result before the next thing needs to access the result.

In computers with only a single CPU core, all code is effectively single threaded like this - While the operating system can and does interrupt and switch the running thread randomly and unpredictably, when it does this it inserts appropriate instructions (memory barriers) which cause any re-ordered operations to be "flushed" before the thread is context switched out.

The problem arises in multi-CPU systems.
While each CPU is keeping track of and fixing up all it's shenanigans, it's not telling the other CPU's about what it's doing. When other CPU's read or write, they don't get this "fixing up" logic applied, and they see reads and writes in the actual (rearranged) order they're happening.

OK, so how do we fix it?

The first thing we might like to do is somehow tell the CPU to turn this reordering off. Unfortunately (or fortunately, depending on your point of view) - you can't. Furthermore, there's no tracing or other kinds of diagnostics to tell you when you're being affected by this. Unfortunately, you have to resort to good old logic and knowledge. You have to know what the rules for reordering are for your CPU, analyze the code, and work out if reordering could possibly cause your code to fail. If it could - you have to defend against it.

And how do we defend against it? This is where the Memory Barrier comes in. A memory barrier acts as a "fence" or "line in the sand", or other such metaphor. Basically, it's a wall that the CPU can't move reads or writes through.

There are 3 kinds of memory barriers. Most of the information that I've been able to find is quite confusingly worded, so I'll do my best to be clear:

  • Read barriers (also known as "Load" or "Acquire" barriers/fences). These prevent loads from memory from moving upwards in the instruction sequence.
  • Write barriers (also known as "Store" or "Release" barriers/fences). These prevent writes to memory from moving downwards in the instruction sequence.
  • Full barriers. These prevent all re-oredering

Wednesday, September 05, 2012

TechEd 2012 background 2 - Compiler optimizations and Volatile

Note: This goal of this post is to provide more in-depth information and reference for attendees about a presentation I'm giving at Tech Ed New Zealand 2012.
The session is DEV402 - Multithreaded programming in .NET from the ground up.

This second post shows one possible effect of compiler optimizations on a C# program, and explains how to fix it using the oft-misunderstood volatile keyword.

Here's a sample program which illustrates this.
This program looks suspiciously similar to one of the samples in Joseph Albahari's threading book because that was by far the best piece of code I found or could invent to best illustrate the issue. All credit due to Joseph for his great work


class Program
{
    static int stopFlag = 0;

    static void ThreadProc()
    {
        Console.WriteLine("{0}: worker thread start", DateTime.Now.TimeOfDay);
        bool toggle = false;

        while (stopFlag == 0)
        {
            toggle = !toggle;
        }
        Console.WriteLine("{0}: worker thread done", DateTime.Now.TimeOfDay);
        Console.ReadLine();
    }

    public static void Main(string[] args)
    {
        stopFlag = 0;
        var t = new Thread(ThreadProc);
        t.Start();
        Thread.Sleep(1000);

        stopFlag = 1;

        Console.WriteLine("waiting...");
        t.Join();
    }
}


In summary, it's pretty simple. The main thread spins up a worker thread, sleeps for a second, then sets the stopFlag. The worker thread sits on a loop waiting for the stopFlag to be set, then exits.

If you run a Debug build of this code, or launch a 32-bit Release build with the debugger attached, you'll get this output:

10:38:25.0815053: worker thread start
waiting...
10:38:26.0955150: worker thread done

However, if you launch a 32-bit Release build without the debugger attached, you'll see this:

10:51:47.1955922: worker thread start
waiting...

The worker thread won't stop, and if you launch Task Manager you'll see that the app is thrashing away using an entire CPU. What's going on here?

To answer this, we need to attach the debugger to the already-running program, pause it and jump over to the worker thread and see what it's up to. The process for this in Visual Studio 2010/2012 goes like this:

  • From the Debug menu, chose Attach to Process...
  • Scroll down the list, select your application, and click Attach
  • Once the attach dialog closes, pause the application by clicking the pause button on the toolbar, or selecting Break All from the Debug menu. This will stop the program looping and let us look at what it's up to
  • Bring up the threads window, ( Debug / Windows / Threads ) and double-click on the worker thread

We should now see the current line of code that is being executed by this thread, which is

while (stopFlag == 0)

So, we're still in the loop waiting for the stopFlag to be set - yet, if you mouse over the stopFlag variable, or Add it to the Watch window, we'll see that it has a value of 1

The flag has been set, but our loop is still going. What gives?
To answer this, we need to bring up the Disassembly window. ( Debug / Windows / Disassembly ). This window shows us the raw x86 machine code that has been generated by the JIT compiler, and is the actual code that is being executed on the CPU.

The VS disassembly window will print each line of C# source code (if it has it), followed by the x86 assembly. For the line we're stuck on, we get this:

while (stopFlag == 0)
0000004a  mov         eax,dword ptr ds:[007A328Ch] 
0000004f  test        eax,eax 
00000051  jne         00000057 
00000053  test        eax,eax 
00000055  je          00000053 

If you know how to read x86 assembly, I congratulate you. I only know the basics, but enough that I can walk you through it and show what's going on.

So first, this line:

0000004a mov eax,dword ptr ds:[007A328Ch]

  • The instruction is mov which is x86 for "set something". The first parameter is the target, and the second is the source.
  • The First parameter is eax which refers to the EAX register inside the CPU. Intel x86 processors have 4 general purpose registers - EAX through EDX - that are used to perform the vast majority if work that happens in an application.
  • The Second parameter is dword ptr ds:[007A328Ch]. The dword ptr bit is a Size directive ( see this page and scroll down to "Size Directives" ) and it's needed to tell the CPU how many bytes we should be moving. dword ptr simply means 32 bits. I'm not entirely sure what the ds: bit means, but the most important part is this: [007A328Ch]. The square brackets mean "the value in memory at the given location". Our stop flag is static, so it's stored in a fixed memory location, in my case 007A328C

Translated: "Fetch 32 bits from memory at location 007A328C, and put them in the EAS register." In other words - load our stopflag into the CPU

The next part is this:
0000004f test eax,eax
00000051 jne 00000057


  • The test eax,eax is basically asking the question "Does EAX contain zero?"
  • The jne 00000057 is the Jump-if-not-equal instruction. It translates to "If it was NOT zero, jump to line 57

We know however, that when the loop starts, the flag was in fact zero, so we'll get past this block and onto the next one:

00000053 test eax,eax
00000055 je 00000053


  • Again, "Does EAX contain zero?"
  • The je 00000053 is the Jump-if-equal instruction. It translates to "If it WAS zero, jump to line 53, which is the start of our loop

Putting this all together, we end up with this sequence of events:

  • Load the stopflag from memory into EAX
  • Loop waiting for EAX to become non-zero

The problem is, at no point does any code ever set eax once we enter the loop.

What's going on here? The answer is that the C# and JIT compiler only consider the single-threaded case when doing optimizations. From the point of view of the WorkerThread, we never set the stopFlag, and we never call any other functions, so there is no possibility that the stopFlag can change. The JIT compiler can apply this, and decide that it only needs to read the stopFlag a single time, leading to this infinite loop!

To fix it, we simply mark the stopFlag variable as volatile.

This causes the assembly to change to this:

0000004a cmp dword ptr ds:[0088328Ch],0
00000051 jne 0000005C
00000053 cmp dword ptr ds:[0088328Ch],0
0000005a je 00000053


The key difference is that we see [0088328Ch] in the repetitive part of the loop. This means that at each iteration through the loop we go to memory and read the value. This means that when the stopFlag gets set after a second, we'll re-read it and the program behaves correctly. Great!

Discussion


There are a couple of things I'd like to go into more detail about:

Firstly, the weird double-check that the compiler generates for loops.
When the JIT compiler generates a loop it always seems to use this pattern:

  • First, check the opposite of the loop condition. If this is true, jump over the entire loop code
  • Now we enter the loop where we do the repeated action, and check the actual loop condition

Technically, there's no reason why we need to check it twice, we could get by with just the second part where we check the positive loop condition. I've no expert, but from what I can guess this is probably to help with branch prediction.
My personal theory is that by splitting the loop into 2 conditions, the branch predictor can track them both seperately.
If we bail out of the loop before it even gets started, the branch predictor won't need to worry about the subsequent code, but if we enter the second phase of the loop, the odds are good that we'll keep looping, so the branch predictor can use this information.

Secondly, the use of the cmp instruction rather than test in the loops
This stackoverflow question/answer sheds some light on the topic

The test instruction is smaller (different x86 instructions take different numbers of bytes to encode) than cmp, so the compiler will prefer it over cmp where both would achieve the same result. test works by doing a bitwise AND on the two values, whereas cmp does subtraction.

In the first case, where the value has already been loaded into a register, ANDing it with itself is a nice fast way to check for zero. In the second case however, where the value is being read from memory every time, in order to use test, we'd have to either use an extra register to store it in, or read the memory twice for both sides of the test operation. Both of which would be slower than against the constant 0.

Thirdly, what happened to the toggling inside the loop?
You may have noticed that the disassembly at no point shows anything to do with the loop body of toggle = !toggle;.

This is just the compiler optimizer doing it's job.

toggle is a local variable, it's not returned, and it's not passed to any functions - therefore, whatever we do to it can have no possible effect on anything. The compiler detects this, and simply removes it entirely.

So, what does this mean about volatile?
There seems to be a lot of confusion around what the volatile keyword is supposed to do. There are many questions/answers and blog posts which attempt to explain it. Some say say that it will insert memory barriers either before or after writes to the variable, some say that it causes reads and writes to be translated to calls to the Thread.VolatileRead or VolatileWrite method (both of which insert explicit barriers - but as we can see from our disassembly, it isn't inserting any such barriers, it's just doing normal ordinary reads and writes.

I think a lot of this confusion comes from the MSDN volatile documentation. The first hit in google for "C# Volatile Keyword" is the MSDN documentation, which states The system always reads the current value of a volatile object at the point it is requested, even if the previous instruction asked for a value from the same object. Also, the value of the object is written immediately on assignment.

In order for this to be true (particularly the "value is written immediately" part) one might infer that the compiler would insert memory barriers. Unfortunately, this page is the documentation for Visual Studio 2003. Subsequent revisions (2005, 2010, etc) re-word the documentation to be less misleading, but unfortunately google is still linking everyone to the old out of date page! For more, see http://www.albahari.com/threading/part4.aspx#_The_volatile_keyword

Additionally, the ECMA CLI spec states that reads to volatile fields should have acquire semantics, and writes to volatile fields should have release semantics. This could also cause people to think that memory barriers might be neccessary, but reads and writes on x86 already have these semantics anyway, so the net effect of volatile (other than compiler optimizations) is... Nothing!. On ARM or Itanium however, the JIT will insert memory barriers around reads and writes to volatile fields to give you the acquire/release semantics, as those CPU's don't guarantee it natively.

Tuesday, September 04, 2012

TechEd 2012 background 1 - Race conditions and Interlocked operations

Note: This goal of this post is to provide more in-depth information and reference for attendees about a presentation I'm giving at Tech Ed New Zealand 2012.
The session is DEV402 - Multithreaded programming in .NET from the ground up.

This first post goes over what is happening to cause a basic race condition, caused by the interruption and interleaving of 2 threads accessing a shared variable.

Here is a buggy C# program. Feel free to copy/paste it into visual studio and run it yourself.
I'm guessing from what I've seen that probably 98% of the .NET code people are writing is running in 32-bit mode, so that's what I'm focusing on.

public static class BasicProgram
{
    public static int value = 0;

    const uint Iterations = 100000000;

    static System.Diagnostics.Stopwatch watch = new System.Diagnostics.Stopwatch();

    static Object x = new object();

    static void Main(string[] args)
    {
        watch.Start();
        new Thread(Worker1) { IsBackground = true }.Start();
        new Thread(Worker2) { IsBackground = true }.Start();

        Console.WriteLine("Running... Press enter to quit");
        Console.ReadLine();
    }

    static void Worker2(object _)
    {
        for (uint i = 0; i < Iterations; i++)
        {
            value++;
        }

        Console.WriteLine("Worker2 done at {0}ms", watch.ElapsedMilliseconds);
    }

    static void Worker1(object _)
    {
        var oldValue = value; var newValue = value;

        for (uint i = 0; i < Iterations; i++)
        {
            oldValue = value;
            value++;
            newValue = value;

            if (newValue < oldValue)
                Console.WriteLine("i++ went backwards! oldValue={0}, newValue={1}!", oldValue, newValue);
        }

        Console.WriteLine("Worker1 done at {0}ms", watch.ElapsedMilliseconds);
    }
}


The bug manifests itself by hitting the sanity-check in Worker1. Usually for me this happens anywhere between 1 and 10 times, but sometimes it doesn't happen at all. The loop runs 100 million times, yet the bug only happens rarely - this is a good example of just how random and hard to reproduce these kinds of threading issues can be.

Here's the output for a random run on my laptop (dual-core Intel i5 with hyperthreading):

Running... Press enter to quit
i++ went backwards! oldValue=35777960, newValue=35777925!
Worker2 done at 266ms
Worker1 done at 402ms

This is pretty much the standard textbook race condition.

I'd imagine most developers with a bit of experience will have seen this or something similar before, and so I don't think the purely educational value for this is that high, but my main goal is to use it this as a starting point.
It teaches (or reinforces) several fundamental key points that I want people to have in their minds, so I can build upon these points to explain more complex threading bugs.

These fundamental lessons are:

  • In order to do any work, the computer has to Read from memory, Modify the value inside the CPU, then Write back to memory. These are 3 (or more) discrete steps.
  • Because of this, simple-looking things such as value++ actually turn out to be multi-step operations
  • The OS will interrupt the operation of your threads at arbitrary points... If a thread is in the middle of a multi-step operation, this may mean another thread can come along while it's suspended, and modify data out from underneath it. This is the root cause of the bug.

You can fix this by using a lock - The idea of course is that if you acquire a lock before doing the read-modify-write, then other threads must wait for you to release the lock before they can touch the shared value. If a thread gets suspended in the middle of this read-modify-write operation, well, other threads must wait for it to be resumed so it can release the lock, and the whole "modifying data out from underneath it" simply can't happen.

This is great, and I want to make the point that normally this is exactly the right thing to do, but there is also another way to solve this problem - by using Atomic Operations.

What are atomic operations? All CPU's read and write to memory in certain chunks of bytes

  • A 32-bit processor (or a 64-bit processor running in 32-bit mode) will read or write to memory in blocks of 32-bits
  • A 64-bit processor running in 64-bit mode will read or write to memory in blocks of 64-bits

Handily, this is the same size as a pointer in native C or C++ code for those platforms, and .NET Object references are just pointers. (You don't get to see the pointers directly, they are managed for you, hence why .NET is a "managed" language - but they're still there in exactly the same way they would be in a C++ program)

Additionally, there's the functions in System.Threading.Interlocked

What I don't mention in the talk is memory alignment. Memory alignment affects atomicity, as if your 32-bit int is split over 2 blocks due to alignment issues, you will have to read/write both those blocks, and hence it won't be atomic.

I chose not to mention this because pretty much the only way to get unaligned memory access in .NET is when fiddling with a struct that uses the [StructLayout] attribute , which only really happens for P/Invoke calls.
The remaining 99.9% of .NET code is likely to have properly aligned memory access patterns, so I didn't think it worth spending time on during the talk.

Before applying the interlocked fix, first we can see what happens by simply applying a lock.

value++; becomes lock(x){ value++; }

and we get this output:


Running... Press enter to quit
Worker2 done at 3401ms
Worker1 done at 5539ms

Let's compare that with Interlocked Increment:

value++; becomes Interlocked.Increment(ref value);

Here's the output:

Running... Press enter to quit
Worker2 done at 2269ms
Worker1 done at 2276ms

Hooray! No more bugs. You'll notice however, that it's around twice as fast as using a lock, but still much slower than no synchronization at all - Why does it behave like this? Let's dig in and find out.

If we view the disassembly produced by a call to Interlocked.Increment, we see this:

lock inc dword ptr ds:[00B2328Ch]

  • Our value variable is static, so it will be stored at a fixed location in memory. This location happens to be 00B2328C for me.
  • The dword ptr bit before it is a Size Directive ( see this page and scroll down to "Size Directives" ) - we need to tell the inc instruction how big the thing we're incrementing is - dword ptr basically means 32 bits.
  • The inc instruction does what it says it does... increments a value
The magic comes from the lock prefix. Here's a stackoverflow question with a good answer explaining it. Basically what this does is tell the CPU to lock the memory bus for the duration of the operation. As such the read-increment-write operation can't be interrupted.

So essentially, Interlocked.XYZ operations actually are just using locks, but the difference is that they are CPU-level hardware locks, rather than the normal software locks we're used to. Pretty cool huh. It also explains why the Interlocked version takes longer - all that locking and unlocking of the memory bus takes time, and additionally if one CPU locks it, the other one has to wait, just like a normal lock.

So, what about the lock version? What makes it twice as slow as Interlocked?
Microsoft makes available the Shared Source CLI 2.0. This is a "reference implementation" of .NET version 2.0, and it's sufficiently similar to .NET 4.0 and 4.5 in most areas that we can consider it to be pretty much correct. Think of it as the source code for the core native C++ parts of .NET. The garbage collector, JIT compiler, etc.

If you download this and dig through it, you'll find the code that implements the native CLR lock in \clr\src\vm\syncblk.cpp. I'll spare you the effort of having to parse it all - but what it comes down to is that in the best-case scenario (no lock contention, etc), a .NET lock must do at least one InterlockedCompareExchange to acquire the lock, and another InterlockedCompareExchange to release it.

There's obviously a lot more going on, but it's pretty easy to infer from that why the performance is different. A lock is doing two interlocked operations, and Interlocked.Increment is only doing one. This lines up quite nicely with our observed results.

I hope you enjoyed or learned something, I know I certainly had fun figuring all this stuff out. I will follow this post up with more in-depth details of two other causes of threading bugs - compiler optimizations, and memory re-ordering

Monday, September 12, 2011

Update: I had previously posted a helper class for creating memory dumps of .NET processes. That class turned out to have some bugs in it, I've updated accordingly

http://orionedwards.blogspot.co.nz/2011/07/helper-class-for-creating-memory-dumps.html

Wednesday, August 24, 2011

Advanced .NET Debugging TechEd 2011 Presentation

I've just finished my TechEd 2011 presentation on advanced .NET debugging. I covered using WinDBG and SOS to troubleshoot memory leaks, deadlocks, race conditions.
You can download the powerpoint presentation here

Sunday, July 31, 2011

Helper class for creating memory dumps of a Managed Process

I'm giving a talk at TechEd NZ 2011 in about a month. As part of that talk, I'll mention creating memory dumps using the MiniDumpWriteDump function, and show a helper class which P/Invokes it

Here is that helper class (Updated 19 Sept 2011 to fix some bugs).


using System.Runtime.InteropServices;
using System;
using System.IO;
using System.Diagnostics;
using System.Threading;

public static class DbgHelp
{
    [StructLayout(LayoutKind.Sequential, Pack = 4)]
    struct MINIDUMP_EXCEPTION_INFORMATION
    {
        public uint ThreadId;
        public IntPtr ExceptionPointers;

        [MarshalAs(UnmanagedType.Bool)]
        public bool ClientPointers;
    }

    [DllImport("Dbghelp.dll")]
    static extern bool MiniDumpWriteDump(
        IntPtr hProcess,
        uint ProcessId,
        IntPtr hFile,
        [MarshalAs(UnmanagedType.I4)] MiniDumpType DumpType,
        IntPtr ExceptionParam, // Ptr to MINIDUMP_EXCEPTION_INFORMATION
        IntPtr UserStreamParam,
        IntPtr CallbackParam);

    [DllImport("kernel32.dll")]
    static extern uint GetCurrentThreadId();

    enum MiniDumpType : int
    {
        MiniDumpNormal = 0x00000000,
        MiniDumpWithDataSegs = 0x00000001,
        MiniDumpWithFullMemory = 0x00000002, // required for .NET apps
        MiniDumpWithHandleData = 0x00000004,
        MiniDumpFilterMemory = 0x00000008,
        MiniDumpScanMemory = 0x00000010,
        MiniDumpWithUnloadedModules = 0x00000020,
        MiniDumpWithIndirectlyReferencedMemory = 0x00000040,
        MiniDumpFilterModulePaths = 0x00000080,
        MiniDumpWithProcessThreadData = 0x00000100,
        MiniDumpWithPrivateReadWriteMemory = 0x00000200,
        MiniDumpWithoutOptionalData = 0x00000400,
        MiniDumpWithFullMemoryInfo = 0x00000800,
        MiniDumpWithThreadInfo = 0x00001000,
        MiniDumpWithCodeSegs = 0x00002000,
        MiniDumpWithoutAuxiliaryState = 0x00004000,
        MiniDumpWithFullAuxiliaryState = 0x00008000,
        MiniDumpWithPrivateWriteCopyMemory = 0x00010000,
        MiniDumpIgnoreInaccessibleMemory = 0x00020000,
        MiniDumpWithTokenInformation = 0x00040000
    };

    public static void WriteExceptionDump(string filePath)
    {
        var proc = Process.GetCurrentProcess();
        int win32Error;
        if(!TryCreateDump(filePath, proc.Handle, (uint)proc.Id, GetCurrentThreadId(), Marshal.GetExceptionPointers(), out win32Error))
            throw new Exception("Couldn't create dump file! Error: 0x" + win32Error.ToString("X8"));
    }

    public static bool TryCreateDump(string dumpFilePath, IntPtr processHandle, uint processId, uint threadId, IntPtr exceptionPointers, out int win32Error)
    {
        bool success = false;
        int lastError = 0;

        // Dump on a seperate thread - IsBackground=false is important to stop the process exiting while we write the dump
        // also works around an issue of VS not being able to walk the callstack of the crashing thread
        var thread = new Thread(new ThreadStart(() => {
            // In-process dumps must ClientPointers = false
            // If ClientPointers is false, or if there are no ExceptionPointers we must pass IntPtr.Zero as ExceptionInfo
            var exceptionParam = IntPtr.Zero;
            if (processId != Process.GetCurrentProcess().Id && exceptionPointers != IntPtr.Zero)
            {
                var ei = new MINIDUMP_EXCEPTION_INFORMATION {
                    ClientPointers = true, // in-process dump. True if we're dumping external processes
                    ExceptionPointers = exceptionPointers, // may be IntPtr.zero for CLR exceptions
                    ThreadId = threadId,
                };
                exceptionParam = Marshal.AllocHGlobal(Marshal.SizeOf(ei));
                Marshal.PtrToStructure(exceptionParam, ei);
            }

            using (var outputFile = new FileStream(dumpFilePath, FileMode.Create))
            {
                success = MiniDumpWriteDump(
                    processHandle,
                    processId,
                    outputFile.SafeFileHandle.DangerousGetHandle(),
                    MiniDumpType.MiniDumpWithFullMemory,
                    exceptionParam,
                    IntPtr.Zero,
                    IntPtr.Zero);
            }

            if (!success)
                lastError = Marshal.GetLastWin32Error();

            if (exceptionParam != IntPtr.Zero)
                Marshal.FreeHGlobal(exceptionParam);
        })) { IsBackground = false, Name = "MiniDump thread" };

        thread.Start();
        thread.Join();

        win32Error = lastError;
        return success;
    }
}




Sunday, January 10, 2010

DNUG Reactive Framework Presentation

On December 17th (2009) I gave a talk to my local .NET user group about the Microsoft Reactive Extensions for .NET
Here's a brief outline of what I talked about:
Note: I had a recurring theme throughout the presentation that shorter code is better. I had random slides thrown in with quotes from programming "celebrities" to keep trying to push the point. I did this because I personally believe in it, and also because being able to do more with less code is one of the big benefits of the Reactive Framework.
As I've met more than a few developers who are still running .NET 2.0 on VS2005, I started by giving a brief recap on the C# 3.0 language features that Rx makes heavy use of (lambdas, linq, and so forth)
I then did a quick overview defining exactly what Asynchronous programming is, and why you'd want to do it (Rx is all about asynchronous programming after all)
With the overviews out of the way, I talked a bit about the IObservable interface and how it related to IEnumerable, and showed a few short code snippets of code using Rx (looking surprisingly the same as ordinary Linq), and some diagrams showing the timeline of things that happen when using both IEnumerable and IObservable
I then cut to a demo. Sample code shown in the demo is available as on Google Code and I'm placing it under the creative commons by attribution license. You don't have to provide attribution (it's a code sample!), but I can't find a CC license other than public domain that doesn't require attribution.
Most of the demo focused on the WcfClient and WcfServer projects - they're the most interesting, so I'd suggest looking at those if you're interested.
I followed up with a few more slides containing other tidbits (such as the fact that you get a backport of the .NET 4 parallel task library for free with Rx), and some links.
Enjoy!

Saturday, March 28, 2009

Programming podcast roundup

When the stackoverflow podcast first launched, I downloaded it and gave it a listen. I enjoyed it, and I was sick of listening to the same old music when going running... and so, my podcast-listening-habit was born.

Thus far I've been stuck in the microsoft-centric technology podcasts. This not because I'm a microsoft shill, but because I haven't been able to find any non-microsoft-centric podcasts out there.

At any rate, here's the ones I regularly listen to, ranked by preference.

1. The Stack Overflow Podcast

Admittedly I'm biased as this was the first podcast I listened to, and I've been following it since day one. Even had this not been the case, I think I'd still rank it highly.
The SO podcast primarily consists of Jeff Atwood and Joel Spolsky chatting about the stackoverflow site, and programming/IT topics in general. Major topics are either pulled from the SO site, from reader questions, or often based on current events, or whatever they're each up to.
Even if you think Joel and Jeff are a pack of jumped up blowhards (as many no doubt do), they're still really entertaining to listen to. Both speak well and have a wide variety of experience to draw on (Joel in particular is the king of 'back in my day' type stories, which are usually very interesting). They also complement eachother well which makes for good listening.
I like the fact that the show is centered around them and on the stackoverflow site. They'll occasionally have third party participants on the show, but the majority of shows are just Jeff and Joel. I find this helps you feel like you "know them" better, rather than that they're just interviewers or reporters.
Finally, the SO podcast gets a big bonus for not being full of annoying ads. It has a small bit at the beginning and end from IT conversations, who provide their hosting, and that's it. It typically runs for about an hour.

2. Hanselminutes

Hanselminutes is Scott Hanselman interviewing people about technology. Every week there's a different guest, always talking about some recent technology. Most (but not all) are microsoft based, which is no doubt a side effect of Scott being a microsoft employee.
This does however have it's upsides, as you'll get to find out stuff by hearing it directly from other microsofties that come on the show, rather than hearing things through the rumourmill / blogosphere / reddit.
Hanselminutes is very well produced, and Scott really knows how to do a good interview. Most of the guests are top-notch, and often the tech talk gets pretty deep, which IMHO is great, as it provides the substance of the show.
Recently the podcasts took a bit of a detour, while Scott was in south africa - and interviewed some of the local people about non-technical things, and his family (his wife is from Zimbabwe). I really enjoyed these, as it was really cool to get a bit of insight into the way things are in some other (non-westernised) countries.
Hanselminutes has some advertising at the beginning, and typically has a single "spliced in" ad in the middle. These tend not to be too long though. The show usually runs for half an hour.

3. The Australian Gamer Podcast

OK, this is not programming related at all, but I'm an ex gamer so I like to keep up with that scene periodically.
At any rate, Matt and Yug (the hosts) are _very_ funny (in a crude guy-humour kind of way. It's R18, you have been warned.)
I know of many people who have little to no interest in cars, yet enjoy watching Top Gear, because the presenters simply put on a really great entertaining show.
In my humble opinion at least, the AG podcast is similar. I played it in the car while driving somewhere with my girlfriend (who is not a gamer in the slightest), and she said that apart from all the swearing, it was actually pretty good. That's high praise :-)

4. Herding Code

Herding code is a "technology round-table" run by K. Scott Allen, Kevin Dente, Scott Koon and Jon Galloway, who are all either microsofties, MVP's, or otherwise working using the microsoft technology stack. It's usually just them sitting around chatting (4 people is more than enough to keep a conversation going), with occasional interviews.
It's not as professionally produced as some other podcasts, but it does have a really good friendly atmosphere. You feel like these guys are good mates, and they'd be sitting around talking about this stuff anyway, recording or not.
I really like this, as you feel like you get to know them a lot more, which keeps you engaged.
Herding code doesn't have any sponsored introductions, or inline ads at all. This is awesome, but I wonder if it's just because the show hasn't picked up any yet?

5. Deep Fried Bytes

DFB is another interview driven show, run by Chris Woodruff and Keith Elder. It's kind of similar to hanselminutes, in that there's a guest being interviewed or talked to every show, but Chris and Keith bring their own brand of humour and atmosphere, and also have great character. The "feel like you know them" factor is high.
My one complaint about DFB is that the format thus far seems to be like this: 1) Go to a conference and record interviews with conference speakers and attendees. 2) publish interview as a podcast, repeat until you've run out of interviews, then go to another conference
This is not in and of itself bad, but when you're putting out shows in february which were recorded at the PDC in october, it starts to wear thin a bit.
DFB doesn't have any sponsored intros or ads, and runs for around 45 minutes.

6. .NET Rocks!

DNR is the granddaddy of the tech podcast. Carl Franklin and Richard Campbell are currently on show Number 432, and put out new ones every week like clockwork.
It's another interview-centric show, focused around the Microsoft .NET ecosystem. It sometimes has a bit of an overlap with Hanselminutes (the same guests will appear on each show in quick succession when a new technology is coming out of MS), but it's different enough to always be worth listening to both.
The production quality is top notch (the best out of all the podcasts I've heard). Carl is a musician, and his obvious knowledge of all things audio shows itself here. Apart from the fact that they're talking about .NET programming, you could easily believe this was a show coming from your local radio station's best morning dj crew.
I feel kind of bad putting DNR down the bottom of the list, as it really is a very good podcast, but sadly it is the one I will listen to after I've heard all the others. This is more a testament to how much I like the others than an indictment against DNR. Carl and Richard have been doing this for a long long time, and they're very good at it.
The main thing that bugs me about DNR is actually the ads. They have a 2 minute sponsored intro, then usually 2 or 3 ads spliced into the middle of the show. The ads tend to be longer than on other podcasts, and more intrusive. Shows tend to run between 45 minutes to an hour, but if you skip the ads and the leadout, it's about ten minutes less. If you know of any other good tech podcasts, just drop a comment! Cheers, Orion

Monday, March 09, 2009

Duplicate Line in Visual Studio

Visual Studio ships with many features built in. "Duplicate the current line" doesn't appear to be one of them for some strange reason.

CodeRush Express ships with a "duplicate line" function, but it's WAY too clever for it's own good. It tries to work out whether you're duplicating a line with a variable, function, etc, and act accordingly. If it can't understand your line (perhaps it's just a string), then it FAILS. Unfortunately it only understands about 40% of actual lines of code, so this severely limits it's usefulness.

This is stupid. I just want the equivalent of "copy/paste the current line be done with it", so without further ado, here's a macro to do it. You can then bind a keyboard shortcut to the macro, and get on with more important things.

Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Diagnostics

Public Module Misc
    Sub Duplicate_Line()
        DTE.UndoContext.Open("Duplicate Line(s)")
        Try
            Dim ts As TextSelection = DTE.ActiveDocument.Selection
            Dim epStart As EditPoint = ts.TopPoint.CreateEditPoint()
            Dim epEnd As EditPoint = ts.BottomPoint.CreateEditPoint()

            Dim lineText As String = epStart.GetLines(epStart.Line, epEnd.Line + 1)

            epEnd.EndOfLine()
            epEnd.Insert(Environment.NewLine)
            epEnd.Insert(lineText)

            ts.MoveToLineAndOffset(epEnd.Line, epStart.LineCharOffset())
        Finally
            DTE.UndoContext.Close()
        End Try
    End Sub
End Module

PS: Why can't I write VS macros in C#? VB just looks so ugly :-(

Monday, December 29, 2008

Windows 7 beta 1: sound does not work on Macbook Pro (RealTek HD Audio)

This post is googlebait: I couldn't find the solution for this on google, so here's how I solved it. Hopefully others will be spared the messing around.

After installing windows 7 beta 1 (7000) on my macbook pro, and installing the bootcamp drivers off the Leopard Disc, as well as The vista 2.1 bootcamp update, everything worked very nicely... Except sound.

Win7 detected "High Definition Audio Device" and everything looked like it should have worked, but no sound came out of the speakers.

After much mucking around, here's what I did:

Go into the leopard drivers folder. There should be a directory called Drivers, and under that is a file called RealTekSetup.exe. If you try run this normally, it will fail.

What I did next was:

  • Right click it, and select Troubleshoot Compatibility
  • Click Next and wait for it to finish 'Detecting Issues'
  • Select The program Worked in earlier versions of windows...
  • Select Windows Vista
  • Click Next a few times, let the Realtek installer run, reboot, and Presto!
  • As for win7 itself? Well, the beta is faster, nicer, and all around better than vista. I'll never go back. They didn't do a good enough job of copying the dock... but it's still miles ahead of vista, and that's another blog post.
    Byebye!

    Monday, September 29, 2008

    Embedded IronRuby interactive console

    Screenshot!

    What this is, is a small dll which you can add to any .net winforms project. When run, it brings up the interactive console, and you can poke around with your app. It's running live inside your process, so anything your app can do, it can do. I thought this was kind of cool :-)

    How to get it going:

    1. Download and build IronRuby by following the instructions on IronRuby.net - I built this against IronRuby SVN revision 153. As of RIGHT NOW the current revision is 154 which doesn't build.
    2. Download the Embedded IronRuby project from the following URL - you can use SVN to check it out directly from there. (I'm assuming familiarity with SVN in the interests of brevity)
      http://code.google.com/p/orion-edwards-examples/source/browse/#svn/trunk/dnug/ironruby-presentation/EmbedIronRuby
    3. Open the EmbeddedIronRuby/EmbeddedIronRuby.sln file in visual studio, and remove/add reference so that it references IronRuby.dll, Microsoft.Scripting.dll, Microsoft.Scripting.Core.dll, and IronRuby.Libraries.dll. These will be in the IronRuby build\debug folder that you will have built in step 1.
    4. Compile!
    5. For some reason, when you compile, Visual Studio will only copy IronRuby.dll, Microsoft.Scripting.dll and Microsoft.Scripting.Core.dll to the bin\debug directory. It also needs IronRuby.Libraries.dll in that directory (or in the GAC) to run, otherwise you get a stack overflow in the internal IronRuby code when you run it.
      The joys of alpha software I guess :-)
    6. Run the app and click the button!
    You can also add this embedded console to your own app. Just stick all the dlls in your app's folder (or the GAC) so it can see them, add a reference to EmbeddedRubyConsole.dll, and in your app do this: new EmbeddedRubyConsole.RubyConsoleForm().Show();

    Credit: Some of the 'plumbing' code (the TextBoxWriter and TextWriterStream) come from the excellent IronEditor application. Full credit to and copyright on those files to Ben Hall. Thanks!

    IronRuby Presentation!

    I recently gave a presentation to my local .NET user group about IronRuby.

    Click on the image to download the slides as a PDF file.
    Note: This was exported from keynote with speaker notes, which I've revised slightly since giving the presentation.

    As part of this, I demoed a small library I wrote which gives you a live interactive ruby console as part of your running app.

    Basically it lets you poke around your program and modify things while it's running. I'll post the code and notes about that shortly

    Tuesday, July 29, 2008

    Ruby Unit Converting Hash

    I'm currently working on a project where I need to convert from things in one set of units to any other set of units ( eg centimeters to inches and so forth)

    I had a bunch of small helper functions to convert from X to Y, but these kept growing every time we needed to handle something which hadn't been anticipated.

    This kind of thing is also exponential, as if we have 4 'unit types' and we add a 5th one, we need to add 8 new methods to convert each other type to and from the new type

    A few hours of refactoring later, I have this, which I think is kind of cool, and will enable me to delete dozens of small annoying meters_to_pts methods all over the place.

    Disclaimer: This is definitely not good OO. A hash is not and never should be a unit converter. In the production code I will refactor this to build an actual Unit Converter class which stores a hash internally :-)

    
    # Builds a unit converter object given the specified relationships
    #
    # converter = UnitConverter.create({
    #  # to convert FROM a TO B, multiply by C
    #  :pts    => {:inches => 72},
    #  :inches => {:feet   => 12},
    #  :cm     => {:inches => 2.54, 
    #              :meters => 100},
    #  :mm     => {:cm     => 10},
    # })
    #
    # You can then do
    #
    # converter.convert(2, :feet, :inches) 
    # => 24
    #
    # The interesting part is, it will follow any links which can be inferred
    # and also generate inverse relationships, so you can also (with the exact same hash) do
    #
    # converter.convert(2, :meters, :pts) # relationship inferred from meters => cm => inches => pts
    # => 5669.29133858268
    #
    class UnitConverter < Hash
      
      # Create a conversion hash, and populate with derivative and inverse conversions
      def self.create( hsh )
        returning new(hsh) do |h|
          # build and merge the matching inverse conversions
          h.recursive_merge! h.build_inverse_conversions
          
          # build and merge implied conversions until we've merged them all
          while (convs = h.build_implied_conversions) && convs.any?
            h.recursive_merge!( convs )
          end
        end
      end
      
      # just create a simple conversion hash, don't build any implied or inverse conversions
      def initialize( hsh )
        merge!( hsh )
      end
      
      # Helper method which does self.inject but flattens the nested hashes so it yields with |memo, from, to, rate|
      def inject_tuples(&block)
        h = Hash.new{ |h, key| h[key] = {} }
        
        self.inject(h) do |m, (from, x)|
          x.each do |to, rate|
            yield m, from, to, rate
          end
          m
        end
      end
      
      # Builds any implied conversions and returns them in a new hash
      # If no *new* conversions can be implied, will return an empty hash
      # For example
      # {:mm => {:cm => 10}, :cm => {:meters => 100}} implies {:mm => {:meters => 1000 }}
      # so that will be returned
      def build_implied_conversions
        inject_tuples do |m, from, to, rate|
          if link = self[to]
            link.each do |link_to, link_rate|
              # add the implied conversion to the 'to be added' list, unless it's already contained in +self+,
              # or it's converting the same thing (inches to inches) which makes no sense
              if (not self[from].include?(link_to)) and (from != link_to)
                m[from][link_to] = rate * link_rate 
              end
            end
          end
          m
        end
      end
      
      # build inverse conversions
      def build_inverse_conversions
        inject_tuples do |m, from, to, rate|
          m[to][from] = 1.0/rate
          m
        end
      end
      
      # do the actual conversion
      def convert( value, from, to )
        value * self[to][from]
      end
    end
    

    I'm not sure if deriving it from Hash is the right way to go, but it basically is just a big hash full of all the inferred conversions, so I'll leave it at that.


    Update

    Woops, this code requires 'returning' which is part of rails' ActiveSupport, and an extension to the Hash class called recursive_merge!, which I found on an internet blog comment somewhere (so it's only fitting that I share back with this unitconverter)

    Code for recursive_merge

    
    class Hash
      def recursive_merge(hsh)
        self.merge(hsh) do |key, oldval, newval|
          oldval.is_a?(Hash) ? 
            oldval.recursive_merge(newval) :
            newval
        end
      end
      
      def recursive_merge!(hsh)
        self.merge!(hsh) do |key, oldval, newval|
          oldval.is_a?(Hash) ? 
            oldval.recursive_merge!(newval) :
            newval
        end
      end
    end
    

    Code for returning

    class Object
      def returning( x )
        yield x
        x
      end
    end
    

    Monday, July 14, 2008

    HaveBetterXpath

    I'm rspeccing some REST controllers which return XML, and wanting to use XPath to validate the responses.

    I came across this

    http://blog.wolfman.com/articles/2008/01/02/xpath-matchers-for-rspec

    Thanks to him. It worked nicely (couldn't be bothered messing about with hpricot to get that to go), but I didn't like the API as much as I could have.

    Example of that API:

    response.body.should have_xpath('/root/node1')
    response.body.should match_xpath('/root/node1', "expected_value" )
    response.body.should have_nodes('/root/node1/child', 3 )
    

    I didn't like the fact that there were 3 distinct matchers, and that match_xpath didn't work with regexes. I re-worked it, so the API is now

    response.body.should have_xpath('/root/node1')
    response.body.should have_xpath('/root/node1').with("expected_value") # can also pass a regex
    response.body.should have(3).elements('/root/node1/child') # Note actually extends string class and uses normal rspec have matcher
    

    Extending the String class to support elements(xpath) is a win also because it lets you do things like

    
    response.body.elements('/child').each { |e| more complex assert for e here }
    

    Without further ado, new code here:

    
    # Code borrowed from
    # http://blog.wolfman.com/articles/2008/01/02/xpath-matchers-for-rspec
    # Modified to use one matcher and tweak syntax
    
    require 'rexml/document'
    require 'rexml/element'
    
    module Spec
      module Matchers
    
        # check if the xpath exists one or more times
        class HaveXpath
          def initialize(xpath)
            @xpath = xpath
          end
    
          def matches?(response)
            @response = response
            doc = response.is_a?(REXML::Document) ? response : REXML::Document.new(@response)
            
            if @expected_value.nil?
              not REXML::XPath.match(doc, @xpath).empty?
            else # check each possible match for the right value
              REXML::XPath.each(doc, @xpath) do |e|
                @actual_value = e.is_a?(REXML::Element) ? 
                  e.text : 
                  e.to_s # handle REXML::Attribute and anything else
      
                if @expected_value.kind_of?(Regexp) && @actual_value =~ @expected_value
                  return true
                elsif @actual_value == @expected_value.to_s
                  return true
                end
              end
              
              false # our loop didn't hit anything, mustn't be there
            end
          end
          
          def with_value( val )
            @expected_value = val
            self
          end
          alias :with :with_value
    
          def failure_message
            if @expected_value.nil?
              "Did not find expected xpath #{@xpath}"
            else
              "The xpath #{@xpath} did not have the value '#{@expected_value}'\nIt was '#{@actual_value}'"
            end
          end
    
          def negative_failure_message
            if @expected_value.nil?
              "Found unexpected xpath #{@xpath}"
            else
              "Found unexpected xpath #{@xpath} matching value #{@expected_value}"
            end
          end
    
          def description
            "match the xpath expression #{@xpath}, optionally matching it's value"
          end
        end
    
        def have_xpath(xpath)
          HaveXpath.new(xpath)
        end
        
        # Utility function, so we can do this: 
        # response.body.should have(3).elements('/images/')
        class ::String
          def elements(xpath)
            REXML::XPath.match( REXML::Document.new(self), xpath)
          end
          alias :element :elements
        end
    
      end
    end
    
    

    Monday, July 07, 2008

    How to: load the session from a query string instead of a cookie

    We use SWFUpload to upload some images in a login-restricted part of the site.

    There is a problem however, in that we weren't able to get SWFUpload to send the normal browser cookie along with it's HTTP file uploads, so the server couldn't tell which user was logged in.

    The 'normal' solution to this is to add the session key to the query string, and have the server load the session from the query string if the cookie isn't present, only ruby/rails doesn't support doing that.

    a nice guy with the handle 'mcr' in #rubyonrails on irc.freenode.org worked out how to make this work, by patching ruby's cgi/session.rb

    Instructions

    1. Copy cgi/session.rb out of your ruby standard library into your rails app's lib folder
    2. explicitly load the file out of lib, which will then overwrite the built in code

    Needless to say this will stop working if the ruby standard library version of cgi/session changes, but I don't see that as being very likely

    Patch in unified diff format:

    
    --- /usr/lib/ruby/1.8/cgi/session.rb 2006-07-30 10:06:50.000000000 -0400
    +++ lib/cgi/session.rb 2008-07-07 21:07:12.000000000 -0400
    @@ -25,6 +25,9 @@
     
     require 'cgi'
     require 'tmpdir'
    +require 'tempfile'
    +require 'stringio'
    +require 'strscan'
     
     class CGI
     
    @@ -243,6 +246,20 @@
         #       undef_method :fieldset
         #   end
         #
    +    def query_string_as_params(query_string)
    +      return {} if query_string.blank?
    +      
    +      pairs = query_string.split('&').collect do |chunk|
    + next if chunk.empty?
    + key, value = chunk.split('=', 2)
    + next if key.empty?
    + value = value.nil? ? nil : CGI.unescape(value)
    + [ CGI.unescape(key), value ]
    +      end.compact
    +
    +      ActionController::UrlEncodedPairParser.new(pairs).result
    +    end
    +
         def initialize(request, option={})
           @new_session = false
           session_key = option['session_key'] || '_session_id'
    @@ -253,6 +270,7 @@
      end
           end
           unless session_id
    + #debugger XXX
      if request.key?(session_key)
        session_id = request[session_key]
        session_id = session_id.read if session_id.respond_to?(:read)
    @@ -260,6 +278,12 @@
      unless session_id
        session_id, = request.cookies[session_key]
      end
    +
    + unless session_id
    +   params = query_string_as_params(request.query_string)
    +   session_id = params[session_key]
    + end
    +
      unless session_id
        unless option.fetch('new_session', true)
          raise ArgumentError, "session_key `%s' should be supplied"%session_key