Thursday, October 24, 2019

How to pin a custom certificate trust chain and DNS name in C#

This code tested on dotnet core 3.0, however the API's are stable and this most likely works across all dotnet versions


// create a single long-lived HttpClient, that's what microsoft's documentation says to do
var handler = new HttpClientHandler {

    ServerCertificateCustomValidationCallback = (message, certificate, chain, policyErrors) => {
        // first check the DNS name in the certificate
        var expectedHostName = new Uri(host).DnsSafeHost;
        var certificateDnsName = certificate.GetNameInfo(X509NameType.DnsName, forIssuer: false);

        if (!certificateDnsName.Equals(expectedHostName, StringComparison.InvariantCultureIgnoreCase))
        {
            log.LogError($"Server Certificate rejected due to invalid subject name." + 
               " Certificate provides {certificateDnsName}, expecting {expectedHostName}");
            return false;
        }

        chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
        chain.ChainPolicy.ExtraStore.Add(caCertificate);

        // refer https://social.msdn.microsoft.com/Forums/vstudio/en-US/1966a6e8-b6f4-44d1-9102-ec3a26426789/how-can-i-verify-a-certificate-manually-without-installing-its-parents?forum=clr
        // we can't have a "trusted" root, because we're not affecting the machine-wide trust store, however
        // if a cert correctly chains back to our CA, then the only "error" will be a single untrustedRoot error, which means
        // everything else was good, just the root wasn't in the machine-trust store, so we're good

        // Note that if the chain terminates at a different root certificate that is trusted by windows, we
        // do NOT want to accept that cert, but it will log something like Validation errors: None which may look a bit weird.
        // Also, For future consideration: Do we want to allow customers to specify multiple extra CA's to handle server migrations?

        // If the CA is in the windows trusted roots, this will return true and chainStatus will be empty
        // If the CA is not in the windows trusted roots, this will return false and chainStatus will contain UntrustedRoot. Either are acceptable
        var chainOk = chain.Build(certificate);
        var chainOkUntrustedRoot = chain.ChainStatus.Length == 1 && chain.ChainStatus[0].Status == X509ChainStatusFlags.UntrustedRoot;

        // PROVIDED the chain terminates in the correct CA, and not some other CA that might randomly be in the windows trusted roots
        var chainTerminatesInCorrectCA = chain.ChainElements[chain.ChainElements.Count - 1].Certificate.Thumbprint == caCertificate.Thumbprint;

        if (chainTerminatesInCorrectCA && (chainOk || chainOkUntrustedRoot))
        {
            return true;
        }

        log.LogError("Server Certificate rejected due to invalid trust chain. Info: {0}", string.Join("", chain.ChainStatus.Select(c => c.StatusInformation))); // statusInformation has \r\n at the end itself
        return false;
    }
};

var client = new HttpClient(handler) {
    BaseAddress = new Uri(host)
};

The above code will pin a specific custom Certificate Authority. It's useful for internal deployments where you might have an internal certificate authority provided by your IT department, and you
want to ONLY trust certificates issued by that authority (and not random other certificates from LetsEncrypt, etc)

If you want to just pin the certificate itself then you don't need to verify the chain or anything like that.

Thursday, October 25, 2018

Native Mobile apps for iOS and Android; Writing it twice doesn't have to cost 2x

This post inspired by René Wiersma on Twitter:



As (I think I have) mentioned previously, my team is responsible for the company's Mobile Applications (and supporting cloud/on-premise services they connect to).

We need to support both iOS and Android as equals - in almost all cases we'll ship new features for both iOS and Android simultaneously, so the problem of developing for both of these different platforms keeps coming up; we've been doing this for the last 4-5 years.

We're not using any cross platform tools, so how do we do it? Simple, we write the app twice - once for each platform. For iOS we use Xcode and Swift (older code being ObjC) and for Android we use Android Studio and Kotlin (older code being Java).

Note: a key overall goal here from day one has been to try keep the code structurally similar, so that when you're fixing a bug on one platform, it's easy to find the corresponding piece of code to fix on the other platform. There is a tension here though against using the different abilities of each platform.

Original plan: One developer creates a major feature for a platform, another creates that same major feature for the other platform in parallel

Note: When I say "whole feature" I'm talking about a significant chunk of work - at least a month's worth.

We thought at the time that this would work, and perhaps the following might happen:
  • Each developer could work (and up-skill) on their preferred platform
  • Devs could work for a longer time without context-switching which would be good for productivity
  • We thought this would be OK as each dev would code-review the other's code, and they would both be involved in design reviews.

It was awful. I felt it took more than 2xtime/cost and generated technical debt like crazy..
  • Both devs would arrive at the same problems, and both would come up with designs to solve them. Usually these designs were different. This caused a whole lot of discussion and back-and-forth arriving at a consensus. The thing was - in almost all cases either design would have worked fine, and this discussion wasn't ultimately productive.
  • The devs didn't code review each-other as strictly as perhaps they should have, and when they did, their code had diverged enough that they simply reviewed it for correctness and didn't want to spend additional time going back over it and bringing it into line with the other platform.
  • And besides - if we need to align two diverged code-bases, whose code is the right code to align with?

Next plan: One dev creates a minor feature for a platform, then another dev does it for the other platform on a delayed basis; the feature is mostly complete by the time the second dev starts

We sort of fell into this due to devs coming free from other tasks at different times, but after the previous bad experience, we decided to make a deliberate go at this.
Note: our "minor feature" here is about 2-3 weeks or more. We simply could not use the delayed approach at the "major feature" level as it would have added so much time to the final delivery date that we wouldn't have been able to ship both platforms together on time.

This was only a little better. I felt it took about 2x time/cost
  • We thought the delay would prevent the "two people designing the same thing at the same time" problem and be more efficient. This did turn out to be true, BUT
  • The second developer burned a significant amount of time having to go back and learn how the first developer had done everything
  • Because the second developer didn't have the original context, when they had to make a different choice due to a platform difference, they didn't fully understand the code/design and would be more likely to make bad choices
  • The second developer ended up writing structurally quite different code just due to personal preference, which is a nasty trap for maintenance in future
  • This is terrible for testing. It makes sense for a tester to pick up the feature on both platforms at the same time, so they can check for consistent behaviour, etc.

Take 3: One developer creates a small sub feature for a platform, then another developer does it for the other platform after the first is pretty much done

The idea here was to try and minimise the bad effects caused by the large delay of the previous approach. We aimed for sub features being around 3 days or less worth of work, but sometimes up to a week.

This was better again, but still not great. I felt it had about 1.8x time/cost
  • I wasn't sure what would happen with designs, but it turns out even a small delay stops two people clashing trying to design the same thing in different ways, which was good.
  • The smaller delay meant that the second developer didn't have as large a chunk of work to catch up on
  • When the second developer needed the original context, it was relatively fresh in the mind of the first dev, so better discussions could be had
  • Testing waited til the second dev finished, but that was usually only a couple of days
  • BUT, the second dev still doesn't fully understand the problem in the same way the first dev did, so they still burn time coming up to speed constantly
  • ALSO, the second dev ends up using a different style/structure which hurts maintenance down the line. This is a big deal

Take 4: One developer creates a small sub feature for a platform, then that same developer creates it again for the other platform.

The idea was to try address the "second dev missing context and coming up to speed" problem.
Now we're getting somewhere. I feel this is about 1.5x cost overall
  • This means all the devs must learn to work on both platforms. If you're the kind of place that wants specialist iOS-only and Android-only developers you may not like this. Personally, I think pigeonholing people like that is terrible in any environment (Don't get me started on frontend vs backend developers), but hey, my team is small - perhaps when you have 50 devs you need to silo them like that, who am I to say?
  • This *is* effective; if it's the same dev doing both platforms, then they inherently understand the problem equally both times, and nobody else has to second-guess their decisions. Seems obvious, doesn't it.
  • BUT, when the dev would do the feature for the second platform, they would sometimes structure their code differently out of trying to better fit the other language/platform, or simply because it was a different week and they felt differently. We still had the maintenance pain down the line. Ouch.

Fifth time lucky: One developer creates a small sub feature for a platform, then that same developer ports the code in as literal a way as possible.

Goal: Stop the code diverging as much due to the devs being creative when writing the feature for the second.

Process: Write your small sub-feature for a platform, then open File.swift on one monitor and File.kt on the other, and go through function-by-function and line-by-line; Copy paste and then fix the errors and tweak the code.
The code inside each function doesn't have to match perfectly, but structural similarity is key here.

This takes discipline, but it is the best way we have discovered so far. It feels like it gets the cost/time down to about 1.3x
  • To quote a team member: "Porting code is boring", so that's a downside
  • Boring though it may be, porting code like this is really fast and often easy and safe. Things don't get missed nearly as much.
  • MAJOR Upside: If your code is in the same-named files, with the same-named methods in the same order, then when you make a fix on one platform it's very very easy to make that exact change safely on the other platform
  • Kotlin and Swift are much more similar than ObjC and Java, so now that we've moved to the newer languages this process is a lot easier than it used to be.

Conclusion

So here you have the recipe:
  • Break your tasks down into as small as possible sub-features
  • Have ONE dev do the sub-feature on one platform
  • Have the SAME dev do a "boring" port of that code over to the other platform. Don't be clever about it!
  • Enjoy :-)
Personally, I feel this is quite good. I've spent a good chunk of time using Xamarin and while the "porting code" cost is zero there, it has it's own problems and overheads. I feel like writing your app in Xamarin is about a 1.2x cost compared to writing a single-platform app in either native Swift or Kotlin, so to be able to write both native swift and kotlin apps with a 1.3x overhead seems pretty darn good to me! Note: the cost ratios are purely subjective based on my experience writing code in these various ways, and observing other team-members doing it too.
Note: We have definitely looked at various cross-platform tools, but found them all lacking:
  • Web-based (either PWA or cordova-type): Seriously evaluated and for what we wanted, we couldn't get good enough performance and nice enough UI
  • Xamarin: Seriously evaluated and found it to be really buggy, and also quite an additional burden to learn not only how the UI works (UITableViewController etc) but the quirks and oddities of how those things project into C#.
  • Flutter: Played around - maybe promising in future, but I think it's too early to depend on yet
  • React Native: Only read about it; Personally I can't get past the JavaScript. Ugh. Some people say that even for a single-platform app react native is more productive than Swift or Kotlin, and if this is true (while still maintaining the level of quality and performance we want) then it could be very compelling. It's on the list to investigate soon

Monday, October 01, 2018

Beware of locking helper functions in swift

If, like me, you are a swift programmer with a background in basically any other programming language, you'll be familiar with the lock statement.

C# Example:

class MyClass {
    private bool m_initialised = false;
    private int m_value = 0;

    void Initialise() {
        lock(this) { // lock stops two threads initialising at the same time
            if(m_initialised) {
                return; // already initialised
            }

            m_value = LoadExpensiveValue();
            m_initialised = true;
        }
        Console.WriteLine("initialised");
    }
}

Java has synchronized, heck even Objective-C has @synchronized - but swift has no such thing.

If you google for it, you most probably wind up on stack overflow, probably at this article: What is the Swift equivalent to Objective-C's “@synchronized”?.

It boils down to the following

1. Use objc_sync_enter to lock
2. Use objc_sync_exit to unlock
3. Don't forget to use defer to make sure the unlock always happens

Now, if you simply do the above, it's fine, however you (like me) are most probably tempted by that answer on that stack overflow post, which shows that you can write your own synchronised/lock function. Once you apply all the bells and whistles, you get something like this

Swift:

func lock(lockObj: AnyObject, closure: () throws ->T ) rethrows -> T
{
  objc_sync_enter(lockObj)
  defer { objc_sync_exit(lockObj) }
  return try closure()
}

And you can use it like this!

Swift Example:

class MyClass {
    private var _initialised = false
    private var _value = 0

    func initialise() {
        lock(self) { // lock stops two threads initialising at the same time
            if(_initialised) {
                return // already initialised
            }

            _value = loadExpensiveValue();
            _initialised = true
        }
        print("initialised");
    }
}

How cool is that? Swift didn't have a lock keyword, but we were able to write one ourselves, and it's just as nice to use as the native ones from C#, Java or anywhere else! Hooray!

Unfortunately, it is also wrong and broken

The above code compiles with zero errors or warnings, and there are many cases in which you'll never realise there's a bug there. I was bitten by this exact thing just today, hence my motivation to write this post.

The bug is, in the swift implementation, the print("initialised"); runs when _initialised is true; If two threads call the function, they will *both* print it.

Why? In swift, our lock keyword isn't a real keyword. It is a function which is being passed a closure.
That is to say, the locked region is a closure, and not part of the outer function.

The return statement for the "already initialised" case returns from the closure, but does not return from the outer initialise() function itself. This is in contrast to languages with a real lock keyword, which does not introduce a new return scope, and where the return statement actually returns from the outer function.

My advice: Do not write or use this handy swift lock function. One day you too will write this bug, and when it occurs it's not at all obvious as to what's going on.

Do this instead.


class MyClass {
    private var _initialised = false
    private var _value = 0

    func initialise() {
        do {
            objc_sync_enter(self); defer{ objc_sync_exit(self) }

            if(_initialised) {
                return // already initialised
            }

            _value = loadExpensiveValue();
            _initialised = true
        }
        print("initialised");
    }
}

The code is very slightly longer and very slightly uglier, however there are no closures, and the return keyword works how you expect it to.

Perhaps one day swift will add a proper lock keyword and we can all move on from this kind of thing

Tuesday, November 21, 2017

Real-world native mobile development. From ObjC to Swift and from Java to Kotlin

My team is responsible for our Company's Mobile Apps - Our biggest app is a reasonably sized one (39 ViewControllers at the current time of writing, with about a dozen significant "service" components) with both iOS and Android versions which goes back a few years. The first version was released in about 2013 off the top of my head.

Historical iOS Development in Objective-C

We originally developed the App in question for iOS only, targeting Objective-C (This was before Swift existed.) Later on we built up an Android version using Java (again, predating Kotlin which is only a "stable" option for Android development in the last 6 months or so).

We were quite excited when Apple announced Swift, and immediately played around with it, but we didn't consider it seriously until Swift 2 - We started to port small parts of our iOS app over to Swift 2 in mid 2016 - probably about 4 months before Swift 3 was released. Swift 2 / Xcode 7 was definitely unstable, however we found Swift 3 / Xcode 8 to be pretty stable and nice to work with, and I definitely feel like with Swift 4 and Xcode 9 is mature and reliable now.

Why port Objective-C to Swift?

We felt that we would very much like the benefits of Swift (strong typing, null safety, nicer syntax, etc). As such, we have an informal policy that all new code should be written in Swift (within reason).

While there's a strong argument to be made that old code isn't broken and should be left alone, nobody on the team really wants to have old stuff sticking around forever, and everyone's in agreement that in a perfect world, the entire app would be Swift. Porting everything in one go would be an enormous time investment and generate a huge pile of bugs, so we're not doing that at all. Instead the policy is (informally)

  1. If you're doing new code, do it in Swift
  2. If you're between tasks, and you feel like you want to port some code to Swift, go forth and just do it.
  3. If you're working on some old Objective-C code, you should seriously consider porting it to swift while you're at it

Between 2 and 3, I've found that over the last year or so the app is slowly but surely transitioning it's way over to more and more Swift. We'll still have Objective-C for at least another year or maybe two, but I can definitely see it going away eventually.

Thoughts on Swift

  • Null-safety is fantastic, however there is definitely a learning curve. I found everyone on the team had no problems dealing with the concept of null safety, it's just a bit of a job to learn all the syntax and rules and how they apply everywhere (i.e. nullable generic types and stuff like that)
  • Swift's more concise syntax is great, compared to Objective-C. It's hands down nicer than Java or C# or anything else (except maybe Kotlin, see below)
  • Swift protocols with associated types are weird and very complex. I feel like I understand what's going on and why they're done that way, but I find this very hard to explain to anyone else in a satisfactory manner. The only saving grace is that you tend not to have to use PAT's very much
  • Having programmed C# and C++ for a long time too, I very much like having generics and strong typing available in Swift (Compared to Objective-C)
  • Supporting "mixed projects" where you can have Swift and Objective-C files side by side in the same binary really is a killer feature for porting and upgrading.
    Our company has huge amounts of C++ and C# code, and if Microsoft had somehow pulled off same-project-different-language-per-file compatibility like Apple has here, the world would be a truly different place. It's quite likely all our C++ would have been replaced by C# years ago, rather than hanging around forever. (Now, since C++11 has revitalised the C++ world, it's no longer in danger of going away, but in the mid to late 2000's things were indeed looking dire for C++ for us)
  • Overall, I find Swift nicer to work in than Objective-C, Java or C#, however it is absolutely a much more complicated language. The upside of Swift borrowing lots of features from lots of other languages is that it offers many great, modern options for problem solving and development of safe and high quality code. The downside is there's a boatload of stuff that you have to learn, which has caused basically every developer on my team who's encountered it to feel incompetent for the first few months.

Android Development

As mentioned previously, we'd done our Android stuff entirely in Java. I personally am not a fan of Java, as I find it more verbose and clunky when compared to other languages, however I must say it hasn't been nearly as bad as I thought it would before we started it. A lot of this comes down to the fact that the language itself is really only about 30-40% of the development experience. The rest comes down to platform API's, and while I find Android definitely lags behind iOS in this area, it's still a darn sight better than the standard Java stuff I'd done in the past.

When Google announced that Kotlin was going to be officially supported in Android Studio 3.0, like with swift, I was quite excited. I'd seen kotlin before, but never really taken it seriously, and I'd lumped it in with other "experimental" languages like Groovy, Scala and Clojure that look awesome on paper, but don't look stable/widely supported/used enough to use for "real" development. Google promising official support immediately catapulted kotlin out of this group and into the "big league" - at least that was my feeling anyway.

Since Android Studio 3.0 has released, we've started porting parts of our app over to kotlin, following a similar process to what we did with Swift. As with our Swift port, we had pretty much the same reasons - wanting to take advantage of the nicer syntax, strongly safety guarantees and stuff like that. Refer to the "Why port to swift" section above.

Thoughts on Kotlin

  • As with swift, null safety is great. We had been extensively using @NonNull attributes all over our Java already, but Kotlin's support for working with nullable types with smart casting and the ? operator really make a big difference.
  • I definitely appreciate Kotlin's more concise syntax as compared to Java. Type inference is great.
  • Kotlin supports the same-project-different-language-per-file which (as with Swift) a killer feature for porting code. While the auto-converter is certainly far from perfect, it provides an enormous head-start when porting code.
  • I'm stunned by how good Android Studio (IntelliJ) is when dealing with Kotlin. I was expecting it to be like Xcode/Swift were, and have lots of bugs and missing features for the first year or two, but my experience has been pretty much since the Android Studio 3.0 release that it's of equal quality to the Java support. Hat's off to JetBrains here
  • Kotlin has some weird stuff though.
    - WTF Companion Object?
    - Wherefore art thou ternary operator? (Yeah you can use inline if/else, but that looks weird, and even Java has a ternary operator)
    - Returns from lambdas are bonkers. return in a lambda should return from the lambda as that's the common and usual case - the wonky return@thing syntax should be reserved for explicit non-local returns which are out-of-the-ordinary. I feel like they got this entirely backwards
  • Kotlin gets burned by it's extensive Java compatibility a bit - mostly because it inherits the same broken type-erased generics that Java has, which kind of sucks

Kotlin vs Swift

Now here's the best part of any programming blog! Comparing things so we can say which is better. Alas it's not so simple, as they both have ups and downs relative to each other. Here's some things in no particular order.

Syntax

I would personally give a very close win to Swift, but you could go either way here. Both are close (see Swift is like kotlin). Kotlin just has slightly more things which are weird for no seeming purpose. My pet peeve is the val keyword. It's visually incredibly similar to var, and there was strong precedent from many other languages to use let like Swift did, but Kotlin's just made a weird call here. On the other hand, Kotlin does have a lot more 'syntactic-sugar' things built in, which is nice.

Backward Compatibility/Interop

Clear win to Kotlin. One thing which surprised me is how close Kotlin actually is to Java. Not syntactically, but at runtime the majority of Kotlin types are just projections or wrappers around pre-existing Java types like String, HashMap, ArrayList or whatever, which I've found makes Java interop seamless and obvious in 99.9% of cases. Swift on the other hand departs significantly from the Objective-C runtime, defining it's own String, Array and Dictionary types, amongst other things, which can sometimes make things tricky. As evidence of this, the latest version of the Kotlin standard library which you must bundle with your app is 861KB, whereas the Swift 4 stdlib is something like 20+ MB. It's not a fair comparison, as Swift is native code and not JVM bytecode, and also a lot of the Swift std lib is stripped out of your app if not needed, but it does clearly show the point that comparatively, Swift needs a lot more stuff over and above the baseline platform to function.

Modern features/New Paradigms

Clear win to Swift. While I find Kotlin to be great to work with, it doesn't try to depart from Java nearly as much as Swift does from ObjC. As above this makes for much better interop and is easier to learn, BUT there's not as much opportunity to push things forward. Take for example Swift's value types. They can enable paradigms and ideas that you simply can't express in Kotlin. Likewise, Generics in Kotlin are held back by Java compatibility and so are half-assed like the Java ones. Swift makes the choice to break compatibility (Objective-C code literally can't see Swift generic code at all) in order to have "proper" generics without doing things halfway. I find swift to be a fundamentally different thing to Objective-C (or even C#/Java) whereas Kotlin doesn't seem to enable any fundamental new things that you couldn't already do in Java - everything's just much much nicer.

Learning curve

Java->Kotlin is way easier than Objc->Swift. See above. Even though Kotlin has more crazy stuff like companion object, the fact that there's not as much "distance" between it and Java outweighs it all.

App/Runtime stability

Both have been great so far.

IDE/Development Experience

Definite win to Kotlin. I think Swifth with Xcode 9 is definitely good, but the work JetBrains have done for IntelliJ/Android studio is simply fantastic (Around language support anyway. Some of the other things in IntelliJ like Build Configurations and the Git integration are awful, but it's unfair to penalise Kotlin for that)

Compilation Performance

Clear win to Kotlin. It seems a bit slower to compile than Java, but Swift compilation times are more in line with C++ (i.e. Ages)

Productivity

I'd probably say Swift here but you could go either way. They honestly both feel very good, and so much of productivity is subjective and down more to the platform API's than the language itself.

Runtime performance

Both are great. While in theory Swift should be slightly faster than Objective-C and Kotlin should be slightly slower than Java, in practice for all normal app-level stuff we do like rendering UI, network requests, JSON parsing and so forth, I can't tell the difference between code written in ObjC vs Swift or Kotlin vs Java. Because they both have great interop, if you did happen to have a piece of code which actually was performance sensitive, you could just write that one small piece in the other language anyway. If you're worried about performance issues from moving to Swift or Kotlin, don't be.

Porting code

A LANDSLIDE win for Kotlin here. It's relative closeness to Java, combined with the auto-converter in Android Studio make it fairly easy to port most things. The auto-converter screws things up quite a bit, but when it goes wrong I've found the problems tend to be quite easy to fix, once you've done it a few times and learn the patterns of how it goes wrong. Swift doesn't have an auto-converter, so you need to go through your objective-C code and manually port across things line by line. For example, I ported a significant chunk of code from Java to Kotlin in about a day. That same code would have taken me at least 2 days to do ObjC->Swift, most likely 4+.

I have an Objective-C app, should I port it to Swift?

Maybe, however it will be a fair bit of work spread out over a long time, and will bring about a lot of change. Most of the changes will improve your code and make it more readable, stable and maintainable, but some will not. In particular anywhere you depended on the dynamic behaviour of Objective-C. My team and I chose to port ultimately because we like the language better - mostly for the cleaner syntax and static typing. Compared to ObjC, it's more similar to C# (which everyone on the team had prior knowledge of), so that helps too. These reasons don't hold for everyone though. Porting to Swift seems more like a long term thing - Apple is pushing the community hard towards Swift, and it is following. There will come a time (if it hasn't already) where having objective-C code means it's harder to find developers, harder to find code samples, help and documentation online, and other such things, so it makes sense to move to swift for those reasons.

I have a Java app, should I port it to Kotlin?

Yes>. To me, it seems like a no brainer. Kotlin is better than java in many many ways, but doesn't seem to have really any downsides. You (like me) may not like some of the weird things it has like Companion Object, but in my experience those are all fairly superficial and you can get past them without much worry. The auto-converter makes porting your code take a lot less time, and it's actually quite fun.

Conclusion

If I had to sum it up, it goes back to my point before that Kotlin really does feel like super nice modern Java. I like it and it's great to work with but it doesn't bring any significant new paradigms to the table.

Swift on the other hand is a significant departure Objective-C, and with Structs, ARC, and in the future the Memory ownership model stuff, seems to be moving more into the space of Rust and things like that. Don't forget that Objective-C is a dynamic language and actually has more in common with Ruby or JavaScript than you might think.

What's my preference? Superficially Kotlin and Swift are very similar.
Assuming you could somehow magically eliminate all the platform specific differences between iOS and Android - say perhaps you were writing server-side software on linux or something - I think I'd chose Swift. There's a variety of reasons, but the thing that I guess I place the most weight on is the modern features/paradigms (see above). From that point of view, Kotlin and Swift are actually very different.

Tuesday, February 09, 2016

MiniRxSwift

I wanted to provide asynchronous networking for my StockFighterApiClient library, but I had 2 problems

  • Callback-based asynchronous programming is not nice. It quickly leads to nested callback hell
  • The Reactive Framework provides a much nicer async model, and RxSwift is great. I've used it in other projects and I much prefer it to ReactiveCococa, however RxSwift is a multi-megabyte download with dozens of source files, and I want to keep the ApiClient library as self contained as possible
So, I wrote a small miniaturised version of RxSwift which only contains a few of the core operators.

RxSwift and ReactiveCocoa get bloated by UI bindings and stuff like that which are completely unnecessary for the core Rx functionality, so it's great not to have any of those

It's a single file, approx 400 lines including comments and whitespace, and you can find it here:
https://github.com/borland/StockFighterApiClient/blob/master/MiniRxSwift.swift

It is missing a few things which might end up being important, but even if those get added it will still remain a single file and I doubt it'd ever get over 1000 LOC.

While this is silly wheel re-inventing, and in a serious project I'd just use RxSwift, I found that there were some benefits that came of it.

Protocol Associated Types

It was the first time I'd used protocols with associated types in Swift before. Implementing Observable/Observer as protocols in swift was an interesting journey learning about how associated types worked, and how they are different to generics.

Overall I found associated types to be profoundly strange compared to anything I'd used before in other languages.
I can appreciate that associated types can be more powerful and flexible than generics, but I couldn't help feeling that I didn't actually want or need that extra power/flexibility, and the cost and complexity it imposes (having to create type-erased wrappers such as Apple's AnySequence, or my Observer class) was annoying.
I can't help but think that this is going to be somewhat of a pain for people as they learn swift. I can imagine trying to explain to a new developer on my team why we have to write wrapper classes to bridge between protocols and generics, and I'm pretty sure that's going to be a very confusing and difficult to grasp situation for a newbie.
At any rate, I learned a fundamentally new (to me) concept. That doesn't happen very often once you get past your first few years of programming, so it's been well worth the effort.

Learning/Refreshing Rx knowledge

At work, we've got a lot of C# code that uses the Reactive extensions for .NET. I initially learned Rx way back when it was a pre-release binary dll that Microsoft shipped (I vaguely recall) privately as a helper library in some version of Silverlight or something like that.
I learned it primarily by reading the decompiled source code using Reflector, as there wasn't any documentation or even an official download of Rx for quite some time.

This ended up being great for me, as I got a deep understanding of exactly what is going on behind the scenes for each method and task. No matter how good the documentation for something is, there's no substitute for knowing that (for example) a method takes a lock just before it does some critical thing and when it releases it.

I was only really able to do this though, because Rx back then was much smaller and simpler. The Rx of today has gone through many years of Architecting, Refactoring, Indirecting, Abstracting and Organising. While I've no doubt that it's benefitted the project overall, the downside is that everything is now very hard to find and to learn.
For example, take a look at SelectMany - It's 1739 lines of code just for that class, and to understand it you also need to know how it's base class works.
In contrast, the old original code for SelectMany (known as flatMap in languages other than C#), was a single function, and looked a lot like the one in MiniRxSwift which is entirely self contained, has no classes, and is under 40 LOC including whitespace.
The real RxSwift has taken the approach of porting from C#, so it keeps mostly the same class hierarchy and complexity as the C# implementation, and is equally hard to find your way around.

Hopefully someone one day can look at MiniRxSwift and get a better understanding of Rx

P.S.

I wrote most of this from my understanding of how Rx works. I did look at the Rx documentation, and I definitely looked at the RxSwift source code along the way - not to copy it - but to learn how they were doing things with Protocol-Associated-Types. The swift compiler produces a lot of confusing error messages if you get your generics/PAT bridging code wrong and I wouldn't have been able to do it without some reference code to learn from.

Thanks RxSwift!


Thursday, January 28, 2016

StockFighterApiClient

TL;DR

I wrote a client library in Swift for StockFighter. It's on github at https://github.com/borland/StockFighterApiClient, feel free to use it or check it out!



If you're a programmer and you haven't heard of it, I'd recommend checking out StockFighter.
It's a "programming game" wherein you have to write code to perform some task in a correct/fast/clever/etc way. In this specific instance, the goal is to write bots to perform high-frequency stock trading (well, actually pretty slow frequency trading compared to real stock markets, but much faster than a human would).

I'm midway through the third level so far, but I've found it quite interesting and challenging to think about some of the strategies for solving some of the problems.

The way it works is you write some code to connect to their cloud servers via a REST api (and/or WebSockets) and have your bot buy and sell stocks on their fake stock exchanges. Different levels have different challenges.

I'm a big fan of Swift at the moment, so I wanted to use it for stockfighter. Swift has

  • Nice syntax and a nice high level - I find swift code to be mostly similar to C#, but with less semicolons and brackets which is always great, and it's often more concise. Code is roughly about the same size as it would be in Ruby or JavaScript.
  • Decent functional programming style support - It has nice lambda functions, things like map/filter over collections are built in and you can extend existing types to add more complicated methods as needed)
  • Null safety - The compiler can enforce that things may or may not be null which is a bit weird at first, but once you get used to it is very nice. I find on average my swift code has a much higher chance of "just working on the first run" primarily due to this.
  • An exception model which I like - I always found Java's explicit checked exceptions to cause too much code bloat, especially having to declare catch blocks for multiple different exception types all over the place. At the same time, I've sometimes found the "transparent" exceptions of C# or C++ aren't as nice because often you simply won't be aware that a method could throw an exception until you find out later in a crash report. The swift model strikes a middle ground, and while it has some shortcomings too, on balance I like it better than either of the other approaches
  • Good performance - It seems to be roughly in the ballpark of C# or Java, but uses less memory and starts up faster due to lack of a "runtime" and garbage collector, which is great
Anyway, I looked online and found one existing client library - RSStockFighter, however, I wanted to write my own client library from the ground up for fun, and for practice.

Additionally RSStockFighter uses a "command object" pattern, wherein you create different kinds of objects representing each different request you might want to make, then you submit them to the server. While such a pattern offers many benefits and is probably good OO design, I find it adds friction to the programming experience and is not what I'd like to use (at least for a project the size of a stock fighter client. It might make more sense when you have an API with hundreds of methods).

Rather than this:

let order = StockFighterOrder(symbol:SF_STOCK_SYMBOL, price:95.00,qty: 100,direction:SF_OrderDirection.Buy,orderType:SF_OrderTypes.Limit.rawValue)
       
StockFighterHelper.sendStockFighterTransaction(order, success: {
    (dict:NSDictionary!) -> Void in
        self.checkOrderResponse(dict)
}, failure: {(str: String) -> Void in
        print(str)
})


I'd much rather write this:

let response = try venue.placeOrderForStock(SYMBOL, price: 9500, qty: 100, direction: .Buy)

HTTP calls in my library are synchronous, and either return a result or throw an exception, which generally makes the code shorter and clearer.

Personally I prefer asynchronous I/O, however then you have to deal with callbacks. The Reactive frameworks - RxSwift in this case - provide a much nicer model than the nested-callbacks approach, so I think if I were to provide async methods I'd use RxSwift - however that means if anyone wants to use my Api they must also take a dependency on RxSwift, so I left it out for now.

My library also provides access to the StockFighter web socket API using SocketRocket behind the scenes. I embedded a copy of SocketRocket in my github repo so there's only one thing to download, but adding third party frameworks such as SocketRocket to a project in Xcode is a bit of a pain. I've documented the process in my README, but it's still not ideal.

I didn't want to use something like CocoaPods / Carthage because they are yet another barrier to entry. I don't want to ask people to go and cruft up their system and install some magical dependency manager which is going to do who-knows-what when I could offer people a simpler "copy some files" option instead. Perhaps that's a topic for another post in future.

Anyway. Head over to github and check it out if you're interested. It's on github at https://github.com/borland/StockFighterApiClient and I'd appreciate any feedback!

Wednesday, March 25, 2015

Using ReactiveCocoa from Swift

We're trying out swift at work, and have an existing app which makes quite a bit of use of ReactiveCocoa.
Looking online, I found MVVM, Swift and ReactiveCocoa - It's all good! from Colin Eberhardt. It's a good intro, but I found I had trouble when trying to use the tricker subscribe overloads.

Without further ado, here's my own take on the extensions.
Licensing wise I declare it to be public domain, do with it as you will.
(Note: Implemented on Xcode 6.2 with Swift 1.1)


import Foundation

extension RACSignal {

    // Subscribe with an on-next block
    // Equivalent of - (RACDisposable *)subscribeNext:(void (^)(id x))nextBlock {
    func subscribe(next nextClosure:(T)->Void) -> RACDisposable {
        return self.subscribeNext { (x: AnyObject!) -> () in nextClosure(x as T) }
    }
    
    // Subscribe with an on-next and on-completed block
    // Equivalent of - (RACDisposable *)subscribeNext:(void (^)(id x))nextBlock completed:(void (^)(void))completedBlock;
    func subscribe(next nextClosure:(T)->Void, completed completedClosure:()->Void) -> RACDisposable {
        return self.subscribeNext(
            { (x: AnyObject!) -> () in nextClosure(x as T) },
            completed:{ () in completedClosure() })
    }
    
    // Subscribe with an on-next and on-error block
    // Equivalent of - (RACDisposable *)subscribeNext:(void (^)(id x))nextBlock error:(void (^)(NSError *error))errorBlock {
    func subscribe(next nextClosure:(T)->Void, error errorClosure:(NSError)->Void) -> RACDisposable {
        return self.subscribeNext(
            { (x: AnyObject!) -> () in nextClosure(x as T) },
            error:{ (x:NSError!) in errorClosure(x) })
    }
    
    // Subscribe with an on-next, on-error and on-completed block
    // Equivalent of - (RACDisposable *)subscribeNext:(void (^)(id x))nextBlock error:(void (^)(NSError *error))errorBlock completed:(void (^)(void))completedBlock;
    func subscribe(next nextClosure:(T)->Void, error errorClosure:(NSError)->Void, completed completedClosure:()->Void) -> RACDisposable {
        return self.subscribeNext(
            { (x: AnyObject!) -> () in nextClosure(x as T) },
            error:{ (x:NSError!) in errorClosure(x) },
            completed:{ () in completedClosure() })
    }
    
    // Subscribe with an on-error block
    // Equivalent of - (RACDisposable *)subscribeError:(void (^)(NSError *error))errorBlock;
    func subscribe(error errorClosure:(NSError)->Void) -> RACDisposable {
        return self.subscribeError { (x: NSError!) -> () in errorClosure(x) }
    }
    
    // Subscribe with an on-error and on-completed block
    // Equivalent of - (RACDisposable *)subscribeError:(void (^)(NSError *error))errorBlock completed:(void (^)(void))completedBlock;
    func subscribe(error errorClosure:(NSError)->Void, completed completedClosure:()->Void) -> RACDisposable {
        return self.subscribeError(
            { (x: NSError!) -> () in errorClosure(x) },
            completed:{ () in completedClosure() })
    }
}

Now, how to use it (this can be a bit unclear sometimes in swift)

To subscribe for on-next notifications


someSignal.subscribe { (arg:Type) in .... }

e.g. with an RACSignal publishing strings it would be
someSignal.subscribe { (x:String) in doSomething(x) }
or
someSignal.subscribe(
    next:{ (x:String) in 
        doSomething(x) 
    })

To subscribe for on-error notifications


someSignal.subscribe(error:{ (e) in .... })
or
someSignal.subscribe(
    error:{ (e) in 
        logError(e) 
    })


e.g. with an RACSignal publishing strings it would be
someSignal.subscribe(error:{ (e) in logError(e) }

You don't need to specify (error:NSError) as the compiler is smart enough to infer that, however you do need to specify the explicit error: parameter name, so the compiler knows this is an error handler, not an on-next handler

To subscribe for on-next and on-completed notifications


someSignal.subscribe({ (arg:Type) in .... }, completed:{ ... })

e.g. with an RACSignal publishing strings it would be
someSignal.subscribe({ (x:String) in doSomething(x) }, completed:{ allDone() })
or
someSignal.subscribe(
    next:{ (x:String) in 
        doSomething(x) 
    }, 
    completed:{ 
       allDone() 
    })


Warning If the code in your completed block (e.g. the allDone function) doesn't return void, the compiler will infer it as the return type of the completed block, and you'll get a confusing-looking Extra argument 'completed' in call error. The fix is to insert an explicit return, e.g.

someSignal.subscribe(
    next:{ (x:String) in 
        doSomething(x) 
    }, 
    completed:{ 
       allDone() 
       return
    })


To subscribe for on-next and on-error notifications


someSignal.subscribe({ (arg:Type) in .... }, error:{ (e) in ... })

e.g. with an RACSignal publishing strings it would be
someSignal.subscribe({ (x:String) in doSomething(x) }, error:{ (e) in logError(e) })
or
someSignal.subscribe(
    next:{ (x:String) in 
        doSomething(x) 
    }, 
    error:{ (e) in
       logError(e) 
    })


To subscribe for all 3 - on-next, on-error and on-completed notifications


someSignal.subscribe({ (arg:Type) in .... }, error:{ (e) in ... }, completed:{ ... })

e.g. with an RACSignal publishing strings it would be
someSignal.subscribe({ (x:String) in doSomething(x) }, error:{ (e) in logError(e) }, completed:{ allDone() })
or
someSignal.subscribe(
    next:{ (x:String) in 
        doSomething(x) 
    }, 
    error:{ (e) in
       logError(e) 
    },
    completed:{
        allDone()
    })


The warning above (sometimes you might need an explicit return in the completed block) applies here too.

To subscribe for on-error and on-completed notifications


someSignal.subscribe(error:{ (e) in ... }, completed:{ ... })

e.g. with an RACSignal publishing strings it would be
someSignal.subscribe({ error:{ (e) in logError(e) }, completed:{ allDone() })
or
someSignal.subscribe(
    error:{ (e) in
       logError(e) 
    },
    completed: {
        allDone()
    })


The warning above (sometimes you might need an explicit return in the completed block) applies here too.

Wrap-up

I hope this helps!

P.S. - if you're wondering where the "on-completed only" version is - there isn't one, because ReactiveCocoa doesn't seem to provide one for some reason. If you want that behaviour, you can subscribe with an empty on-next or on-error callback.

Monday, May 05, 2014

Codemania 2014 Presentation

I gave a presentation at Codemania 2014 titled "Understanding C++ Templates".

The abstract is as follows (posted here for posterity, as codemania will take all the 2014 details down when the 2015 conference ramps up)

Understanding C++ templates

Templates are a key feature of C++. They enable you to write safer code with less duplication and better performance. Almost all C++ programmers have encountered them, but few know them well. Orion will show you how to think about and put to use some of the more advanced things you can do with templates, and show that they're simpler than you might think.

Even if you're not a C++ developer, understanding these concepts will give you a better perspective for programming in other languages.

The source code/examples I used in the presentation are available here on Github

The slides are available on here on iCloud

Monday, November 11, 2013

Integrating an automated iOS build with a Windows Team Foundation Server build environment

My current job is a "Microsoft shop" - our source control is Microsoft's Team Foundation Server, we primarily write applications for windows using C++ and C#, and all the developers use windows workstation PC's.

We'd like to develop an iOS app, but we'd like to set it up with a repeatable, automatable build process. Simply asking the guy with the mac on his desk to compile a new build and hope it works doesn't cut it.
All our existing build processes are using Team Foundation Server's build agents and build scripts, which have some very nice properties, and we'd like to be able to queue and publish iOS builds in the same way.

The problem is that we must build iOS apps on a Mac using Xcode, but the Team Foundation Server build agent only runs on Windows

How do we bridge this gap? Using SSH from a windows build agent to a remote mac.

  • The TFS build script uses putty (specifically the command line plink application) to tell the mac to remotely do things.
  • We queue a TFS build onto one of our windows TFS build agent PC's. It basically just runs plink, captures the output and waits for it to exit.
  • We get the files from the TFS repository onto the mac via SCP - the windows TFS build agent checks everything out, then runs pscp to copy it all over to the mac
  • The mac then uses xcodebuild and xcrun to compile and package the iOS app for ad-hoc distribution
  • The windows build agent then uses pscp to copy the packaged iOS app off the mac back onto the windows build agent, which then copies it into the appropriate drop folder alongside all the other windows builds.
  • We host a basic web server to serve the iOS application's .ipa package over HTTP, and everyone installs/upgrades our app by accessing this webpage using iOS' over the air beta testing facilities.

There are a lot of pieces to this, particularly given that we do not want to put secure data such as user account passwords in source control or have them hardcoded into build scripts. Here are the details, in the order that (for me) makes the most sense.

Setting up the Mac


Install Xcode

Obviously we need to install Xcode on the mac. The easiest way is just via the Mac app store, but you can also download a DMG from the apple developer centre for offline installs.

Install Team Explorer Everywhere

I did this by unzipping microsoft's zip file into /usr/local/bin. There is likely a better/more modular way to manage this than putting everything directly in /usr/local/bin, but this works fine on my isolated build machine.

User account

In the interests of keeping the build environment clean, and repeatable (so we can set up other mac build machines easily in the future), we want all of the builds to be performed by a dedicated user account. We do not ever want to use this account to do normal day to day development. Ideally we should never log into it interactively at all, only via SSH. This helps ensure that our builds really are clean and repeatable, and that we don't end up with accidental dependencies on random files/data in a normal person's user account.

Enabling passwordless SSH logins

We need the windows build agent PC to SSH into the mac, but we don't want the TFS build scripts to have the password of the mac's build account, we want to enable passwordless logins. To do this, I followed the instructions here: http://www.tonido.com/blog/index.php/2009/02/20/ssh-without-password-using-putty/

Once this has been set up, on the mac the /Users/buildaccount/.ssh/authorized_keys file will contain the details of the ssh key, and on the windows PC there will be the matching .ppk file. We can use plink to get the mac to run various scripts, as follows

plink -ssh -batch -l buildaccount -i buildaccountkey.ppk mac01.domain.local /Users/buildaccount/build_script.sh

Enabling our non-admin account to login using SSH

Apple's default SSH security settings are such that only users with Admin rights can login via SSH (At least this is the case on OSX 10.9, which I am using). We want our build account to be a non-admin user, so we'll have to edit the access control list that controls who can access ssh, and add our build user to it.

sudo dseditgroup -o edit -n . -a buildaccount -t user com.apple.access_ssh

Setting up the build process


Certificates and Provisioning Profiles

As we're setting up for Ad-hoc deployment of our app, we need an iOS Distribution Certificate, and an Ad-hoc provisioning profile. You generate both of these via the Apple Developer website's Member Centre. When building through the Xcode UI, it handles most of this for you, but as we want to be building via the command line, and via SSH into a clean user account, we have a few more hoops to jump through.

Putting the certificates in a custom keychain

By default Xcode and it's command line tools will look for your distribution certificate in the user default keychain (the login keychain). As we want our build account to be "clean", we don't want to have to put them in the build account's login keychain. To resolve this, we can create a custom keychain, and put the certificates/provisioning profile keys in there.

We can then decide to either check this custom keychain file into source control, or put it in some other special location.

I'm assuming that you've already created and loaded/installed a distribution certificate, and an ad-hoc provisioning profile.

  1. From your normal user account, run Keychain Access
  2. Right-click on the list of keychains in the top-left, and create a new keychain. Give it a meaningful filename and store it somewhere you can find later.
  3. From your login keychain, hold the option key (to copy) and drag your iPhone Distribution certificate, and the Ad-hoc distribution private key.

Using this custom keychain in the build script

To get the Xcode command line tools to use this custom keychain file, add the following before you invoke xcodebuild

keychain_file="$SCRIPTPATH/iOSCustomBuild.keychain"
keychain_pass=secretpassword
security list-keychains -s "$keychain_file" ~/Library/Keychains/login.keychain  # put the iOS keychain in ahead of the login keychain
security unlock-keychain -p "$keychain_pass" "$keychain_file"

The security list-keychains -s command sets the search order, telling the system first to look in our custom keychain, followed by the default login keychain. This command is persistent - I haven't worried about resetting it after our script completes, but you may want to

The security unlock-keychain command is required because keychains are locked by default when code is run in a remote SSH session. We must issue security unlock-keychain or else xcodebuild will not be able to read the keychain. This is another reason to use a custom keychain file - if we were using the user's login keychain, we would be required to hard code the user account's password into our build scripts.

Compiling and packaging the app

To build our iOS app, we'll need to run xcodebuild, which is xcode's command line build tool. This requires a number of flags, which we will create variables for and then re-use. Here's mine:

# hack to get the full path to the current directory
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd`
popd > /dev/null

application_name=MyiOSApp # our app name - should correspond to MyiOSApp.xcodeproj
sdk="iphoneos7.0" # which version of iOS are we targeting

# Name of the distribution certificate. Get this by downloading the certificate from 
# the apple developer center, installing it, and then viewing it's name in keychain
codesign="iPhone Distribution: My Name (AAA12345ZZ)"

# build the project (must sign as xcode requires signing for all non-simulator bulds)
# CONFIGURATION_BUILD_DIR is only required for cordova/phonegap projects.
xcodebuild -project "$SCRIPTPATH/$application_name.xcodeproj" -target $application_name -configuration Release -sdk $sdk clean build CODE_SIGN_IDENTITY="$codesign" CONFIGURATION_BUILD_DIR="$SCRIPTPATH/build"

This only gets us halfway - xcodebuild will produce a .app package (a directory). This doesn't have the provisioning profile embedded in it, and we also need to package it up as a .ipa file to distribute it via the ad-hoc mechanisms. To do this step, we'll need to run xcrun. Here's my part of the script to do that:

# Get this by downloading it from the apple developer center
provisioning_profile_file="$SCRIPTPATH/My_Ad_Hoc_Provisioning_Profile.mobileprovision"
provisioning_profile_name="My Ad Hoc Provisioning Profile" # get this from the apple developer center when you download the file

# install the provisioning profile so xcode can use it 
provisioning_profile_uuid=`grep UUID -A1 -a "$provisioning_profile_file" | grep -o "[-A-Z0-9]\{36\}"`
cp "$provisioning_profile_file" ~/Library/MobileDevice/Provisioning\ Profiles/$provisioning_profile_uuid.mobileprovision
 
# this is where xcode will drop it's output, as a .app package
build_dir="$SCRIPTPATH/build"

# this is where we want the created .ipa file to be put
drop_dir="$SCRIPTPATH"

# package it and embed the provisioning profile (must re-sign as packaging alters the app)
/usr/bin/xcrun -sdk $sdk PackageApplication -v "${build_dir}/${application_name}.app" -o "${drop_dir}/${application_name}.ipa" --sign "$codesign" --embed "$provisioning_profile_file"

I chose to simply check my provisioning profile file into source control.

Also note that we have the extra step of "installing" the provisioning profile. This is because we're trying to run under a clean account. If we were running as a normal user account where someone had manually run Xcode, Xcode would have put the provisioning profile into the ~/Library/MobileDevice/Provisioning Profiles directory already for us and we wouldn't need to do this step.

The full build script


keychain_file="$SCRIPTPATH/iOSCustomBuild.keychain"
keychain_pass=secretpassword
security list-keychains -s "$keychain_file" ~/Library/Keychains/login.keychain  # put the iOS keychain in ahead of the login keychain
security unlock-keychain -p "$keychain_pass" "$keychain_file"

# hack to get the full path to the current directory
pushd `dirname $0` > /dev/null
SCRIPTPATH=`pwd`
popd > /dev/null

application_name=MyiOSApp # our app name - should correspond to MyiOSApp.xcodeproj
sdk="iphoneos7.0" # which version of iOS are we targeting

# Name of the distribution certificate. Get this by downloading the certificate from 
# the apple developer center, installing it, and then viewing it's name in keychain
codesign="iPhone Distribution: My Name (AAA12345ZZ)"

# build the project (must sign as xcode requires signing for all non-simulator bulds)
# CONFIGURATION_BUILD_DIR is only required for cordova/phonegap projects.
xcodebuild -project "$SCRIPTPATH/$application_name.xcodeproj" -target $application_name -configuration Release -sdk $sdk clean build CODE_SIGN_IDENTITY="$codesign" CONFIGURATION_BUILD_DIR="$SCRIPTPATH/build"

# Get this by downloading it from the apple developer center
provisioning_profile_file="$SCRIPTPATH/My_Ad_Hoc_Provisioning_Profile.mobileprovision"
provisioning_profile_name="My Ad Hoc Provisioning Profile" # get this from the apple developer center when you download the file

# install the provisioning profile so xcode can use it 
provisioning_profile_uuid=`grep UUID -A1 -a "$provisioning_profile_file" | grep -o "[-A-Z0-9]\{36\}"`
cp "$provisioning_profile_file" ~/Library/MobileDevice/Provisioning\ Profiles/$provisioning_profile_uuid.mobileprovision
 
# this is where xcode will drop it's output, as a .app package
build_dir="$SCRIPTPATH/build"

# this is where we want the created .ipa file to be put
drop_dir="$SCRIPTPATH"

# package it and embed the provisioning profile (must re-sign as packaging alters the app)
/usr/bin/xcrun -sdk $sdk PackageApplication -v "${build_dir}/${application_name}.app" -o "${drop_dir}/${application_name}.ipa" --sign "$codesign" --embed "$provisioning_profile_file"

Putting it all together

All we need to do now is create TFS build script which Exec's plink and pscp and asks the mac to run the various .sh scripts which do all the work. Here's some snippets which you could use


<PropertyGroup>
    <!-- general -->
    <IpaFileName>MyiOSApp.ipa</IpaFileName>
    
    <!--File locations on the windows build machine-->
    <Plink>pathtoPLINK.EXE</Plink>
    <Pscp>pathtoPSCP.EXE</Pscp>
    
    <!--SSH connection info-->
    <SshHost>mac.domain.local</SshHost>
    <SshKey>buildaccountkey.ppk</SshKey>
    <SshUser>buildaccount</SshUser>

    <!--File locations on the mac-->
    <RemoteiOSBuildPath>/Users/buildaccount/iOSBuild</RemoteiOSBuildPath>
    <RemoteBuildScriptPath>$(RemoteiOSBuildPath)/build.sh</RemoteBuildScriptPath>
    <RemoteIpaFilePath>$(RemoteiOSBuildPath)/$(IpaFileName)</RemoteIpaFilePath>
</PropertyGroup>

...

<!--this "echo y | exit" causes plink/putty to cache the remote host key in the registry for subsequent operations.-->
<Exec command="echo y | &quot;$(Plink)&quot; -ssh -l $(SshUser) -i $(SshKey) $(SshHost) exit"/>

<!--Wipe any old files-->
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) rm -rf $(RemoteiOSBuildPath)" />
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) mkdir -p $(RemoteiOSBuildPath)" />

<!-- copy the source over to the mac -->
<Exec command="&quot;$(Pscp)&quot; -sftp -batch -r -i $(SshKey) -l $(SshUser) &quot;$(SolutionRoot)*&quot; $(SshHost):$(RemoteiOSBuildPath)" />

<!--run the build script-->
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) chmod +x $(RemoteBuildScriptPath)"/>
<Exec command="&quot;$(Plink)&quot; -ssh -batch -l $(SshUser) -i $(SshKey) $(SshHost) $(RemoteBuildScriptPath)"/>

<!-- copy the packaged ipa file back -->
<Exec command="&quot;$(Pscp)&quot; -sftp -batch -i $(SshKey) -l $(SshUser) $(SshHost):$(RemoteIpaFilePath) ." />

Wednesday, October 02, 2013

Programmatically controlling Hyper-V Server 2012 R2 virtual machines from C#

Recently I've wanted to remote control some virtual machines on a Hyper-V server in aid of automated testing.
I've got a server set up running Hyper-V Server 2012 R2, so the code samples in this are likely to work against Server 2012 non-R2, but may not work against Server 2008.

All the virtual machines start off at a known clean snapshot, and have a startup script which runs on boot that goes and asks a central database for a testing job to execute. So, for our system to work, we need to accomplish the following:
  1. Roll back a VM to the last known snapshot / checkpoint
  2. Power it on
There are several other things we need to do for this to work at all, which are:
  1. Connect to the Hyper-V server (this includes authentication)
  2. List out or otherwise ask the server about it's VM's so that we can rollback the correct one

To achieve thisk I'm using Hyper-V's WMI provider (V2). I don't really know much about WMI, so these code samples are just things I've hacked together. They are NOT production quality and they don't handle errors well or anything else. Please use them as guidance, not for copy/pasting.

Pre-requisites

Using WMI from C# is done by the classes in the System.Management namespace. To access this you'll want using System.Management at the top of your C# file, as well as a reference to the System.Management assembly.

Connecting and Logging on


To do this we need to create a ManagementScope object set to use the appropriate server and using the Hyper-V virtualization v2 namespace, and we also need to supply the username and password of a windows account that has privileges to administer the Hyper-V server. I did it like this:

var connectionOptions = new ConnectionOptions(
    @"en-US",
    @"domain\user",
    @"password",
    null,
    ImpersonationLevel.Impersonate, // I don't know if this is correct, but it worked for me
    AuthenticationLevel.Default,
    false,
    null,
    TimeSpan.FromSeconds(5);

var scope = new ManagementScope(new ManagementPath { 
    Server = "hostnameOrIpAddress", 
    NamespacePath = @"root\virtualization\v2" }, connectionOptions);
scope.Connect();

Note: Most of the other documentation or sample code I found refers to the root\virtualization namespace. This didn't work at all for me in Server 2012 R2, and I had to use dotPeek to decompile the Hyper-V powershell commandlets to figure out to put \v2 on the end. Perhaps the non-v2 one is for Server 2008?

Note: If your PC and the target server are on the same domain, and your windows user account has privileges to administer the remote server, You don't need the ConnectionOptions object at all. WMI will use windows authentication, and it will magically work:

Listing out the VM's and finding the one we want

Virtual machines in Hyper-V get exposed via the Msvm_ComputerSystem WMI class. In order to list them out, we simply ask WMI to give us all the objects of that class in the virtualization namespace. To do this, I'm going to create two helper extension methods, that we'll use from hereon:

public static class WmiExtensionMethods
{
    public static ManagementObject GetObject(this ManagementScope scope, string serviceName)
    {
        return GetObjects(scope, serviceName).FirstOrDefault();
    }
    
    public static IEnumerable<ManagementObject> GetObjects(this ManagementScope scope, string serviceName)
    {
        return new ManagementClass(scope, new ManagementPath(serviceName), null)
            .GetInstances()
            .OfType<ManagementObject>();
    }
}

It turns out that the Host PC is also exposed via Msvm_ComputerSystem, so we need to filter out things that are not virtual machines. This helper method will return a list of all the virtual machines. Note: I like extension methods, so I'm using them a lot here:

public static IEnumerable<ManagementObject> GetVirtualMachines(this ManagementScope scope)
{
    return scope.GetObjects("Msvm_ComputerSystem").Where(x => "Virtual Machine" == (string)x["Caption"]);
}

You can use it to get a specific virtual machine as follows:

var vm = scope.GetVirtualMachines().First(vm => vm["ElementName"] as string == "myvmname");

Rolling the VM back to it's latest snapshot / checkpoint

In order to roll a Hyper-V VM back to a snapshot, we need to get a reference to the snapshot object itself.

There are ways to simply list all the snapshots for each VM, but there is also a special Msvm_MostCurrentSnapshotInBranch class which represents the "latest" snapshot. We can use it to create a helper method as follows:

public static ManagementObject GetLastSnapshot(this ManagementObject virtualMachine)
{
    return virtualMachine.GetRelated(
        "Msvm_VirtualSystemSettingData",
        "Msvm_MostCurrentSnapshotInBranch",
        null,
        null,
        "Dependent",
        "Antecedent",
        false,
        null).OfType<ManagementObject>().FirstOrDefault();
}

And use it like this:

var snapshot = vm.GetLastSnapshot();

Now, to actually roll the VM back, we need to call the ApplySnapshot method on the Msvm_VirtualSystemSnapshotService class.
Note: Logically I thought that the snapshot methods should be on the Virtual Machine object, but Hyper-V puts them all in their own service for some reason. There is also only one global Snapshot service - not a service per VM. I've no idea why they've designed it this way.

We can create a helper method:

public static uint ApplySnapshot(this ManagementScope scope, ManagementObject snapshot)
{
    var snapshotService = scope.GetObject("Msvm_VirtualSystemSnapshotService");

    var inParameters = snapshotService.GetMethodParameters("ApplySnapshot");
    inParameters["Snapshot"] = snapshot.Path.Path;
    var outParameters = snapshotService.InvokeMethod("ApplySnapshot", inParameters, null);
    return (uint)outParameters["ReturnValue"];
}

And use it like this:

scope.ApplySnapshot(snapshot);

When I execute this with valid parameters, the VM applied it's snapshot, and I always got a return value of 4096. According to the documentation, this indicates that there's a Job in progress to asynchronously track the actual snapshot applying and determine the final success or failure. We could fetch the job out of outParameters["Job"] and use it to determine when the apply completes, but I'm not going to worry about that here. Refer to the ApplySnapshot MSDN page for other possible return codes.

Note: If the VM is not powered off, you are likely to find the ApplySnapshot call fails. You must power the VM off (and wait for the Power Off job to complete) first.

Powering on the VM to boot it up


Turning on the VM is done by the RequestStateChange method on the Msvm_ComputerSystem class. We already have the Msvm_ComputerSystem object representing the virtual machine we found earlier, so we can create a helper method and enumeration like this:

public enum VmRequestedState : ushort
{
    Other = 1,
    Running = 2,
    Off = 3,
    Saved = 6,
    Paused = 9,
    Starting = 10,
    Reset = 11,
    Saving = 32773,
    Pausing = 32776,
    Resuming = 32777,
    FastSaved = 32779,
    FastSaving = 32780,
}

public static uint RequestStateChange(this ManagementObject virtualMachine, VmRequestedState targetState)
{
    var managementService = virtualMachine.Scope.GetObject("Msvm_VirtualSystemManagementService");

    var inParameters = managementService.GetMethodParameters("RequestStateChange");
    inParameters["RequestedState"] = (object)targetState;
    var outParameters = virtualMachine.InvokeMethod("RequestStateChange", inParameters, null);
    return (uint)outParameters["ReturnValue"];
}

And use it like this:

vm.RequestStateChange(VmRequestedState.Running);

As for ApplySnapshot, when things work, the return value is usually 4096, indicating there is a Job to asynchronously track the progress of the operation.

Note: Although we invoke the method on the Msvm_ComputerSystem object, we need to get the method parameters by asking the Msvm_VirtualSystemManagementService object (which represents the host server) instead. I've no idea why.

Final fixups


ApplySnapshot will fail if the virtual machine is running. To turn it off, we can simply call vm.RequestStateChange(VmRequestedState.Off); and wait a bit for Job to complete.

RequestStateChange will fail if the state doesn't make sense. For example, if you try and turn the vm Off when it's already Off, the method will fail. You can check the current state of a Vm by reading it's EnabledState property. Valid values for EnabledState are documented with the Msvm_ComputerSystem class. I created an enum, and used it as follows:

public enum VmState : ushort
{
    Unknown = 0,
    Other,
    Running, // Enabled
    Off, // Disabled
    ShuttingDown,
    NotApplicable,
    OnButOffline,
    InTest,
    Deferred,
    Quiesce,
    Starting
}

if ((VmState)vm["EnabledState"] != VmState.Off)
{
    vm.RequestStateChange(VmRequestedState.Off); // needs to be off to apply snapshot
    Thread.Sleep(2000); // todo wait for the state change properly
}

You can do many more things by using other methods and classes from the Hyper-V WMI API. Hopefully this gives you a decent starting point.

P.S. The Hyper-V powershell commandlets are all implemented on top of this WMI Api. Using a tool like .NET reflector or dotPeek is an interesting way to see how Microsoft calls the WMI API.

Monday, October 08, 2012

TechEd 2012 background 3 - Memory Barriers

Note: This goal of this post is to provide more in-depth information and reference for attendees about a presentation I'm giving at Tech Ed New Zealand 2012.
The session is DEV402 - Multithreaded programming in .NET from the ground up.

This third post shows one the effect of the CPU re-ordering reads and writes to memory in a C# program, and explains how to fix it using Thread.MemoryBarrier

This sample program basically fires 2 functions at the threadpool over and over. In theory we can't tell which will run first, it's effectively random - This program just tries to provide an empirical measurement.

class Program
{
    static volatile int a = 0;
    static volatile int b = 0;
    static volatile bool a_ranFirst, b_ranFirst;

    static object gate = new object();

    static void ThreadA()
    {
        a = 1;
        //Thread.MemoryBarrier();
        if (b == 1)
            b_ranFirst = true;
    }

    static void ThreadB()
    {
        b = 1;
        //Thread.MemoryBarrier();
        if (a == 1)
            a_ranFirst = true;
    }

    static void Main(string[] args)
    {
        int aFirst = 0, bFirst = 0;
        int together = 0;
        int WTF = 0;

        var sw = new Stopwatch(); sw.Start();

        const int Iterations = 500000;

        for (int i = 0; i < Iterations; i++)
        {
            a = b = 0;
            a_ranFirst = b_ranFirst = false;
            Parallel.Invoke(ThreadA, ThreadB);

            if (a_ranFirst && !b_ranFirst)
                aFirst++;
            else if (!a_ranFirst && b_ranFirst)
                bFirst++;
            else if (a_ranFirst && b_ranFirst)
                together++;
            else
                WTF++; //
        }
        Console.WriteLine("Done in {0}ms", sw.ElapsedMilliseconds);

        Console.WriteLine("Thread A ran first: {0} times ({1} percent)", aFirst, ((double)aFirst / (double)Iterations) * 100);
        Console.WriteLine("Thread B ran first: {0} times ({1} percent)", bFirst, ((double)bFirst / (double)Iterations) * 100);
        Console.WriteLine("Ran together: {0} times ({1} percent)", together, ((double)together / (double)Iterations) * 100);
        Console.WriteLine("WTF: {0} times ({1} percent)", WTF, ((double)WTF / (double)Iterations) * 100);
    }
}

If I run this as-is, I get something like this:
Done in 1571ms
Thread A ran first: 379388 times (75.8776 percent)
Thread B ran first: 31140 times (6.228 percent)
Ran together: 59 times (0.0118 percent)
WTF: 89413 times (17.8826 percent)
76% of the time, A runs first (this makes sense given that it's the first argument to Parallel.Invoke) - 6% of the time B runs first, and a very small percentage of the time they run together in lockstep. However, 18% of the time, something else is happening. What's going on? To break this down, the important operations that happen on each thread are:
  • Store the value 1 to the memory location A (or B for thread B)
  • Load the value from the other memory Location
  • Figure out whether this thread was first on second based on what we read
  • Set a flag to indicate whether this thread was first or second
The first 2 steps are the key to explaining the bug. Here's what we might expect to be happening
  • When "A runs first", thread A performs both the store to A and load from B before thread B gets started.
  • When "B runs first", thread B performs both the store to B and load from A before thread A gets started.
  • When they run together, thread A stores, thread B stores, then they both load

So, what about the Fourth, "WTF" scenario? We've already marked the a and b variables as volatile, so this should tell the compiler to leave them alone. Something else is happening, but what?

Let's have a look at the assembly that's getting generated:
a = 1;
00000000  mov         dword ptr ds:[0111328Ch],1 
if (b == 1)
0000000a  cmp         dword ptr ds:[01113290h],1 
00000011  jne         0000001A 
b_ranFirst = true;
00000013  mov         byte ptr ds:[01113295h],1 
0000001a  ret 

A quick run through:
  • The first mov operation is writing 1 to memory location 0111328C (where the compiler has stored a)
  • The next cmp/jne instructions read 01113290 (where the compiler has stored b), and jump to the end of the function if it's not set to 1.
  • If it doesn't get jumped over, the final mov operation is writing 1 to location 01113295 (where b_ranFirst lives)

This all adds up - the instructions we'd logically expect to happen are indeed happening, and it's all in the order we'd expect. Clearly neither the C# or JIT compiler is at fault here... so what is?

Well, as it turns out there are actually 3 more scenarios that are occurring.

  • Thread A performs load from B; thread B then does it's normal store/load; finally, thread A performs store to A
  • Thread B performs load from A; thread A then does it's normal store/load; finally thread B performs store to B
  • thread A loads, thread B loads, then they both store

In spite of the fact that our code very explicitly always stores before it loads, the operations are being performed in the wrong order.

This is Out-of-Order Execution in action. All Intel CPU's (except the Atom) since 1995 do this, as do pretty much all AMD processors. I've got a Core i5 here, which certainly does this reordering.

Why reordering?

It takes many hundreds of CPU cycles to do a read or write to memory. The number varies quite a lot depending on what CPU you have, but it's always a significant amount. To try compensate for this, out-of-order CPU's will try and rearrange reads and writes such that they can carry on doing work while waiting for the data to arrive or the write to complete. Writes can also be batched in some situations, so the CPU may hold off doing a write until a second one arrives to issue a single batch.

When is reordering a factor?

For the majority of code, reordering isn't an issue.
While it is executing code, a CPU keeps track of whatever re-ordering it does, and makes sure that all operations appear to execute as you'd expect. If it moves something before or after another operation, then it will make sure to fix up the result before the next thing needs to access the result.

In computers with only a single CPU core, all code is effectively single threaded like this - While the operating system can and does interrupt and switch the running thread randomly and unpredictably, when it does this it inserts appropriate instructions (memory barriers) which cause any re-ordered operations to be "flushed" before the thread is context switched out.

The problem arises in multi-CPU systems.
While each CPU is keeping track of and fixing up all it's shenanigans, it's not telling the other CPU's about what it's doing. When other CPU's read or write, they don't get this "fixing up" logic applied, and they see reads and writes in the actual (rearranged) order they're happening.

OK, so how do we fix it?

The first thing we might like to do is somehow tell the CPU to turn this reordering off. Unfortunately (or fortunately, depending on your point of view) - you can't. Furthermore, there's no tracing or other kinds of diagnostics to tell you when you're being affected by this. Unfortunately, you have to resort to good old logic and knowledge. You have to know what the rules for reordering are for your CPU, analyze the code, and work out if reordering could possibly cause your code to fail. If it could - you have to defend against it.

And how do we defend against it? This is where the Memory Barrier comes in. A memory barrier acts as a "fence" or "line in the sand", or other such metaphor. Basically, it's a wall that the CPU can't move reads or writes through.

There are 3 kinds of memory barriers. Most of the information that I've been able to find is quite confusingly worded, so I'll do my best to be clear:

  • Read barriers (also known as "Load" or "Acquire" barriers/fences). These prevent loads from memory from moving upwards in the instruction sequence.
  • Write barriers (also known as "Store" or "Release" barriers/fences). These prevent writes to memory from moving downwards in the instruction sequence.
  • Full barriers. These prevent all re-oredering

Wednesday, September 05, 2012

TechEd 2012 background 2 - Compiler optimizations and Volatile

Note: This goal of this post is to provide more in-depth information and reference for attendees about a presentation I'm giving at Tech Ed New Zealand 2012.
The session is DEV402 - Multithreaded programming in .NET from the ground up.

This second post shows one possible effect of compiler optimizations on a C# program, and explains how to fix it using the oft-misunderstood volatile keyword.

Here's a sample program which illustrates this.
This program looks suspiciously similar to one of the samples in Joseph Albahari's threading book because that was by far the best piece of code I found or could invent to best illustrate the issue. All credit due to Joseph for his great work


class Program
{
    static int stopFlag = 0;

    static void ThreadProc()
    {
        Console.WriteLine("{0}: worker thread start", DateTime.Now.TimeOfDay);
        bool toggle = false;

        while (stopFlag == 0)
        {
            toggle = !toggle;
        }
        Console.WriteLine("{0}: worker thread done", DateTime.Now.TimeOfDay);
        Console.ReadLine();
    }

    public static void Main(string[] args)
    {
        stopFlag = 0;
        var t = new Thread(ThreadProc);
        t.Start();
        Thread.Sleep(1000);

        stopFlag = 1;

        Console.WriteLine("waiting...");
        t.Join();
    }
}


In summary, it's pretty simple. The main thread spins up a worker thread, sleeps for a second, then sets the stopFlag. The worker thread sits on a loop waiting for the stopFlag to be set, then exits.

If you run a Debug build of this code, or launch a 32-bit Release build with the debugger attached, you'll get this output:

10:38:25.0815053: worker thread start
waiting...
10:38:26.0955150: worker thread done

However, if you launch a 32-bit Release build without the debugger attached, you'll see this:

10:51:47.1955922: worker thread start
waiting...

The worker thread won't stop, and if you launch Task Manager you'll see that the app is thrashing away using an entire CPU. What's going on here?

To answer this, we need to attach the debugger to the already-running program, pause it and jump over to the worker thread and see what it's up to. The process for this in Visual Studio 2010/2012 goes like this:

  • From the Debug menu, chose Attach to Process...
  • Scroll down the list, select your application, and click Attach
  • Once the attach dialog closes, pause the application by clicking the pause button on the toolbar, or selecting Break All from the Debug menu. This will stop the program looping and let us look at what it's up to
  • Bring up the threads window, ( Debug / Windows / Threads ) and double-click on the worker thread

We should now see the current line of code that is being executed by this thread, which is

while (stopFlag == 0)

So, we're still in the loop waiting for the stopFlag to be set - yet, if you mouse over the stopFlag variable, or Add it to the Watch window, we'll see that it has a value of 1

The flag has been set, but our loop is still going. What gives?
To answer this, we need to bring up the Disassembly window. ( Debug / Windows / Disassembly ). This window shows us the raw x86 machine code that has been generated by the JIT compiler, and is the actual code that is being executed on the CPU.

The VS disassembly window will print each line of C# source code (if it has it), followed by the x86 assembly. For the line we're stuck on, we get this:

while (stopFlag == 0)
0000004a  mov         eax,dword ptr ds:[007A328Ch] 
0000004f  test        eax,eax 
00000051  jne         00000057 
00000053  test        eax,eax 
00000055  je          00000053 

If you know how to read x86 assembly, I congratulate you. I only know the basics, but enough that I can walk you through it and show what's going on.

So first, this line:

0000004a mov eax,dword ptr ds:[007A328Ch]

  • The instruction is mov which is x86 for "set something". The first parameter is the target, and the second is the source.
  • The First parameter is eax which refers to the EAX register inside the CPU. Intel x86 processors have 4 general purpose registers - EAX through EDX - that are used to perform the vast majority if work that happens in an application.
  • The Second parameter is dword ptr ds:[007A328Ch]. The dword ptr bit is a Size directive ( see this page and scroll down to "Size Directives" ) and it's needed to tell the CPU how many bytes we should be moving. dword ptr simply means 32 bits. I'm not entirely sure what the ds: bit means, but the most important part is this: [007A328Ch]. The square brackets mean "the value in memory at the given location". Our stop flag is static, so it's stored in a fixed memory location, in my case 007A328C

Translated: "Fetch 32 bits from memory at location 007A328C, and put them in the EAS register." In other words - load our stopflag into the CPU

The next part is this:
0000004f test eax,eax
00000051 jne 00000057


  • The test eax,eax is basically asking the question "Does EAX contain zero?"
  • The jne 00000057 is the Jump-if-not-equal instruction. It translates to "If it was NOT zero, jump to line 57

We know however, that when the loop starts, the flag was in fact zero, so we'll get past this block and onto the next one:

00000053 test eax,eax
00000055 je 00000053


  • Again, "Does EAX contain zero?"
  • The je 00000053 is the Jump-if-equal instruction. It translates to "If it WAS zero, jump to line 53, which is the start of our loop

Putting this all together, we end up with this sequence of events:

  • Load the stopflag from memory into EAX
  • Loop waiting for EAX to become non-zero

The problem is, at no point does any code ever set eax once we enter the loop.

What's going on here? The answer is that the C# and JIT compiler only consider the single-threaded case when doing optimizations. From the point of view of the WorkerThread, we never set the stopFlag, and we never call any other functions, so there is no possibility that the stopFlag can change. The JIT compiler can apply this, and decide that it only needs to read the stopFlag a single time, leading to this infinite loop!

To fix it, we simply mark the stopFlag variable as volatile.

This causes the assembly to change to this:

0000004a cmp dword ptr ds:[0088328Ch],0
00000051 jne 0000005C
00000053 cmp dword ptr ds:[0088328Ch],0
0000005a je 00000053


The key difference is that we see [0088328Ch] in the repetitive part of the loop. This means that at each iteration through the loop we go to memory and read the value. This means that when the stopFlag gets set after a second, we'll re-read it and the program behaves correctly. Great!

Discussion


There are a couple of things I'd like to go into more detail about:

Firstly, the weird double-check that the compiler generates for loops.
When the JIT compiler generates a loop it always seems to use this pattern:

  • First, check the opposite of the loop condition. If this is true, jump over the entire loop code
  • Now we enter the loop where we do the repeated action, and check the actual loop condition

Technically, there's no reason why we need to check it twice, we could get by with just the second part where we check the positive loop condition. I've no expert, but from what I can guess this is probably to help with branch prediction.
My personal theory is that by splitting the loop into 2 conditions, the branch predictor can track them both seperately.
If we bail out of the loop before it even gets started, the branch predictor won't need to worry about the subsequent code, but if we enter the second phase of the loop, the odds are good that we'll keep looping, so the branch predictor can use this information.

Secondly, the use of the cmp instruction rather than test in the loops
This stackoverflow question/answer sheds some light on the topic

The test instruction is smaller (different x86 instructions take different numbers of bytes to encode) than cmp, so the compiler will prefer it over cmp where both would achieve the same result. test works by doing a bitwise AND on the two values, whereas cmp does subtraction.

In the first case, where the value has already been loaded into a register, ANDing it with itself is a nice fast way to check for zero. In the second case however, where the value is being read from memory every time, in order to use test, we'd have to either use an extra register to store it in, or read the memory twice for both sides of the test operation. Both of which would be slower than against the constant 0.

Thirdly, what happened to the toggling inside the loop?
You may have noticed that the disassembly at no point shows anything to do with the loop body of toggle = !toggle;.

This is just the compiler optimizer doing it's job.

toggle is a local variable, it's not returned, and it's not passed to any functions - therefore, whatever we do to it can have no possible effect on anything. The compiler detects this, and simply removes it entirely.

So, what does this mean about volatile?
There seems to be a lot of confusion around what the volatile keyword is supposed to do. There are many questions/answers and blog posts which attempt to explain it. Some say say that it will insert memory barriers either before or after writes to the variable, some say that it causes reads and writes to be translated to calls to the Thread.VolatileRead or VolatileWrite method (both of which insert explicit barriers - but as we can see from our disassembly, it isn't inserting any such barriers, it's just doing normal ordinary reads and writes.

I think a lot of this confusion comes from the MSDN volatile documentation. The first hit in google for "C# Volatile Keyword" is the MSDN documentation, which states The system always reads the current value of a volatile object at the point it is requested, even if the previous instruction asked for a value from the same object. Also, the value of the object is written immediately on assignment.

In order for this to be true (particularly the "value is written immediately" part) one might infer that the compiler would insert memory barriers. Unfortunately, this page is the documentation for Visual Studio 2003. Subsequent revisions (2005, 2010, etc) re-word the documentation to be less misleading, but unfortunately google is still linking everyone to the old out of date page! For more, see http://www.albahari.com/threading/part4.aspx#_The_volatile_keyword

Additionally, the ECMA CLI spec states that reads to volatile fields should have acquire semantics, and writes to volatile fields should have release semantics. This could also cause people to think that memory barriers might be neccessary, but reads and writes on x86 already have these semantics anyway, so the net effect of volatile (other than compiler optimizations) is... Nothing!. On ARM or Itanium however, the JIT will insert memory barriers around reads and writes to volatile fields to give you the acquire/release semantics, as those CPU's don't guarantee it natively.