Mark Gilbert's Blog

Science and technology, served light and fluffy.

Will you just wait a minute?! NUnit and Async/Await

I was being a good-doobie.  Honest.

I had written a prototype – just something to prove that a particular approach could work.  It did, and so now it was time for me to actually bring that code up to production quality.  Part of that meant modifying how I was invoking the third-party web services – they  needed to be done asynchronously.

So I went through and wrote my unit tests to also run asynchronously:

        [Test]
        public async void Divide_4DividedBy2_Equals2()
        {
            AsyncUnitTest.Math MathLibrary = new AsyncUnitTest.Math();

            float Quotient = await MathLibrary.Divide(4, 2);

            Assert.AreEqual(2, (int)Quotient);
        }

I ran it through NUnit on my machine, and everything was peachy-keen.  I ended up writing nearly 50 unit tests* like that, converting over all of my calls.  I committed the code, and let TeamCity take over.

And watched every one of those new tests break.

When I looked at the TeamCity log, the errors seemed to hint that the test runner was simply not waiting for the thing under test to complete before trying to run the asserts.  I started searching for things like "nunit async", and pretty quickly across this two-part series by Stephen Cleary:

In this series, Cleary says that the underlying problem of running async tests is that they don’t have a proper context:

We’ve encountered a situation very similar to async in Console programs: there is no async context provided for unit tests, so they’re just using the thread pool context. This means that when we await our method under test, then our async test method returns to its caller (the unit test framework), and the remainder of the async test method – including the Assert – is scheduled to run on the thread pool. When the unit test framework sees the test method return (without an exception), then it marks the method as “Passed”. Eventually, the Assert will fail on the thread pool.

His solution is to simply give the test an async context, and he provides a very handy wrapper to do just that.  I first had to install his Nito.AsyncEx NuGet package, and then wrap my test in AsyncContext.Run:

        [Test]
        public void Divide_4DividedBy2_Equals2_Asynchrofied()
        {
            AsyncContext.Run(async () =>
            {
                AsyncUnitTest.Math MathLibrary = new AsyncUnitTest.Math();

                float Quotient = await MathLibrary.Divide(4, 2);

                Assert.AreEqual(2, (int)Quotient);
            });
        }

Notice that I’ve removed the "async" keyword from the test itself; AsyncContext.Run does all the work here.  After updating and committing my first test using AsyncContext.Run – a test test, if you will – it ran successfully on TeamCity.  I updated the other 48, and finally got a green build.

***

My build was stable again, but Cleary’s explanation didn’t answer the question of why this worked on my machine in the first place – without using his very awesome library – so, I kept digging.

I first looked up exactly what TeamCity was using to run the tests – it was NUnit, the same as what was on my machine, with a minor different in the version.  My local copy was 2.6.2, while the version on the build server was 2.6.1.  Could there be a difference in how 2.6.1 was handling async?

Why yes.  Yes there was.  In the NUnit 2.6.2 release notes I found this:

When running under .NET 4.5, async test methods are now supported. For test cases returning a value, the method must return Task<T>, where T is the type of the returned value. For single tests and test cases not returning a value, the method may return either void or Task.

– Source: http://nunit.org/index.php?p=releaseNotes&r=2.6.2

Are you serious?  I just happen to have the first version of NUnit that would properly handle async on my machine, but the build server was one notch older, and therefore couldn’t?  *facepalm*

To further prove that this was the real source of my issue, I installed NUnit 2.6.1 and 2.6.2 side by side on my machine.  I took my two tests from above, both of which should have tried to execute the MathLibrary.Divide function which included a 2-second delay:

    public class Math
    {
        public async Task<float> Divide(int Numerator, int Denominator)
        {
            await Task.Delay(2000);
            return Numerator / Denominator;
        }
    }

When I ran these two tests through NUnit 2.6.1, Divide_4DividedBy2_Equals2 completes in a couple hundredths of a second, while Divide_4DividedBy2_Equals2_Asynchrofied takes just over 2 seconds to complete, for a total of just over 2 seconds:

2-6-1

When I ran these through NUnit 2.6.2, EACH test takes just over 2 seconds to complete, for a total of just over 4 seconds:

2-6-2

So, I have two choices – switch my builds on TeamCity to use at least NUnit 2.6.2 to run the tests, or use Cleary’s Nito.AsyncEx library, which will allow me to leave the build server as is.  In any event, at least I have a reasonable explanation for what was happening. 

The funny thing is that it’s usually MSBuild that messes with me.  Apparently NUnit gave him the week off.

 


* Yes, I realize that by calling the service directly, this no longer counts as a "unit" test, but rather an integration test.  That distinction isn’t relevant to the issue described in this post, though, so I’m going to gloss over the mock objects in the real code.

Advertisements

October 23, 2014 - Posted by | Agile, Visual Studio/.NET

Sorry, the comment form is closed at this time.

%d bloggers like this: