It's the Tests

Software projects often publish comparisons with other projects, with which they compete. These comparisons typically have a few characteristics in common:

• They aim at highlighting reasons why one project is superior – that is, they are marketing material.
• While they may be accurate when initially published, competitor information is rarely updated.
• Pure factual information is mixed with opinion, sometimes in a way that doesn’t make clear which is which.
• Competitors don’t get much say in what is said about their projects.
• Users can’t be sure how much to trust such comparisons.

Of course, we’re used to it. We no longer expect the pure, unvarnished truth from software companies – no more than from drug companies, insurance companies, car salesmen or government agencies. We’re cynical.

But one might at least hope that open source projects might do better. It’s in all our interests, and in our users’ interests, to have accurate, up-to-date, unbiased feature comparisons.

So, what would such a comparison look like?

• It should have accurate, up-to-date information about each project.
• That information should be purely factual, to the extent possible. Where necessary, opinions can be expressed only if clearly identified as opinion by it’s content and placement.
• Developers from each project should be responsible for updating their own features.
• Developers from each project should be accountable for any misstatements that slip in.

I think this can work because most of us in the open source world are committed to… openness. We generally value accuracy and we try to separate fact from opinion. Of course, it’s always easy to confuse one’s own strongly held beliefs with fact, but in most groups where I participate, I see such situations dealt with quite easily and with civility. Open source folks are, in fact, generally quite civil.

So, to carry this out, I’m announcing the .NET Test Framework Feature Comparison project – ideas for better names and an acronym are welcome. I’ll provide at least a temporary home for it and set up an initial format for discussion. We’ll start with MbUnit and NUnit, but I’d like to add other frameworks to the mix as soon as volunteers are available. If you are part of a .NET test framework project and want to participate, please drop me a line.

I just released Beta 4 of NUnit Test Adapter for Visual Studio, with support for VS 2012 Update 1. The new release, version 0.94, works with the VS 2012 RTM as well but shows off some extra features when used with the update.

The biggest addition to the Adapter in this release is that it now supports Traits. NUnit Properties and Categories are translated in to Visual Studio traits and you can sort the display of tests by trait or filter on them. This goes a long way toward making the Visual Studio Test Explorer useful in running large sets of tests.

For details on this release see the release notes. You can download the adapter from Visual Studio 2012 using the extensions menu.

I just released version 0.8 of NUnitLite. This new release has almost all the framework features of full NUnit, plus one new feature of it’s own… Asynchronous tests. You can download NUnitLite from our Launchpad project page or using NuGet.

I’ve been a bit surprised about the recent upsurge of interest in NUnitLite. It’s being used as the basis for unit testing in both MonoTouch and Mono for Android, which isn’t totally surprising given that it was originally designed with an eye toward mobile platforms. But beyond that usage, I’ve noticed that it’s getting a fair amount of play in desktop applications.

So what’s that about? Well, one possibility is that a minimalist testing framework packaged as a single assembly has a lot of appeal to developers. Whatever the reason, I’m paying attention to this new interest in NUnitLite. The release cycle has been accelerated to the point where it now comes quite close to matching NUnit itself, feature for feature, in most common usage scenarios. It won’t do if you need to write your own addins, but then a framework in a single assembly has less need of addins: you can add a lot of functionality right in the test by inheriting from classes in the framework.

The next releases of NUnitLite will focus on adding more platform support. Silverlight is coming first, followed by Windows Phone 7 and 7.1, .NET 4.5, MonoTouch and MonoDroid. This growth in platform support will require folks willing to serve as testers for one or more platforms, particularly those I don’t have available myself. Please contact me if you’d like to help.

As mentioned in an earlier post, I’m working toward a 1.0 release of NUnitLite, which will include support for Silverlight. I have now created an experimental branch that runs NUnitLite under Silverlight. The image below shows a run of the Silverlight 5.0 build, executing it’s own tests.

You can browse the code for this branch at http://code.launchpad.net/~charlie.poole/nunitlite/silverlight or create your own local bazaar branch to build it using the following bazaar command:

bzr branch lp:~charlie.poole/nunitlite/silverlight

It’s not yet clear just how we should support Silverlight in the final release. Right now, there are separate builds for Silverlight 3.0, 4.0 and 5.0, with only minor differences among them. It might be simpler to maintain a single 3.0-compatible build. Please tell me what you think!

I’m also making the assumption that we can forego support of Silverlight 1.0 and 2.0. Once again, please let me know what you think, especially if you think my assumption is wrong!

Another important issue is how tests should be run. The approach I have chosen, as shown in the image above, is to make the test assembly a Silverlight application that uses the TestPage object embedded in the nunitlite assembly. However, there are at least two other approaches possible:

1. Create a test app that runs under the Microsoft Silverlight Teting Framework
2. Use a console runner that loads tests using the Silverlight assemblies

Use of the Microsoft Silverlight Testing Framework would involve a great deal of work to adapt or modify that framework. Using the default test harness, it’s not really feasible to support parameterized or generic test fixtures. That harness is based on the assumption that the framework tests must be invoked externally, through reflection. While true for NUnit 2.x, that assumption is no longer true for either NUnitLite or NUnit 3.0. Those frameworks load, filter and run tests internally, exposing a much smaller interface surface to the external runner. It may be possible to create a custom harness for use with the MS test framework, but it looks like a lot of work. Ideally (for NUnit, anyway) the harness should be abstracted to allow for frameworks that don’t rely on an external runner to invoke tests.

Use of a separate console runner would be useful for those tests that do not rely on being run in a browser environment. It wouldn’t solve all problems but would permit automated CI testing, at least for non-Gui components. This may be an option we will want to explore. It’s possible that we’ll want to support several ways of running Silverlight tests.

I’ve been travelling for the past six weeks, so nothing much has happened on NUnit. Now that I’m back, it seems like a good time to summarize what the next steps will be for NUnit, NUnitLite and related programs.

NUnit 2.6.1
A number of bugs and fixes have accumulated against NUnit 2.6. I expect to release NUnit 2.6.1, with the fixes, this month. Note that the 2.6 series is stable and will only receive bug fixes and minor improvements from now on. The next feature release will be NUnit 3.0.

Visual Studio Test Adapter
Now that VS 2012 RC is out, I’ll be re-testing the adapter, fixing a few bugs and will release a new version this month. This will be the third Beta release, with version number 0.93. This release will use NUnit 2.6.1.

NUnitLite
The NUnitLite 0.7 release attracted more attention than earlier releases, leading to a number of bug reports. That’s a good thing! I expect to put out an 0.8 release in August. Following that, my plan is to bring NUnitLite up to the equivalent level of functionality of NUnit 2.6 for all common features and work toward a 1.0 release by the fall. Further NUnitLite releases from that point on will be part of NUnit 3.0 and will use the same codebase as 3.0.

NUnit 3.0
There will be another pre-release of NUnit 3.0 in August, numbered 2.9.6. You can expect one or two more of these over the summer. By fall, I expect to have the first actual 3.0-numbered version, an alpha-level release, incorporating both NUnit and NUnitLite.

The NUnitLite project has been somewhat dormant for a while, with the last release, 0.6, having come out in 2010. About a month ago, I learned from Sebastien Pouliot that MonoTouch is using NUnitLite as the basis for it’s unit tests, which motivated me to put on some finishing touches to a new release, NUnitLite 0.7.

This release puts NUnitLite on a somewhat equal footing with full NUnit, at least in terms of the supported syntax for writing tests. Most attributes and constraints are now supported, at least when running on a desktop operating system. When running under the compact framework, certain things are not available because the framework doesn’t support them. If you don’t need the gui or a features implemented above the framework, like running in a separate process, NUnitLite can pretty much do the job for many projects.

This release of NUnitLite is the first one to provide pre-built binaries. Separate builds are produced for .NET 4.0, 3.5, 2.0 and 1.1 and for compact framework versions 2.0 and 3.5. You can download them from our Launchpad project site or using NuGet

Today, I’m releasing version 0.92 of the NUnit Adapter for Visual Studio and creating a new project under the NUnit umbrella to hold it’s source code, track bugs, etc.

In case you didn’t know, Visual Studio 11 has a unit test window, which supports running tests using any test framework for which an adapter has been installed. The NUnit adapter has been available on Visual Studio Gallery since the initial VS Developer preview. This release fixes some problems with running tests that target .NET 2.0/3.5 and with debugging through tests in the unit test window.

For a bit more info about how to install and use the adapter, see the docs

Here are some links to the NUnit Adapter

• Our project page on Launchpad
• Our bug tracker
• Our Visual Studio Gallery page

Let’s say you have an array of ints representing years, all of which should be leap years.

One way to test this would be to write a custom constraint, LeapYearConstraint. You
could then use it with the Matches syntax element to write your test as

Assert.That( array, Is.All.Matches( new LeapYearConstraint() );

But creating a new constraint for this adhoc problem seems like a bit of overkill.
Instead, assuming you are working with C# version 3, try this:

Assert.That( array, Is.All.Matches( (x) => x%4 == 0 && x%100 != 0 || x%400 == 0 );

If it fails, it will give a generic message: “Expected: matching lambda expression” since NUnit is actually
built with .NET 2.0, but for a quick test it may be just the tool you need.

NUnit 2.5 has so many new features (see the release notes) that I thought I’d try to come up with my top-ten favorites. It was hard to get down to ten, but here’s what I came up with…

Reason 1: Data-Driven Tests

Users of mbUnit and xUnit.net have enjoyed the flexibility that data-driven (aka parameteried) tests provide for some time. NUnit implements this paradigm in its own way, with it’s own set of attributes. Test methods may have arguments and the data for them may be supplied in a number of ways: inline, from a separate method or class or randomly. This feature gives you a succinct way to express a set of examples to be used in running individual test cases.

Reason 2: Theories

As used in NUnit, a Theory is a generalized statement of how a program should operate, like “For any positive number, the square root is defined as the positive or negative number, which, when multiplied by itself, gives the original number.” Traditional, example-based, testing allows you to select one or more sets of values to use in testing such a program. A Theory, on the other hand, allows you to express the generalization itself, writing a test that will pass for whatever values are passed to it, provided they meet the stated constraints. David Saff has written a number of papers about the use of Theories in testing and has implemented this construct as a part of JUnit. Now you can use the same construct in any .NET language.

Reason 3: Inline Expected Exception Tests

Testing that an expected exception is thrown correctly has always been an issue in NUnit. The ExpectedExceptionAttribute has been available since early releases but has a number of problems. It tests that the exception was thrown somewhere in the test, without specifying the exact place in the code, and it is subject to the syntactic limitations that apply to use of an attribute. With the introduction of the Assert.Throws assertion method and the even more powerful constraint expressions Throws.Exception, Throws.InstanceOf and Throws.TypeOf, exception testing logic can now be moved right into the test along with any other necessary assertions.

Reason 4: Generic Support

NUnit 2.5 provides separate framework assemblies for .NET 1.x and 2.0+. Using .NET 2.0 or higher, a number Up to 2.4, NUnit avoided any use of Generics, in order to maintain backward compatibility. In 2.5, the framework assembly used under .NET 2.0 provides a number of generic Asserts and Constraint expressions for convenience. More significantly, your test methods and classes may now be generic, and NUnit will specialize them using the types you provide.

Reason 5: Lambda Support

If you write your tests using C# 3.0, you may use Lambda expressions in a number of places where NUnit expects a delegate. This is particularly useful in providing a custom definition of equality without explicitly defining an IComparer<T> and can even be used to apply an arbitrary predicate to the members of a collection.

Reason 6: Out-of-Process Execution and Runtime Selection

NUnit 2.4 ran all tests within the same process, using one or more AppDomains for isolation. This works fine for many purposes, but has some limitations. NUnit 2.5 extends this concept to running tests under one or more separate processes. Aside from the isolation it provides, this allows running the tests under a different version of the .NET runtime from the one NUnit is currently using.

Reason 7: PNUnit

PNUnit stands for “parallel NUnit” and is an extension developed by Pablo Santos Luaces and his team at Codice Software and contributed to NUnit. It’s a new way to test applications composed of distributed, communicating components. Tests of each component run in parallel and use memory barriers to synchronize their operation. Currently, pNUnit uses a special executable to launch its tests. In the future, you will be able to run pNUnit tests from the standard NUnit console or gui runner.

Reason 8: Source Code Display

The new stack trace display in the Errors and Failures tab of the Gui is able to display the source code at the location where a problem occured, provided the source is available and the program was compiled with debug information. Currently, syntax coloring for C# is provided and other languages are treated as simple text, but additional syntax definitions will be available in the future.

Reason 9: Timeout and Delayed Constraints

These are two separate features, but they are related. Besides, I’m working hard to keep this down to only ten points! It’s now possible to set a timeout, which will pre-emptively fail a test to fail if it is exceeded. This may be done on a method, fixture, assembly or as a global default. On the other hand, if you need to wait for an action to take place after a delay, you can use the After syntax to delay the application of the constraint. NUnit will subdivide a long delay and apply your test repeatedly until the constraint succeeds or the specified amount of time is up!

Reason 10: Threading Attributes

In past releases, if any test needed to run in the STA, the entire test run had to use the STA. With 2.5, any method, fixture or assembly may be given an attribute that causes it to run on a separate thread in the STA. Other attributes allow requiring an MTA or simply running on a separate thread for isolation. This can eliminate a lot of boilerplate code now required to create a separate thread, launch it and capture the results for NUnit.

This is my own list, of course. Yours may vary. Download the release, try it out and let me know what your own favorites are.

The latest code for NUnit 2.5 includes seven generated files, including the Assert class and most of the classes that allow you to write constraint expressions using the NUnit fluent syntax. Some people have asked if generating these files is worth the effort, since the code created is very simple anyway.

There are two reasons for generating this code. The first relates to the syntactic constructs. While it’s relatively straight forward to create a custom constraint and various people have done so, such constraints must be used by invoking their constructors rather than by use of a simple key word. So, for example, if you have written an OddNumberConstraint that tests whether a number is odd and displays an appropriate failure message, you are still not able to write Assert.That(num, Is.Odd) without directly modifying NUnit.

It turns out, based on experience of several people who have tried, that the syntactic modification has a lot of places where you can go wrong. You have to modify at least three additional files, even after you have written the constraint. Using NUnit’s code generation facility, you would simply add a line like this to NUnit’s SyntaxElements.txt:

Gen3: Is.Odd=>new OddConstraint()

Then, after running NUnit’s code generation tool, the files Is.cs, ConstraintFactory.cs and ConstraintExpression.cs would be updated. After rebuilding NUnit – or just the framework – the statement Assert.That(num, Is.Odd) would compile and work correctly. If you wanted a classic assert, you could add the line

Gen: Assert.IsOdd(int num)=>Assert.That(num, Is.Odd)

and Assert.IsOdd would become available for your use, including overloads with an error message and optional arguments.

So, one good reason for generating code is to make it easier to extend NUnit. But an even more important reason is reliability. Take the Assert class as an example. Some of the methods have as many as 24 overloads. In the past, we have seen hidden bugs that affected only one infrequently used overload. By generating the code, we can ensure that the same logic is used in each overload. This doesn’t prevent errors, but it does make it likely that the error will be caught, since it will generally impact many of the overloads in the same way. What’s more, the layout of the SyntaxElements file puts things that need to be updated together right next to one another, so it’s much harder to forget a step.

The NUnit code generation program, GenSyntax.exe, is distributed with the NUnit source, in the tools directory.