This brings the new features from the Silverlight 4 version to developers who have a business reason to continue with Silverlight 3 development for the time being:

• Updated user interface
• Integrated “tag expression” support for selecting subsets of tests to run
• Performance improvements
• Removal of dependencies on XLinq and System.Windows.Browser assemblies

These are also the bits you want for Windows Phone testing.

Why wasn’t there one for Silverlight 3 already?

Since the April 2010 toolkit release was targeted for Silverlight 4 developers only, these bits were not released for version 3 (though were built from the same source).

Why doesn’t the April 2010 toolkit work for Windows Phone development & testing?

The Windows Phone development environment is based on a version of Silverlight 3 today, and so that means that you need to use Silverlight 3 binaries and class libraries when re-using Silverlight apps and code – not Silverlight 4.

These bits work well for phone developers and replace the older bits I had posted to my MIX talk site.

Download the binaries

[Silverlight 3 binaries for the Silverlight Unit Test Framework, Zip 452KB]
You may need to ‘unlock’ the binaries for security reasons once downloading and extracting the assemblies from the .Zip. To do this, right-click on the file and select the ‘Unlock’ button.

These binaries are strong named as before, but are not Authenticode signed, so they are not official. This helps work around some Windows Phone signing issues I’ve heard reports of.

Hope this helps,
Jeff

The session will be more comprehensive than the overview talk I presented at MIX 10 (which is available as a free online video stream now).

Click here to view the talk information on the Tech Ed site. The tentative information is as follows, but please double-check with the conference schedule builder before showing up:

Unit Testing in Microsoft Silverlight
Wednesday, June 9, 2010
11:45 AM – 1:00 PM
Room 287
Session Code WEB205
http://northamerica.msteched.com/topic/list?keyword=WEB205

Hope to see some friendly faces in the audience! I’ll be up there in some sort of speaker polo shirt uniform with khakis, apparently this conference isn’t quite as trendy and cool as MIX in Vegas .

If you’re going to be attending Tech Ed, I’d appreciate any early comments you have as there is still time to shape the content of the talk.

Rich, modern user interface

The user interface no longer uses the antiquated HTML DOM bridge for displaying results, but now is a data bound, rich Silverlight application itself.

Rich results information is available, and a tree view control allows you to select results to be included in a test report that can be copied to the clipboard. Failures are automatically selected.

The results pane includes information about the tests that ran, as well as detailed information for failures.

Descriptive text helps identify negative tests, known test issues and bugs, and other important metadata that was not always exposed in the previous test framework.

Integrated tag expression support

The startup experience now allows you to enter a tag expression to select which tests to run, or not to run. You can add the Tag(“”) attribute to any test methods or classes to define your own tags. Also, the full and short names of tests and classes implicitly become test tags, as well as priorities.

This means that you can define tags such as:

Out of Browser support

The framework now supports running out-of-browser. Paired with the new Silverlight 4 Tools for Visual Studio 2010, you can press F5 to run a test project you have marked as an out-of-browser.

This also opens up an opportunity to do testing of elevated apps that make use of AutomationFactory and other advanced features on Windows.

Simplified dependencies

The test framework removed dependencies on System.Windows.Browser (from the platform) and System.Xml.Linq (XLinq, from the SDK).

Building blocks for Windows Phone testing support

This version also contains the full source to the phone testing interface, which is significantly different in visual design and experience to coincide with the Windows Phone device size.

However, the story for phone testing is still developing: in the meantime binaries designed for phone use can be found at http://jeffatmix.com/

New location for binaries

Since the new rich user interface contains a number of controls, the traditional design-time integration system conflicts and presents some issues.

As a result, we had to move the binaries from the Bin directory of the toolkit installation folder into a new Testing directory.

So the two test framework assemblies (Microsoft.Silverlight.Testing and Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight) have moved.

This location change could cause existing applications that reference the test assemblies to fail to compile due to the "missing" references.

If that happens, please delete the two broken references and create references to these assemblies in their new location. Everything else should continue to work as expected.

Simplified test project template

In alignment with the test project changes in Visual Studio 2010, which are more simplified, the unit test project template for Silverlight no longer includes a README.TXT file.

Known issues

Test service

I’ve received reports from a number of people about issues with the test service automation support. Some of the NDA/preview users of the framework reported issues that were corrected, but there may still be some bugs out there. I’m working to investigate and post a fix if and when I know more.

In the meantime I know that StatLight, a third party system, has been updated for the April release and is working well for automation.

Test panel and visuals

Please beware that some of your tests may need to be updated to use the new framework if your application, utility functions, or tests ever used the Application.Current.RootVisual (against the recommended pattern for interface testing).

The root visual is now the test framework application runner itself, and the tests run within the same TestPanel as before. Controls added to the test panel will not be located at the 0, 0 screen position, but likely near the bottom of the screen instead.

Hope this helps. To get started, download and install the April toolkit!

Peppermint-Demo.zip [Zip, 1.2 MB]

Silverlight:

Requires Visual Studio 2010 (or express)
Requires the Silverlight Toolkit
Requires the test framework binaries included in the zip

Windows Phone:

Requires the Windows Phone development tools CTP
Requires the test framework binaries included in the zip

An early preview set of bits are online at http://jeffatmix.com/ – and the next release of the Silverlight Toolkit when Silverlight 4 ships will contain support for the phone officially. Stay tuned for more.

Over the next few weeks, I’ll be blogging plenty on these subjects.

UNIT TESTING SILVERLIGHT AND WINDOWS PHONE APPLICATIONS
Jeff Wilcox in Lagoon B (Mandalay Bay)
Tuesday, March 16, 2010 2:05 PM (30 minutes)

Session materials, slides and code will be posted after the session at http://jeffatmix.com/.

Synposis from http://live.visitmix.com/MIX10/Sessions/CL59:

Learn how to create and maintain Silverlight and Windows Phone 7 Series applications using the Silverlight Unit Test Framework. See what tools are available to easily validate controls and application interfaces, add automatic testing to builds, and gain a solid understanding of test principles to deliver great experiences for your clients and customers.

What’ll be new at MIX for testing?

• Support for testing those Windows Phone 7 Series apps!
• New release of the Silverlight Unit Test Framework, with a modern user interface, Out of Browser support, and more.

Hope to see you all in Vegas! And if not – these sessions always make it online super quick. Let me know if there’s anything in particular you want covered in the talk.

New infrastructure & test tools ship in the Silverlight Toolkit

There is a lot in the release that is joining publicly for the first time, based on some of the internal tools and utilities that we use in building and testing the Silverlight Toolkit, plus things we’d like to have as typical Silverlight developers.

These tools join in the Experimental quality band, and over the next few posts, I’ll dig into the details. This post is a literal walkthrough of how you could go about seeing all the utilities in the meantime.

One important thing to call out is that, as an initial release, there are definitely some rough edges: the test tools are centered around Microsoft Build (msbuild) integration, instead of Visual Studio integration; there’s no add-in or nice right-click project support to use these tools today.

We’ll be collecting feedback along the way and making changes in future releases. Until then, hopefully some of you will find the infrastructure useful, if only to peak behind the curtain. We’re also shipping the full source to these tools, you’ll find an Infrastructure.zip file inside the toolkit install folder.

What we’ll do in this walkthrough

This walkthrough uses Silverlight 4, though the instructions are virtually identical if you’re using Silverlight 3.

• Install the Silverlight Toolkit
• Create a new Silverlight class library with a simple business object
• Add a Silverlight Unit Test Application using the new templates found in the Silverlight Toolkit
• Run the tests in Visual Studio
• Run the tests from the command line in various browsers
• Collect block-level code coverage information for the build

This is screen-shot heavy, as future posts will dig into the details. I appreciate your patience!

Requirements:

• Latest Silverlight Toolkit
• For test automation:

• MSBuild on the machine

• Visual Studio 2010 Beta 2 Ultimate

Installing the tools

When you install either Silverlight Toolkit drop from November, you’ll see that there’s a new feature listed in the setup: the Tools & Templates feature. It’s selected by default, and adds Visual Studio templates, installs the tools, and prepares the tools if your machine has the proper dependencies on it.

Create a Silverlight Class Library

Open Visual Studio 2010 Beta 2, after installing the toolkit. Create a new Silverlight Class Library project, we’ll store a simple business object in it.

Whichever flavor of Silverlight you use, remember to use the same flavor while creating future projects in the same solution!

Create a new Person.cs type, with a few properties and a method. Here’s the source:

using System;

namespace MyApplication
{
    public class Person
    {
        public string First { get; set; }
        public string Last { get; set; }

        public string FullName { get { return First + " " + Last; } }

        public bool MightBe(string substring)
        {
            return FullName.Contains(substring);
        }
    }
}

Add a Silverlight Unit Test Application

Right-click on the solution in the Solution Explorer and select Add New Project. Under the Visual C# (and also Visual Basic) languages node, select the Silverlight subgroup. You’ll see a Silverlight Unit Test Application project template.

You now have two projects:

Next up, we want to add a reference to the class library, so that the unit tests can access the library. Right-click on the new test project and then select the Add Reference option. Under the Project tab, choose the class library you created earlier.

Now, let’s add some unit tests. Clear out the content in the created Tests.cs file and drop this in:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using MyApplication;

namespace PersonTestProject {
    [TestClass]
    public class Tests
    {
        [TestMethod]
        public void TestCtor()
        {
            Person p = new Person();
        }

        [TestMethod]
        public void TestSetName()
        {
            Person p = new Person { First = "Scott", Last = "Guthrie" };
            Assert.AreNotEqual(p.First, p.Last);
        }

        [TestMethod]
        public void TestGetFullName()
        {
            Person p = new Person { First = "Steve", Last = "Ballmer" };
            StringAssert.Equals("Steve Ballmer", p.FullName);
        }
    }
}

The tests are pretty simple, and exercise various simple PMEs on the type.

Run the unit tests from Visual Studio

Right-click on the test project and select Set as StartUp Project from the menu. Then, press F5 or click the Run button to begin execution.

The default web browser will open up, the tests will run, and the window will stay open. When you are satisfied, close the browser. You’ve just run some simple tests!

Running unit tests from the Visual Studio command line

More interesting is being able to automate the tests: through MSBuild, we’ve added a task that can control the browser and save out the log file(s) from the unit tests when they run.

To do this, you need a few things:

• The Visual Studio command prompt open.
• The full path to the test project.

Open the command prompt

Personally, I always pin the Visual Studio 2008 tools command prompt next to my Visual Studio taskbar icons.

You can start it from the Start Menu, under the tools for the Visual Studio version you are using.

Alternatively, make sure MSBuild is in your path.

Or, start a command prompt and then move to your 32-bit Program Files, then ‘Visual Studio 9.0’, then VC. Run ‘vcvarsall.bat x86’

Note: all these command prompts should be 32-bit, even on a x64 machine. Silverlight is a 32-bit world today.

Move to the test project

To get the full path to the test project, I just right-click on the test project in Visual Studio, then Open Folder in Windows Explorer. I copy the path from the resulting Explorer dialog’s crumb navigation bar.

Now, inside the command prompt window, move to that path:

pushd (PASTE PATH HERE) <enter>

To run the unit tests in the default browser, simply type

msbuild /t:test

and press Enter.

You’ll see the browser open, quickly run some tests, and then close. Here’s what it looks like, captured in the middle of the 5-second test run:

And afterwards, you’ll see that Msbuild reports success, and you can read the test to see that 3 passing tests were reported, out of 3 total tests.

Running tests in Google Chrome

It’s easy! Just set the browser property to Chrome.

msbuild /t:test /p:browser=chrome

And just like that, Chrome opens up and runs the tests.

Running tests in Mozilla Firefox

Just set the browser to Firefox.

msbuild /t:test /p:browser=firefox

Tag Expressions

A nice feature to help select a subset of tests is the Tag Expression syntax. By specifying a tag expression at the command prompt, you can include and exclude tests that are marked with the Tag attribute found in Microsoft.Silverlight.Testing.

Also, tags implicitly exist for all test method names, short and full.

So, let’s run the test called TestSetName from our project.

msbuild /t:test /p:tagexpression=TestSetName

You may briefly see indication in the test UI in the browser that a ‘Tag expression’ is in use.

And the build results show that just one test ran:

If you want to run all tests except that one, use this tag expression:

msbuild /t:test /p:tagexpression=!TestSetName

Cool, and easy!

Test results files

Another nice feature of running the tests through MSBuild is that you’ll see test result files (end in *.trx) inside the same folder as the test page for the application.

Similar and conformant to the Visual Studio *.trx format, you can parse and work with this data to understand execution times, results, and read other information. Unfortunately, you cannot open these files in Visual Studio 2010 Beta 2, but they do open in Visual Studio 2008. Note that there isn’t any real test integration with VS here: this is just an informative display.

Here’s a directory with several test result files (plus some coverage stuff we’ll get to later):

You can open up the TestResults trx file and see what kind of information is in it:

And here’s the results file opened in Visual Studio:

Preparing for Code Coverage

Ripe for a Visual Studio add-in, this process requires us modifying the test project’s .csproj file some. Instructions are the same for Visual Studio.

We need to manually select the assembly to be instrumented by setting a property. The assembly to be instrumented must not be signed, and not have a strong name. In this example, the assembly is SilverlightClassLibrary3.

To make changes to the test project, to specify this, first unload the project by right-clicking on the test project and selecting the Unload option:

Then, right-click the project file that is ‘(unavailable) and select ‘Edit’.

Now you’ll see the XML data for the project. Scroll down in the template to where there is a commented out ItemGroup, and some comments about the code coverage support. Remove the comments from the ItemGroup:

And now, change the Include statement from SilverlightClassLibrary1 to whatever your assembly to instrument is called. In our example here, it is called SilverlightClassLibrary3.

Now, that’s it. Let’s close the file and then right-click the project again and select Reload Project:

Now we’re ready to go!

Running Silverlight Code Coverage

Just like the msbuild /t:test from above, we have a similar target named CoverageTest and CoverageView. CoverageTest collects the data, while CoverageView collects the coverage data and then shows the results in an application that explores the hit and not hit portions of your source code.

Let’s collect coverage!

Note: This is assuming that you have a high-level SKU of Visual Studio 2010 Beta 2 installed. The coverage tools depend on the Static Analysis Tools that ship in the beta. You will receive an error if you run this on a machine without those tools. We’ll look for a better experience in the future.

Remember, the same properties from above – browser and tag expression – still can apply and work in the coverage targets.

msbuild /t:coveragetest

Here is what it will look like if everything line up. Unfortunately, the coverage experience is a very early preview and very flaky in some situations due to the dependency on beta components. We also haven’t had I admit enough time to iron out all the issues customers may experience.

You see the Instrumentation messages come out, then the RunTests target, then the coverage data is merged and things should be successful.

If that worked, it is safe to assume that the view target will also work. It will re-run all the tests and instrument again as well.

msbuild /t:coverageview

Now you’ll see the viewer popup, that lets you drill into types and methods to see what is hit (cyan) or not hit (red), to spot code that your tests are missing:

You’ll find a Coverage.xml file in the same directory where the test results go. This is the data that is used by the viewer application. You’ll see in the XML file a set of ‘visited’ blocks, and other information, in the file:

More to come soon!

Download the Silverlight Toolkit November 2009 Release Today

The Silverlight Toolkit is a collection of Silverlight controls, components and utilities that help make Silverlight development a little easier, more fun, and add value outside the regular Silverlight release cycle.

The sixth release of the Silverlight Toolkit, the November 2009 release targets Silverlight 3. There is also a release available that targets the new Silverlight 4 Beta for developers.

Resources of note:

• Learn more about the Silverlight Toolkit
• Download the November 2009 Release
• Get started with Silverlight development
• Participate in the controls community at Silverlight.net
• Check out questions and answers on Stack Overflow using the ‘silverlight-toolkit’ tag
• The Issue Tracker can be used to report issues, add to the feature wish list, and vote on the work to be done by the Microsoft team

Hope you enjoy our new release!

Download the binaries

I’ve updated the MSDN code site for the product with the Silverlight 3-specific binaries, for anyone that is interested.

103K, Zip: Direct link
MSDN Code Gallery web site for the framework

Find the binaries in the Silverlight Toolkit

If you’re using the Silverlight Toolkit July 2009 release for Silverlight 3, you’ve already got the Silverlight unit test framework binaries on your machine. They’re located in the source code package (Source code.zip).

Here’s how to track them down…

When installing the Silverlight Toolkit

Make sure that you keep the “Source Code” feature of the Silverlight Toolkit checked. That will place the zip package on your system with the source, plus the unit test framework binaries.

Locate the source code for the toolkit

Go ahead and find the zip package after installation, and extract it to your system somewhere.

Jump into the Binaries folder of the extracted package

Use the Silverlight unit test framework

And now you’re good to go. You’ll find the binaries in that folder, and you should add references to both Microsoft.Silverlight.Testing and then the unit test metadata assembly, Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll

What’s new in the July 2009 bits of the Silverlight Unit Test Framework?

• Support for the [Timeout(...)] attribute on test methods.
• Improvements to the Test Service interface. The service now has an asynchronous reporting phase that allows for more robust automation options (details coming soon).
• Minor interface changes
• Minor bug fixes

No fundamental differences between the Silverlight 2 and 3 versions

There are no fundamental differences between the versions, and the Silverlight 2 version of the framework works fine with Silverlight 3 application development. But it makes sense to use the appropriate version.

Looking forward, we may move the unit test framework right into the Silverlight Toolkit, for ease of use.

Hope this helps!

• Simple Python unit testing
• Super awesome – Testing a C# app using IronRuby

If there’s anything else nice out there in the wild, let me know! Note that these don’t use the Silverlight unit test framework, but rather some nice dynamic language test goods.

• Test Driven Development for Silverlight – Introduction
• Part 2 – covers a navigation application
• Part 3 – jumps into integration testing (more than just unit testing)

It’s great seeing the community getting involved in expanding knowledge and the references out there. Thanks Eric!

This is a discussion, and a great whiteboard exercise, that I’ve had with many folks. I figure, it is finally time for me to at least blog a little bit about it. For the history behind this decision, see the bottom of this post.

As usual, you’ll find the unit test framework binaries and sample tests in the Silverlight Toolkit download. Other posts by me on this topic can be found here.

If you use the asynchronous work item functionality in the test framework, do realize that your tests will no longer be subset compatible with the full desktop Visual Studio unit test framework.

What’s happening during a unit test run

The unit tests that run in the browser need to leave time for the browser, the native Silverlight plugin, the .NET CLR, and other components to do work between calls. This means that much of the execution is driven forward using dispatchers.

If you compare a typical test run on Silverlight to one in a desktop unit test framework, and compare how they run over time, you get this:

So, you get this interesting stack:

• Browser is rendering, calls into
• Native Silverlight plugin
• .NET runtime
• Silverlight test application
• Silverlight unit test harness, test assembly, class, and method management types
• Your test method

If you want to test the DOM being updated by a test, for instance, in the real browser (not mocked), you have to effectively let the call stack unwind enough for the browser to update the DOM… then let the test framework eventually jump back into action to process the remainder of the test.

What happens when an [Asynchronous] test executes

The AsynchronousAttribute from the Microsoft.Silverlight.Testing namespace, and in the assembly of the same name, tells the test framework:

Hello! I’m going to do some stuff now. And when this method returns, well, I’m going to continue doing work. So don’t report any results yet…

… In fact, wait for me to send you a ‘test complete’ message – or throw an unhandled exception – to make a call, OK?

This means that the entire methods executes, and then any queued up work items are run afterwards, each in due time. Here’s a way to compare a standard (single invoke) test to an asynchronous test:

Asynchronous in this test framework’s context means that additional work is queued up – NOT that it happens immediately on a separate thread.

You use “Enqueue” methods to add work items to the list of things to do before the method should be considered a success. And the final thing you should do (in a successful test case) is call the TestComplete method on the base class. The EnqueueTestComplete can do this for you, if you like.

The work items built into the test framework are for calling some actions (EnqueueCallback), a conditional predicate enqueue (EnqueueConditional), and a dirty and evil-to-use EnqueueDelay method that will sleep a minimum amount of time before allowing other work items to run.

So, here’s a look at what happens when some regular tests run:

And then a more complex run, with a test that includes some enqueued work that needs to execute – extending the test method to run beyond the lifetime of the actual initial test method that is executed:

Using the enhanced test base classes

All of the enqueue methods in the Silverlight unit test framework are non-standard, unique, and a little crazy. I say this because I don’t see Microsoft’s larger Visual Studio test team ever adopting this in a unit test framework – it is contrary to most execution principals for a test framework.

So if you use these enhanced base classes, you won’t necessarily be able to cross-compile them on the desktop unit test framework, and make use of them elsewhere. You’ve been warned!

A “work item” as the test framework uses them is a simple type (see reflector) that is evaluated for its completion state, and invoked. Once the work is complete, the work item will be removed from the test dispatcher, and execution will move on to the next work item in due time. A work item could be composite, in that its children are called and completed before the item itself reports a complete state.

If your test inherits from WorkItemTest (or PresentationTest, or SilverlightTest), you’ll be able to call from your tests:

TestComplete()

Instructs the framework that your test method is finished, and to move onto the next result. This can be added to event callbacks, delegates, etc. You should not do any more work or your test after calling this.

EnqueueTestComplete

Enqueues an action to call TestComplete. This is the most-used way of calling TestComplete when a set of work is done, and would typically be the last Enqueue* method call.

EnqueueCallback

Enqueues an Action (delegate, simple lambda, etc.). The work item effectively calls the Action, then moves on. Alternatively, this enqueue method also takes an array of Actions, allowing you to chain many Action calls together in order.

EnqueueConditional

Takes a Func<bool> conditional statement / predicate. Each time the work item is evaluated, the function is called. When it returns True, the work item is complete and execution will continue. After each invoke of the predicate, the test framework will unwind the stack, allowing other work to happen, before coming back to try the condition again.

EnqueueDelay

This method will enqueue a work item that takes either a TimeSpan object or an integer representing the number of milliseconds at minimum to delay before continuing. This is not an exact timer, but rather, a way to ensure that at least a minimum amount of time continues. It is more like a DoEvents call than it is like a Sleep call, since it will not block the UI thread.

We’re also looking to add support for the Timeout attribute in the future, so tests don’t run forever.

Of course, you can browse the complete source code to the test framework within the CodePlex source code for the Silverlight Toolkit.

Typically, you’ll see a Silverlight test that prepares an object or control, adds it to the test surface, hooks up to the Loaded event for the control, then uses a set of enqueued conditionals and callbacks to validate a series of events or properties, finally calling TestComplete.

Alternative Silverlight test frameworks

Today are alternative frameworks available for those of you that would rather not test inside a browser. Many of us have already worked hard using this framework to test the platform itself, so the reward is that you can try something different in your test methods if you want to.

Some people on the web have made improvements or changes to the primary framework, others have come up with new and exciting takes on this all, including removing the runtime and Core CLR from the picture through extensive mocking:

• SilverUnit is one such solution, and it’s written in VB.NET, too! Requires a TypeMock license.
• Einar Ingebrigtsen’s Silverlight Unit Test Runner for Resharper
• Silverlight-Selenium – adds support for Silverlight communication through the HTML interoperability feature of Silverlight 2
• David Anson often runs his chart control tests on the desktop framework by using some clever mocking. I’ll ping him for details sometime.
• I previously blogged how to use the same Silverlight unit test framework for running NUnit tests in the browser

And, when it comes to mocking, there’s the SilverUnit listed above, plus:

• Rhino Mocks 3.5 for Silverlight
• Moq – source available

Let me know if I’ve left anything off the list, I’d like to link to other testing solutions as they come online. This is a space where I think the .NET community at large really is amazing.

Recently covering testing and Silverlight was Tim Heuer, and also Justin Angel. Check those posts out.

History behind the thread decision

When I started on this journey a few years ago, developing a robust, solid testing platform for Silverlight, the first internal release was actually using background threads & even running in a separate host (not the browser). It was effectively a Core CLR testing platform that re-used the metadata from the Visual Studio unit test framework. Soon enough, it became clear that we needed to track down differences in the platform, beyond just letting our code run. There was a cause/effect to many operations, and we were primarily looking at control development and how to enable easy testing for both the development and the test teams in the company, when doing Silverlight work.

The VSTT metadata was still key, regardless of the execution environment, since so many people are familiar with it.

By redesigning the test product to instead be a Silverlight application itself, and run tests on the UI thread, we could improve the actual quality of our code by validating the platform along the way. We’re still finding positive side effects from this, and the only real cost has been the additional execution time compared to full CLR test solutions.

So, we settled on the UI thread for a majority of the testing. Yes, the exact opposite of what almost all other unit testing frameworks do. You would probably be really surprised to hear how many very interesting, and important, bugs we’ve found over the years of using this framework. Silverlight runtime changes, occasional regressions, parser issues, and a lot of key things that have actually improved the quality of Silverlight and the Silverlight control model in general.

Whether this exhaustive, in-the-browser execution may be good for your project may require some thought given your goals, since the tests are slower to run than alternatives. But if you’re doing anything wild (testing staged or live web services, wanting to test in all the browsers your customers use), etc., then you’ll find the Silverlight unit test framework to be real helpful. As a result, it also works on all those fun browsers that Silverlight works on – IE, Firefox, Firefox, and yeah, I even use it with Chrome. Works on Mac and PC, and I’m sure, someday soon, Moonlight.

What we have here is so much more than a “unit test framework,” and in my mind, this is actually a good base for some functional tests, plugging in additional reporting services, and much more involved that just running a few quick assertions on simple types behind the scenes. I know I’m ok with that, at least.

With the rich presentation platform’s emphasis on cross-platform, lightweight components, there’s a need to step away from a typical mocking solution. If you’ve ever wanted to perform verification of default templates, you probably understand the problem: the test needs a little time for the template to be applied after initialization, and you’ll be interested in how Silverlight actually processes that template.

In a nutshell, that’s where the [Asynchronous] attribute came from that the test execution logic handles to enable these test scenarios. Happy testing!

This isn’t a "metadata redirection" trick – but the actual NUnit assertions and metadata running inside the Silverlight unit test engine. This took about 15 minutes to implement from start to finish – shorter than the amount of time it’ll take me to publish this post.

Though NUnit is great for legacy components and tests, I strongly recommend using the built-in Visual Studio metadata for most projects. The metadata is used by enough folks throughout Microsoft and the industry to be well-supported, tested, familiar, and not require the custom unit test provider hookup in your test projects.

That said, I do hear about legacy business objects and NUnit tests that devs would like to be able to verify and run in the browser + Silverlight environment.

You can read about NUnit attributes and assertions in the official documentation. So, here goes…

Running sample & validation tests that ship with NUnit

Inside the traditional NUnit source download, there’s a sample called "Money" that tests a simple interface and types. Here’s that "Money" project running on OS X/Safari, via the Silverlight unit test framework and NUnit:

 
"Money" sample test project: Click on the image or here to run the Silverlight test application on your machine. Tests will start in ~5 seconds.

If you have a set of legacy business objects and associated NUnit tests, you’ll find it easy to run them within Silverlight now.

To further demonstrate the compatibility here, this is the "NUnitTests" test project that validates the core NUnit framework and its assertions, running inside Chrome:

"NUnitTests" that validate the NUnit framework. Click on the image or here to run the tests.

Silverlight unit test framework is easy to extend

When designing the Silverlight unit test framework, it became clear that there were three or four distinct test engine components that could be combined into a single test system. Yet, they were distinct enough to allow alternative implementations and extensions to be built.

The typical test application .XAP file ends up containing the general Silverlight unit testing engine (that performs test execution, and is very different from typical engines in that it runs on the UI thread and is synchronous), your actual tests marked with the appropriate metadata, and then the metadata and unit test assertions that you’re using.

Inside the unit test engine, there’s a base test harness class, and then the UnitTestHarness implementation that does most of the heavy lifting. A set of interfaces abstract out the reading and processing of unit test metadata. Interfaces aren’t always a great answer, but the value added in this case.

The metadata/assertion implementation that you use (NUnit or VSTT, for example) takes care of throwing exceptions when assertions fail, that the framework then processes in a generic way.

The Silverlight unit framework ships with the same metadata used in the Visual Studio team system’s unit test framework, making it easy to move between desktop and Silverlight application and test development. A "VsttProvider" sits between the Visual Studio metadata and the actual unit test engine, and ships in the Microsoft.Silverlight.Testing assembly.

OK. That’s a mouthful, but it means that you can implement your own metadata provider to support running other unit tests. Since NUnit and VSTT are so similar, this was easy – but more intricate frameworks might need to modify or extend the actual unit test system to provide the right experience.

Getting NUnitFramework to build

NUnit is relatively old; it is from a different era of .NET development. NUnit utilizes the old .NET Hashtable and ArrayList types.

Also, the .NET framework base class libaries included in Silverlight are a subset, rely more on .NET 3.5 features, and exclude some of the older types.

To easily support building NUnit with the Silverlight subset of the framework, here’s the list of changes I needed to make to the NUnitFramework project:

• Add simple implementations of Hashtable.cs and ArrayList.cs, utilizing modern generic collections under the hood
• Inside ConstraintBuilder.cs, update the Stack type to be a generic Stack<object>
• Added a shim/empty SerializableAttribute
• Comment out constructors in IgnoreException.cs and AssertionException.cs that were designed for serialization (lines 33-35 and 35-37, respectively)
• I removed the strong name assembly attributes from AssemblyInfo.cs, this isn’t actually required.
• On line 92 of CollectionConstaints.cs, changed the DictionaryEntry typed temporary variable to be of type "var" (since the Hashtable implementation is actually exposing a generic KeyValuePair now)

So, I added a small "CompatibilityShims" Silverlight C# library project to the solution that included Hashtable, ArrayList, and SerializableAttribute.

Wiring up test applications

Unlike standard NUnit test projects, you will need to create an actual Silverlight application project for your tests. Once you create a new empty app, here’s what you should do:

• Remove the Page.xaml and Page.xaml.cs files
• Add a reference to Microsoft.Silverlight.Testing.dll
• Add a reference to the built NUnitFramework and NUnitSilverlight libraries and/or projects from this download
• Inside App.xaml.cs, add using statements for the Microsoft.Silverlight.Testing and Microsoft.Silverlight.Testing.UnitTesting.Metadata.NUnit namespaces
• Update Application_Startup per the instructions below
• Add your tests and, if needed, a reference to your assembly/app under test

Here’s what the startup code should look like for the test application:

        private void Application_Startup(object sender, StartupEventArgs e)
        {
            UnitTestSystem.RegisterUnitTestProvider(new NUnitProvider());
            RootVisual = UnitTestSystem.CreateTestPage();
        }

The code is similar to the regular Silverlight unit test startup code that you use for VSTT tests, except it first registers the NUnitProvider.

You can build and run the Silverlight application like any other: just press F5 and the tests will run!

Using the Silverlight test framework’s asynchronous and Silverlight testing features

Since the unit test engine is metadata-agnostic, it handles all the advanced asynchronous and Silverlight-specific test functions. Many of the core features specific to the Silverlight framework work with NUnit, therefore.

You’ll find that you can use:

• [Bug(...)] attributes
• [Tag(...)] attributes. The Category NUnit attribute isn’t supported in this download, but you could easily modify the provider to expose Category data as Tags.
• [Asynchronous], combined with deriving from SilverlightTest: support functional testing of user interface elements and controls.

Download the bits and samples

I’m offering this as a proof-of-concept, unsupported download. Inside you’ll find everything that you need to explore the concept.

• NUnit.Silverlight.zip [266 KB], contains:

• Unit test framework binaries
• CompatibilityShims
• NUnitFramework
• "NUnitSilverlight" NUnit test provider for the Silverlight unit test framework
• Sample test applications: "Money", "NUnitTests"

Related resources

• Visual Studio unit testing intro
• Silverlight unit test framework source released
• Reference documentation for the Silverlight unit test framework
• New test framework features for RC0/RTW
• Unit testing with Silverlight 2

Hope this helps!

The original post about this technology is here.

Download it today!

• Binaries and control sample tests, part of the Silverlight Toolkit download
  (you’ll find the bits in the .\Source\Binaries\ directory)
• Unit test framework binaries only
• Source code

This interim release corrected a few simple issues that customers have reported, and is included along with the December 2008 release of the Silverlight Toolkit. Changes of note:

• Asynchronous exception handling fixes and improvements
• [Priority(...)] attribute from VSTT is now supported. They receive implicit Tag values of PriorityN, where N is the integer priority.
• WorkItemTest type: EnqueueDelay method added, has two method signatures that accept either the milliseconds to delay, or a TimeSpan object.
• EnqueueCallback has a new overload that takes a number of Action parameters, for easy action chaining
• Enqueue has been marked as obsolete: please use EnqueueCallback, for consistency
• EnqueueSleep has been marked as obsolete: please use EnqueueDelay, which is a more clear name
• Silverlight test service fixes, you can now run test applications from a web server in more scenarios
• Silverlight test service can co-exist with other web servers and hosts

Hope this helps.

Maurice de Beijer has blogged about his solution for creating a testable Flickr web service API. He’s created TestableWebClient and TestableOpenReadCompletedEventArgs types to assist with this, and kept the abstraction simple and to-the-point.

I agree with Maurice’s comments: it isn’t perfect, it isn’t awful; it isn’t pretty; but hey, it does get the job done, and he’s found a good solution for this today.

Maurice also previously posted his own introduction guide to Silverlight unit testing.

Just wanted to pass along, definitely check this out!