Silverlight Unit Test Framework: New version in the April 2010 Silverlight Toolkit
In April we released a new version of the Silverlight Toolkit for targeting Microsoft Silverlight 4. This release contains a new version of the Silverlight Unit Test Framework. Key improvements come in user interface, Silverlight 4 support, and performance.
Rich, modern user interface
The user interface no longer uses the antiquated HTML DOM bridge for displaying results, but now is a data bound, rich Silverlight application itself.
Rich results information is available, and a tree view control allows you to select results to be included in a test report that can be copied to the clipboard. Failures are automatically selected.
The results pane includes information about the tests that ran, as well as detailed information for failures.
Descriptive text helps identify negative tests, known test issues and bugs, and other important metadata that was not always exposed in the previous test framework.
Integrated tag expression support
The startup experience now allows you to enter a tag expression to select which tests to run, or not to run. You can add the Tag(“”) attribute to any test methods or classes to define your own tags. Also, the full and short names of tests and classes implicitly become test tags, as well as priorities.
This means that you can define tags such as:
| !MyTest1 | Run all tests except for the method named MyTest1 |
| MyTest1+MyTest2 | Run the MyTest1 and MyTest2 test methods |
Out of Browser support
The framework now supports running out-of-browser. Paired with the new Silverlight 4 Tools for Visual Studio 2010, you can press F5 to run a test project you have marked as an out-of-browser.
This also opens up an opportunity to do testing of elevated apps that make use of AutomationFactory and other advanced features on Windows.
Simplified dependencies
The test framework removed dependencies on System.Windows.Browser (from the platform) and System.Xml.Linq (XLinq, from the SDK).
Building blocks for Windows Phone testing support
This version also contains the full source to the phone testing interface, which is significantly different in visual design and experience to coincide with the Windows Phone device size.
However, the story for phone testing is still developing: in the meantime binaries designed for phone use can be found at http://jeffatmix.com/
New location for binaries
Since the new rich user interface contains a number of controls, the traditional design-time integration system conflicts and presents some issues.
As a result, we had to move the binaries from the Bin directory of the toolkit installation folder into a new Testing directory.
So the two test framework assemblies (Microsoft.Silverlight.Testing and Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight) have moved.
This location change could cause existing applications that reference the test assemblies to fail to compile due to the "missing" references.
If that happens, please delete the two broken references and create references to these assemblies in their new location. Everything else should continue to work as expected.
Simplified test project template
In alignment with the test project changes in Visual Studio 2010, which are more simplified, the unit test project template for Silverlight no longer includes a README.TXT file.
Known issues
Test service
I’ve received reports from a number of people about issues with the test service automation support. Some of the NDA/preview users of the framework reported issues that were corrected, but there may still be some bugs out there. I’m working to investigate and post a fix if and when I know more.
In the meantime I know that StatLight, a third party system, has been updated for the April release and is working well for automation.
Test panel and visuals
Please beware that some of your tests may need to be updated to use the new framework if your application, utility functions, or tests ever used the Application.Current.RootVisual (against the recommended pattern for interface testing).
The root visual is now the test framework application runner itself, and the tests run within the same TestPanel as before. Controls added to the test panel will not be located at the 0, 0 screen position, but likely near the bottom of the screen instead.
Hope this helps. To get started, download and install the April toolkit!
I like it much more. And it seems to run faster. Very nice.
Are there any guides for setting up TFS 2010 with Silverlight Unit Testing?
@Anders,
.
There’s not really a clean story for integrating TFS and Silverlight unit tests today, unfortunately
Will this toolkit still work with Silverlight 3?
[...] Silverlight Unit Test Framework: New version in the April 2010 Silverlight Toolkit (Jeff Wilcox) [...]
Can you clarify what the possible issues are with SLUT and integration into TFS2010? Thanks.
@Ian,
Some bugs in the web service reporting code causes the integration to fail – the browser never closes and results are not reported.
Can I debug somehow each test separately in Visual Studio 2010?
When I use NUnit, I do right click on [Test] tag and select Test with -> Debugger, and I can debug this appropriate test. This was a usual test class, created in a usual project (not Silverlight).
Now, I create unit test project using Silverlight Unit Test Framework, and I click on [TestMethod] tag, select Test with -> Debugger, and get error like this:
Could not load file or assembly ‘System.Xml, Version=2.0.5.0, Culture=neutral, bla-bla …
What can you suggest to debug only one test?
[...] Silverlight Unit Test Framework: New version in the April 2010 Silverlight Toolkit [...]
Hi Jeff
Any ideas yet on when the problem with closing the browser after running the unit tests with MSBuild will be fixed. We use cruise control to automate our build process and the unit tests are quite an important part of the process.
cheers
@Mark,
I have checked in some fixes internally and am trying to find the best way to share the improved experience. Will blog it when there is time.
@Tanya,
You are using the Visual Studio test features; there is no integration between those and the Silverlight unit test framework.
However when you use the latest (April) version of the framework, at startup you can select a specific test(s) to run using the tag expressions feature: http://www.jeff.wilcox.name/2010/05/new-2010-test-framework/
[...] Silverlight Unit Test Framework updated Microsoft Engineer Jeff Wilcox has the details about the newest version of the Silverlight Toolkit’s improved Silverlight Unit Test Framework. [...]
Was the Silverlight Unit Test Application project template removed from the 5/13/2010 Microsoft Silverlight 4 Tools for Visual Studio 2010? Why? Where can we find a template that is compatible with the latest release?
TIA.
Can we run unit tests automatically with MStest in the MSBuild with latest Silverlight Unit Test Framework in Silverlight Toolkit 2010? thanks
Great job. Thanks a lot~
If you use statlight with the latest version and a continuous integration service you have to use the system process and allow it to interact with the desktop.
Is there any way to disable the startup-experience (with the tag-question). It just slows down the tdd-process which is bad… btw tdd with silverlight isn’t fun at all
Where are the test results saved now? The previous version of the framework saved them to the project’s debug folder but I can’t find the the test results with the new framework.
@Crash,
For phone testing, there isn’t storage of the results.
For desktop, if everything aligns, then it is possible that it is stored in the current directory or the debug folder. There are some issues with this working consistently.
There’s also a tool called StatLight that might be of help: http://statlight.net/
@Jeff
I checked both places and there seem to be no saved test results at all. Searched my drive for .trx files also to no avail. I’ve been testing using Statlight but cannot currently get it to work properly either
I am planning to use this framework for unit test automation on Windows Phone 7. Can you please let me know where the logs are getting generated? Can we have the results in TRX format?
@Ashutosh,
There’s no built-in log saving for the phone, though you could connect to various events on the UnitTestHarness type to generate your own, save it to isolated storage, send to a web service, etc.
Thanks Jeff. But do you have any sample code which can collect the logs/results and then convert it into the TRX?
@Ashutosh,
Nope!