Walkthrough: The power of the November 2009 Silverlight Toolkit testing tools
The November 2009 Silverlight Toolkit is essentially a tools and infrastructure release on top of the October 2009 release (where we first introduced Visual Studio 2010 support). It also adds a Silverlight 4 Beta version.
New infrastructure & test tools ship in the Silverlight Toolkit
There is a lot in the release that is joining publicly for the first time, based on some of the internal tools and utilities that we use in building and testing the Silverlight Toolkit, plus things we’d like to have as typical Silverlight developers.
These tools join in the Experimental quality band, and over the next few posts, I’ll dig into the details. This post is a literal walkthrough of how you could go about seeing all the utilities in the meantime.
One important thing to call out is that, as an initial release, there are definitely some rough edges: the test tools are centered around Microsoft Build (msbuild) integration, instead of Visual Studio integration; there’s no add-in or nice right-click project support to use these tools today.
We’ll be collecting feedback along the way and making changes in future releases. Until then, hopefully some of you will find the infrastructure useful, if only to peak behind the curtain. We’re also shipping the full source to these tools, you’ll find an Infrastructure.zip file inside the toolkit install folder.
What we’ll do in this walkthrough
This walkthrough uses Silverlight 4, though the instructions are virtually identical if you’re using Silverlight 3.
- Install the Silverlight Toolkit
- Create a new Silverlight class library with a simple business object
- Add a Silverlight Unit Test Application using the new templates found in the Silverlight Toolkit
- Run the tests in Visual Studio
- Run the tests from the command line in various browsers
- Collect block-level code coverage information for the build
This is screen-shot heavy, as future posts will dig into the details. I appreciate your patience!
Requirements:
- Latest Silverlight Toolkit
- For test automation:
- MSBuild on the machine
- Visual Studio 2010 Beta 2 Ultimate
Installing the tools
When you install either Silverlight Toolkit drop from November, you’ll see that there’s a new feature listed in the setup: the Tools & Templates feature. It’s selected by default, and adds Visual Studio templates, installs the tools, and prepares the tools if your machine has the proper dependencies on it.
Create a Silverlight Class Library
Open Visual Studio 2010 Beta 2, after installing the toolkit. Create a new Silverlight Class Library project, we’ll store a simple business object in it.
Whichever flavor of Silverlight you use, remember to use the same flavor while creating future projects in the same solution!
Create a new Person.cs type, with a few properties and a method. Here’s the source:
using System;
namespace MyApplication
{
public class Person
{
public string First { get; set; }
public string Last { get; set; }
public string FullName { get { return First + " " + Last; } }
public bool MightBe(string substring)
{
return FullName.Contains(substring);
}
}
}
Add a Silverlight Unit Test Application
Right-click on the solution in the Solution Explorer and select Add New Project. Under the Visual C# (and also Visual Basic) languages node, select the Silverlight subgroup. You’ll see a Silverlight Unit Test Application project template.
You now have two projects:
Next up, we want to add a reference to the class library, so that the unit tests can access the library. Right-click on the new test project and then select the Add Reference option. Under the Project tab, choose the class library you created earlier.
Now, let’s add some unit tests. Clear out the content in the created Tests.cs file and drop this in:
using Microsoft.VisualStudio.TestTools.UnitTesting;
using MyApplication;
namespace PersonTestProject {
[TestClass]
public class Tests
{
[TestMethod]
public void TestCtor()
{
Person p = new Person();
}
[TestMethod]
public void TestSetName()
{
Person p = new Person { First = "Scott", Last = "Guthrie" };
Assert.AreNotEqual(p.First, p.Last);
}
[TestMethod]
public void TestGetFullName()
{
Person p = new Person { First = "Steve", Last = "Ballmer" };
StringAssert.Equals("Steve Ballmer", p.FullName);
}
}
}
The tests are pretty simple, and exercise various simple PMEs on the type.
Run the unit tests from Visual Studio
Right-click on the test project and select Set as StartUp Project from the menu. Then, press F5 or click the Run button to begin execution.
The default web browser will open up, the tests will run, and the window will stay open. When you are satisfied, close the browser. You’ve just run some simple tests!
Running unit tests from the Visual Studio command line
More interesting is being able to automate the tests: through MSBuild, we’ve added a task that can control the browser and save out the log file(s) from the unit tests when they run.
To do this, you need a few things:
- The Visual Studio command prompt open.
- The full path to the test project.
Open the command prompt
Personally, I always pin the Visual Studio 2008 tools command prompt next to my Visual Studio taskbar icons.
You can start it from the Start Menu, under the tools for the Visual Studio version you are using.
Alternatively, make sure MSBuild is in your path.
Or, start a command prompt and then move to your 32-bit Program Files, then ‘Visual Studio 9.0’, then VC. Run ‘vcvarsall.bat x86’
Note: all these command prompts should be 32-bit, even on a x64 machine. Silverlight is a 32-bit world today.
Move to the test project
To get the full path to the test project, I just right-click on the test project in Visual Studio, then Open Folder in Windows Explorer. I copy the path from the resulting Explorer dialog’s crumb navigation bar.
Now, inside the command prompt window, move to that path:
pushd (PASTE PATH HERE) <enter>
To run the unit tests in the default browser, simply type
msbuild /t:test
and press Enter.
You’ll see the browser open, quickly run some tests, and then close. Here’s what it looks like, captured in the middle of the 5-second test run:
And afterwards, you’ll see that Msbuild reports success, and you can read the test to see that 3 passing tests were reported, out of 3 total tests.
Running tests in Google Chrome
It’s easy! Just set the browser property to Chrome.
msbuild /t:test /p:browser=chrome
And just like that, Chrome opens up and runs the tests.
Running tests in Mozilla Firefox
Just set the browser to Firefox.
msbuild /t:test /p:browser=firefox
Tag Expressions
A nice feature to help select a subset of tests is the Tag Expression syntax. By specifying a tag expression at the command prompt, you can include and exclude tests that are marked with the Tag attribute found in Microsoft.Silverlight.Testing.
Also, tags implicitly exist for all test method names, short and full.
So, let’s run the test called TestSetName from our project.
msbuild /t:test /p:tagexpression=TestSetName
You may briefly see indication in the test UI in the browser that a ‘Tag expression’ is in use.
And the build results show that just one test ran:
If you want to run all tests except that one, use this tag expression:
msbuild /t:test /p:tagexpression=!TestSetName
Cool, and easy!
Test results files
Another nice feature of running the tests through MSBuild is that you’ll see test result files (end in *.trx) inside the same folder as the test page for the application.
Similar and conformant to the Visual Studio *.trx format, you can parse and work with this data to understand execution times, results, and read other information. Unfortunately, you cannot open these files in Visual Studio 2010 Beta 2, but they do open in Visual Studio 2008. Note that there isn’t any real test integration with VS here: this is just an informative display.
Here’s a directory with several test result files (plus some coverage stuff we’ll get to later):
You can open up the TestResults trx file and see what kind of information is in it:
And here’s the results file opened in Visual Studio:
Preparing for Code Coverage
Ripe for a Visual Studio add-in, this process requires us modifying the test project’s .csproj file some. Instructions are the same for Visual Studio.
We need to manually select the assembly to be instrumented by setting a property. The assembly to be instrumented must not be signed, and not have a strong name. In this example, the assembly is SilverlightClassLibrary3.
To make changes to the test project, to specify this, first unload the project by right-clicking on the test project and selecting the Unload option:
Then, right-click the project file that is ‘(unavailable) and select ‘Edit’.
Now you’ll see the XML data for the project. Scroll down in the template to where there is a commented out ItemGroup, and some comments about the code coverage support. Remove the comments from the ItemGroup:
And now, change the Include statement from SilverlightClassLibrary1 to whatever your assembly to instrument is called. In our example here, it is called SilverlightClassLibrary3.
Now, that’s it. Let’s close the file and then right-click the project again and select Reload Project:
Now we’re ready to go!
Running Silverlight Code Coverage
Just like the msbuild /t:test from above, we have a similar target named CoverageTest and CoverageView. CoverageTest collects the data, while CoverageView collects the coverage data and then shows the results in an application that explores the hit and not hit portions of your source code.
Let’s collect coverage!
Note: This is assuming that you have a high-level SKU of Visual Studio 2010 Beta 2 installed. The coverage tools depend on the Static Analysis Tools that ship in the beta. You will receive an error if you run this on a machine without those tools. We’ll look for a better experience in the future.
Remember, the same properties from above – browser and tag expression – still can apply and work in the coverage targets.
msbuild /t:coveragetest
Here is what it will look like if everything line up. Unfortunately, the coverage experience is a very early preview and very flaky in some situations due to the dependency on beta components. We also haven’t had I admit enough time to iron out all the issues customers may experience.
You see the Instrumentation messages come out, then the RunTests target, then the coverage data is merged and things should be successful.
If that worked, it is safe to assume that the view target will also work. It will re-run all the tests and instrument again as well.
msbuild /t:coverageview
Now you’ll see the viewer popup, that lets you drill into types and methods to see what is hit (cyan) or not hit (red), to spot code that your tests are missing:
You’ll find a Coverage.xml file in the same directory where the test results go. This is the data that is used by the viewer application. You’ll see in the XML file a set of ‘visited’ blocks, and other information, in the file:
More to come soon!
Download the Silverlight Toolkit November 2009 Release Today
![Toolkit32[2][2][2]](http://www.jeff.wilcox.name/wp-content/uploads/2009/11/Toolkit32222.png "Toolkit32[2][2][2]")
The Silverlight Toolkit is a collection of Silverlight controls, components and utilities that help make Silverlight development a little easier, more fun, and add value outside the regular Silverlight release cycle.
The sixth release of the Silverlight Toolkit, the November 2009 release targets Silverlight 3. There is also a release available that targets the new Silverlight 4 Beta for developers.
Resources of note:
- Learn more about the Silverlight Toolkit
- Download the November 2009 Release
- Get started with Silverlight development
- Participate in the controls community at Silverlight.net
- Check out questions and answers on Stack Overflow using the ‘silverlight-toolkit’ tag
- The Issue Tracker can be used to report issues, add to the feature wish list, and vote on the work to be done by the Microsoft team
Hope you enjoy our new release!
Am I reading this correct, Silverlight now can do automated testing as part of the build process!? That is huge news for us, thanks for this blog entry. Is it safe to assume you can accomplish this with SL3 with the latest toolkit as well?
On Windows XP with VS 2010 Beta 1, I can’t get “msbuild /t:test” to execute any tests. The error seems to point to the fact that the script tries to run Internet Explorer in protected mode that is not available on XP:
Task “RunUnitTestsTask” (TaskId:5)
Protected Mode requires Windows Vista, Windows 7, or later.
Using “/p:browser=firefox” fixes this particular problem
.
Yeah, sorry about that. Hooking up an XP IE is pending.
Hi Jeff,
I’ve successfully been able to integrate the unit test and code coverage during a TFS 2010 Build of a Silverlight 4 project.
But, the coverage.xml file is missing the attribute pathprefix and thus, the application CodeCoverageViewer.exe crash with a beautiful not handled exception
If i set pathprefix manually (like pathprefix=”Test”), it works fine.
Is it a know bug ?
Thanks !
Hi Jeff
I tried using your instructions to run the tests from the vs cmd line but the build fails with a “The target ‘test’ does not exist in the project”.
Is that new for this release of the Toolkit? In that case, how do i upgrade an existing project ?
My main problem, for which you might have a solution, is being able to constrain the tests I am running at the moment. We have a test base of 400+ test (and going bigger) and it’s a pain having to run all those every time. I see you solve that problem by using the Tag attribute, but then, msbuild doesn’t seem to like the test build target….
What can I do?
Thanks in advance
Thanks. One suggestion:
If I run this from cmd line to execute a quick test, I really don’t care much for IE to open and show some GUI like that.
I’d prefer actually to just see the test pass/fail and reason spit out to the console with the option to even write it to log as well.
Thanks!
@Steve,
Right now the design is just like that of the .NET Compact Framework’s test environment: you need to have the UI (or emulator) appear, since most people are testing components very often that require the context of the platform.
If you are only interested in testing business logic (no UI), then you can always cross-compile your components within a desktop class library project, and use the full Visual Studio Team Test features, to get red/green status of your tests.
Any way to upgrade existing test projects?
@Carlzbad,
Yes, it really just requires adding a few lines with the proper Import statements to the .csproj file. I’ll find some time to blog about this. For now you should be able to see the custom Import statement from a simple empty test project created with the template and copy it.
For code coverage I installed vs2010 beta 2.
I get an error message:
The Static Analysis dependencies for instrumentation could not be located
I look in the targets file and it is looking for this dll in this place:
C:\Program Files\Microsoft SDKs\Silverlight\v3.0\Toolkit\Nov09\Tools\phx.dll
but there is no way that vs2010 would install the dll in this place…
what am I doing wrong here?
@Carlzbad,
Your machine did not have the appropriate static analysis tools. During installation, they are silently copied from the appropriate place. This is because of the requirement for a high-end Visual Studio SKU with those files.
If I understand correctly, msbuild /t:test does not work when using VS 2008 solutions. Is that correct and are there workarounds?
How do you switch the web browser in that environment?
@Bernard,
/t:test -does- work with VS2008. Just not /t:coveragetest.
Thanks Jeff, it was a problem between keyboard and chair
As you mention in your article, one needs to run this from the Test project directory – not from the solution directory as I did incorrectly.
“Silverlight Unit Test Application project template” doesn’t appear in the “Add New Project” dialog.
Should I do the work of what addition?
Or, does not the Toolkit work well with Visual Studio 2010 beta 2 Japanese version?
I did the following:
1. I installed Visual Studio 2010 beta 2 (Japanese) on Windows 7.
2. Then, I installed Silverlight 4 Tools for Visual Studio 2010 Beta 2 (Japanese).
3. Last, I installed Silverlight 4 Beta Toolkit November 2009.
Does the following become a trouble?
- My Windows 7 system is in the D: drive.
- I installed VS2010beta2(J) in C.
- When I installed Silverlight Toolkit, it was set up in D.
It took me a while to figure out that this does not work with IE7 installed (it worked on my machine but not on the build server *grr*) … after installing IE8 it worked there too…
But I cannot make the msbuild task fail with an error if there was a failing unit test. I have to check the build result with a powershell script…. and yes, I used /p:TreatTestFailuresAsErrors=true
@Falk Dobermann,
Thanks for investigating those issues. I’ll see what I can do to get issues files on these for the next release. Down-level (XP) and pre-IE8 browsers should be supported, sorry that didn’t make this first attempt.
@biac,
Unfortunately the requirement for VS2010 Beta 2 right now leaves a lot of configurations possibly out in the cold. We’re looking to improve the requirements for the setup in a future release.
To Mr. Jeff
I reinstalled all (VS2010b2(J), Tools(J) and Toolkit) to the same drive (D:) where Windows System existed. Still, the template did not appear.
> requirement for VS2010 Beta 2 right now leaves a lot of configurations possibly out in the cold.
I see.
I’ll try to install the templates manually.
Thanks.
@biac,
The VS2010 requirement is really only for the code coverage tools.
All the templates should work on supported Silverlight Toolkit configurations.
However, the templates are installed in the English (1033) templates location on the machine. Since we do not offer a JPN install SKU for the Toolkit, the best work around would to be use a VWD Express (ENU) or VS ENU 2008 or 2010 version of Visual Studio in the meantime.
Thank you, Jeff
I think that I got the hoped answer (the following two points).
1. Present Toolkit doesn’t support Japanese version VS2010beta2.
2. It’s not trouble that the Project Template doesn’t appear.
I’ll challenge to use the template at my own risk.
Once again, thank you.
Hi Jeff,
I’m using the Nov 09 SL toolkit, and have created my own UT, based on the control’s source code in that package.
I’m interested in checking the code-coverage, however I get the Static Analysis error you’ve mentioned.
How can I install the requried static analysis tools?
Do I need to check the Code-Analysis “Enable Code-Analysis on Build” on the code project’s properties (currently it is unchecked, to allow successful compilation).
Thanks!
Jeff, do you have a solution for this?
When running unit tests with F5 / Start Debugging, the tests break for any Exception.
Workaround: run Start without Debugging
Problem: what if you only want the debugger to break on your breakpoint, not on any Exception thrown within tests?
With classical .NET (non-Silverlight) projects, you would use Debug > Exceptions to configure this behavior.
[TestClass]
public class MyViewModelTest
{
[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
public void Constructor_Null_RaisesArgumentNullException()
{
// VS should not break here, but should stop on breakpoints
new MyViewModel(null);
}
}
Hi Jeff,
I thought you would be able to provide some guideline on how to measure performance of API developed in Silverlight ?
I would like to know
1.) Performance testing framework ( if any) available for Libraries developed in Silverlight ?
2.) What’s the architecture of such tests look like ?
3.) Any helpful articles / links
thanks.
Hi Jeff
We are getting some issues when running tests headless. If there is a Silverlight exception of the kind which results in the plugin not loading (comes with the corresponding javascipt error), the build script hangs.
I thought there may be a timeout for RunUnitTestsTask that we could use to kill the browser, but the one that’s there (RunUnitTestsTask.Timeout) doesnt appear to be being used. I don’t know how a timeout would work anyway, u wouldn’t know what test were broken as they would never run if a Timeout caused the browser to be killed. I guess you could just return false from the RunUnitTestsTask if the timeout was reached thus killing the build.
Anyway, cut a long story short: Anyone come up against hanging build scripts due to plugin load failure and have any suggestions?
Great blog, keep up the good work.
Keith
[...] Jeff Wilcox posted a large walkthrough to demonstrate how to use the Silverlight unit testing tools. [...]
The tagexpression parameter doesn’t work on my SL3/VS2008 project. Is this only supported for 2010? When I run my tests from the command line it shows warning “There were 0 reported scenarios executed. Check that the tag expression is appropriate.” I’ve tried adding the Tag attribute on both the test class and individual test methods but it never gets recognized.
Also the .trx file is never output. The warning says “TestResults.trx was never written by the test service” but doesn’t give a reason why it wasn’t written.
Hey Jeff. I am upgrading an existing test project. I copied over the Import statements from a simple empty test project created with the new template in the project file and I get this error “The target “test” does not exist in the project.”. Roman from a post above had the same issue. I am using the VS 2008 command prompt with the current location being my test project. Any idea on why this is happening?
Hi Jeff,
Thanks a lot for the post. I’m unable to execute the tests using msbuild on XP running VS2008. It appears to be the same issue that Roman encountered:
error MSB4057: The target “test” does not exist in the project
I tried to add the target to the project file manually, but could not get it to work (not sure if I did this right). Could you please post the source for your solution or at least the project file of the test project?
Thanks
@Christian,
If you’ve installed the Toolkit, you should find the template .Zip in your system. You can extract it to find the barebones .csproj, though it’ll have a few Visual Studio template replacement markers in it.
Look in “C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\ProjectTemplates\CSharp\Silverlight\1033\” or similar paths.
@Christian,
Also, you may be msbuild’ing the .sln instead of the .csproj: try msbuild /t:test yourProjectFile.csproj, instead of just calling msbuild – which will favor the solution file I believe.
@Jay,
Try msbuild /t:test yourProject.csproj and let me know how that goes… the “upgrade” scenario from the previous test template projects I shipped isn’t perfect unfortunately…
@Keith,
“Headless”: does this mean there is no UI session at all? Or that the test is running on a VM/TS session that isn’t necessarily able to receive OS focus messages?
The plugin failing to load is a rare one indeed – but one thing you can try and do in case this is just an issue with unhandled exceptions bubbling up to the browser instead of the test framework, you could modify your app’s unhandled exception handler goo inside of App.xaml.cs to no longer include the ReportErrorToDOM() method.
One major issue with the Silverlight unit test framework is that it is a Silverlight application in and of itself, effectively self hosted – so if anything majorly wrong happens (wild browser crash, plugin AV, wild CLR bug, etc.) then you’ll probably not get a chance to respond to the problem and get results. So I dunno what the right answer is.
On the timeouts, I don’t have a great answer. At least on our team, we have tens of thousands of test methods in some test assemblies… so it’s kind of a waste to have a forced timeout of say 30 minutes – because it’s possible on a very slow machine, or VM, that 30 minutes would not even be enough for us… but I will admit, timeout behavior in the past was implemented and has probably regressed in the shipped build. Not something we rely on much.
-Jeff
Hey Jeff, I tried what you suggested and it didn’t work. I get the same error. Can you think of anything else I can try?
Jeff,
Thanks for the help. I finally got the test project to run using Firefox. This is a great framework!
Unfortunately, our build environment runs on XP and is restricted to Internet Explorer only. If I can’t convince our admins to install Firefox, I’ll have to wait for a future release that addresses the XP/IE issue.
Thanks.
Jeff, I finally got the test to work. I had to remove my old test project, create a new one with the new templates, and then transfer all the tests over. One question though, how can I make the build fail when I get a failed test? The failed message came out as a warning and did not make the build fail.
@Angela,
It seems like it’s a browser issue. I tried with chrome and I got the same error you did. Firefox is the only one that seems to be working.
@Angela,
Without the Tag attributes, can you still use tag expressions?
All test names are implicit tag expressions, so if you have a test method called “MyTest1″ you should be able to run with the parameter tagexpression=MyTest1.
I see that only certain information about the test is logged in the trx file.
I get the ‘TestMethod’ information which includes the name (among other).
If I use the ‘Description’ attribute on a test case, then I also get the in the test details in the trx.
However if I use the ‘Priority’ attribute on the test case, then this is not logged in the trx file.
1. Is there any documentation on which attributes are logged in the trx file?
2. Is it possible to customize what is logged?
2. I would like to create my own attribute type, how can I have this logged in the trx?
The reason for asking is that I’d like to do some custom parsing of the log files, and being able to group the tests based on certain criteria would be useful.
Thanks,
How do you integrate the unit test with a tfsbuild project? I would like for our daily build to fail if a test fails and haven’t found anything on how to do this with this new framework.
Jeff, thanks for the great post. I am prepping a vm for some Silvlerlight 4 development research. I have a fresh install of VS2010 Pro Beta 2, the Silverlight 4 Tools for VS2010 Beta 2, and the November 2009 release of the tookit for Silverlight 4 Beta. All seemed to install fine.
When I create a Silverlight 4 Application project (with website) in VS2010 however, I don’t see the Silverlight Tookit controls (like the charts, etc) in the toolbox. The regular Silverlight controls are there, but not the tookit controls.
Im sure I’m being brain dead, but can you point me in the right direction to get those toolkit controls to show up in the toolbox in VS2010? What about in the Expression Blend 4 Preview?
Thanks!
Jeff, I kind of answered my own question by getting the controls added to the toolbox manually, I guess I just though the install would have dropped them in there.
I learned a lot about the controls locations while I looked into it though, and made my own blog post about all of the Silverlight 4 Beta controls, their Namespaces, Assemblies, and Assembly paths at:
http://blogs.netconnex.com/2010/02/silverlight-4-beta-control-namespaces.html
Others may find the info there helpful.
Thanks!
For everyone asking on how to make the MSBuild task fail when a test fails: when I was looking at this, I noticed that ContinueOnError is hardcoded as “true” in the RunTests task.
To fix this, I edited the Microsoft.Silverlight.Toolkit.Build.targets file that came with the toolkit release, and in the target “RunTests”, for the RunUnitTestsTask action, I changed ContinueOnError=”true” to ContinueOnError=”$(ContinueOnError)”. Then at the commandline I could pass /p:TreatTestFailuresAsErrors=true;ContinueOnError=false to make the task fail on failed tests.
Update on my previous comment: I’ve discovered that on my XP setup the tag expression works only with Firefox. On my Win 7 setup it works with IE and Chrome, but not Firefox. Strange.
I used non-silverlight class libraries with the POCO templates for my silverlight application. Because they weren’t from a Silverlight class library it won’t let me add these projects as references to my silverlight test project. Any ideas on what I should do? What’s different about a Silverlight Class Lbirary from a normal class library?