.Net Mocking Frameworks comparison


As I mentioned in the last post, that I have been working on Agile methodologies for more than 18 months now. During this time I been involved in project development using TDD. As part of writing and executing automated Unit Tests, our team has been using Dynamic Mocking Frameworks like NMock 2 and Rhino Mocks. Here is an attempt to compare different mocking frameworks I have worked with.

Different Dynamic Mocks

In simple terms mock is an object which acts as a proxy for some real world object. Its a dummy object which is used to mimic the real world object. Every time I try to explain some person who is new to TDD the usage of mocks the first question that pops out is why do we need mock objects?

1. One of the basic need for having mock object is to follow the unit testing principle which says that we should be testing the System Under Test (SUT) in isolation. If we follow this principle only the logic related to one class or method should be tested using a unit test and we shouldn't be really worried about the dependencies of that class which we are testing. All the dependencies should be tested separately. In order to manage the dependencies we use mock objects. We can replace the dependencies with fake objects or mocks.

2. Another reason why we use mock objects is to improve the performance of executing the unit tests. If we were to construct the real objects or dependencies it might be time consuming.

3. This also helps us in removing dependencies like external web services or the physical database server. We can mimic these behaviours using a mock object as long as the mocks use the same interface.

4. Another alternative to using the dynamic mock object is to create a fake object which are also called as static mocks. But the problem I see with this approach is that every time there is a change to the service contract or the interface which the mock implements, the hard coded mock need to be updated as well. This results in maintenance problem.

Initially my team was using NMock 2 for mocking. It has got some nice features like it provides a very fluent interface for setting expectations as well as for different types of matchers. The problem I feel with NMock 2 is that it uses magic strings for the names of methods and properties. This does not provide much type safety. In terms of Rhino Mocks matchers are referred as argument constraints.

Type safety is the biggest advantage Rhino Mocks has over NMock 2. Apart from type safety Rhino Mocks also has a Record and Playback option. It can also mock concrete as well as abstract classes. Rhino Mocks has 3 types of mocks Strict Mock, Partial Mock and Dynamic Mock. It also has the facility to set expectations in ordered and unordered sequence.

As compared to these two commonly used Dynamic mocking frameworks, Moq is relatively new. It has the distinction of being the only library for .Net which is developed from scratch using the features of C# 3.0 and 3.5 framework. It relies heavily on LINQ and Lambda expressions. So knowledge and experience of these C# features is a must if someone wants to try Moq as the preferred framework.

Instead of creating something from the scratch I decided to use the hypothetical sample which was published on NMock site as a quickstart. You can read more about it from http://www.nmock.org/quickstart.html. I have made minor modifications to the test code to follow the AAA syntax. I have attached the solution code at the end of the article. Instead of going into the details of how to create a mock repository and explain each and every method, I would like to highlight the main differences in the way expectations are set on the mock instance in these 3 frameworks.




                With("GBP", "CAD").


Rhino Mocks

Expect.Call(_mockCurrencyService.GetConversionRate(britishAccount.Currency, canadianAccount.Currency)).




                mockCurrencyService =>

                mockCurrencyService.GetConversionRate(britishAccount.Currency, canadianAccount.Currency)



As we can see from the above statements, Rhino Mocks and Moq have lot of similarity in the way the expectations are set on the mock currency service. But in case of NMock2 we need to specify explicitly what is the method name, the parameters and also the return type. In one way the syntax is quite intuitive, but lacks type safety. Also I feel that having to type something like


is bit too much just to set a return value.


In some future post I would like to highlight how to verify constraints and also do argument matching using matchers. But for the time being I feel Rhino Mocks has advantages over NMock 2. I haven't worked much with Moq. So I am really not in a position to comment which one is good among Rhino Mocks and Moq. Both might be used in different contexts. If a completely new development with .Net 3.5 framework is to be started Moq seems to be a good option.

This article wouldn't be complete without mentioning the difference between a mock and a stub. This is well documented by Martin Fowler in his post Mocks Aren't Stubs. At a very high level I would say you would use mocks to test interactions between different layers or classes of your applications. Stubs are used to stub some behaviour. Using a stub will not fail the unit tests. But mocks can cause tests to fail if all the interactions are not performed as per the expectations.

You can download the complete source code from this location. If you are facing some problem downloading the code from the link, you could also try pasting the following URL in your browser

http://dl.getdropbox.com/u/1569964/MockingComparison.zip alt="http://dl.getdropbox.com/u/1569964/MockingComparison.zip"



Comparing Unit Testing Framework


I have been doing Unit Testing for past 18 months or so and have been ripping benefits of the same. I would suggest every developer should adapt unit testing methods if they have not done so far. Because we follow Agile in our development activities its mandatory for us to unit test our code. But even otherwise I feel its a disciplined effort to test your code. I have personally used NUnit as well as MS Test frameworks for unit testing and would like to highlight some differences here in this post.

Unit Testing Framework comparison

Recently I conducted a training session for the people in my organization who were interested in implementing it in their future projects. One of the feedback I had received during my previous training program was that I should have used NUnit for demo instead of MS Test because people felt that unit testing without NUnit was a crime. Just kidding. It was just that NUnit has become so synonymous with unit testing that people think its not worth attending a training if you are not going to demo NUnit. 

I personally don't find much difference between NUnit and MS test. Both use similar concepts with the exception of attributes. Following are some of the attributes which I have used during unit tests written in those frameworks

MS Test

Many purists feel that Microsoft has copied the NUnit framework and added their own attributes. Only thing that MS Test has advantage over NUnit is that it integrates nicely into the Visual studio 2005 / 2008 IDE. It also provides the additional feature of Code Coverage. A jazzy UI is a good way to impress people to use MS Test.

Another advantage I find with MS Test is the generic syntax for the Asserts. We could specify the data type that we want to check while checking for methods like AreSame or AreEqual. Instead of checking objects, we could specify the exact type, for e.g. Assert.AreSame<string>(expected, actual, "Expected and actual values are not same);

One thing which I find irritating with MS Test though is the speed of execution of unit tests. Its comparatively very slow. Just to measure the performance, I ran the same set of tests both with MS Test and NUnit. To my surprise NUnit was at least 3-4 times faster than MS Test. I found a link which explains why is it so.


Given the fact that NUnit is open source there are more possibilities of it being updated with latest changes to frameworks and related stuff. Whereas MS Test being part of Visual Studio, we need to wait for major releases to get updates for it. In order to overcome the Code Coverage feature of MS Test we could use the TestDriven.Net tool which is an add-in for Visual Studio. It provides code coverage with the help on NCover. Recent versions of NUnit also support the concept of Fluent Interface which allows constraints checking using more readable format. It helps developers write tests and specially asserts in a more natural way.

Along with the two frameworks that I have compared here, there is also a recent addition to this list in the for of XUnit. The learning curve is slightly higher in this case as it doesn't follow the same approach to unit testing as that of NUnit or MS Test. It is still in its nascent stages but someone wanting to test pure .Net code can opt to use it provided there is enough time available to learn new syntax and concepts like Facts.

BankAccount.zip is the solution file which I used to compare the MS test and NUnit.


Do you Spike?

Its been close to a year and half since my team started working on projects which were developed using Agile methodologies. Its been a great experience and a good one as well. In my current project we have hired some consultants from ThoughtWorks to help us improve our processes.

This is the first time we are implementing a project using Agile right from the scratch. We are planning to use various latest technologies for the development purposes. As part of up skilling process we are trying out some sample project. But in Agile terms it seems there is a term called Spike.

The official definition of Spike says, A very simple program to explore potential solutions to address a technical problem. The idea here is to put a pair of developers on the task and reduce the risk which will help in estimating the user story in a reliable way.

This is also the first time we are experimenting with the Agile estimation technique which involves the whole team while estimating for each user story. Traditionally estimation has been done at a very high level by experienced people which might include project manager, tech Lead and or a senior developer. I personally feel using agile estimation helps every member of the team to get a rough estimate of each user story. Every member gives his estimate and the scrum master takes into account an estimate which is acceptable to all in the team. If there is difference in opinion about the efforts required to complete a story, a healthy discussion is held as to why a certain team member feels it should take more or less time as compared to that of another person.

The way we estimated for a user story was based on the acceptance criteria. We were doing an estimation for a web based user interface system. We decided to estimate a story based on following

  • 1 - if the task can be completed within  a day
  • 2 - if the task can be completed in just over a day
  • 3 - if the task can be completed in 2-3 days time and
  • 5 - if the story is too big and needs to broken down into sub stories, or we might not have enough clarity and need more information about it.

The biggest advantage I see in this approach is that everyone's view is taken into consideration and it reduces the risk of one or other person influencing the estimated efforts. It also gives the team an overall idea of all the work involved in the developing a product. It makes the team more self reliant and reduces dependencies on individuals. It definitely helps in the collective ownership of the project.