Unit Testing Best Practices / NUnit

Introduction

In software development phase and in any programming level, from low level firmware development and embedded architectures to middle level C family and to very high level Java and Microsoft .NET interpreter languages, all need some steps to ensure the correctness and robustness of the software implementation. Sometimes developers would say “It Just doesn’t Make Sense!” complaining some parts of code doesn’t work as expected, and everyone would certainly confess that at least once in his whole life seen such surprises in coding. The worst of all it-doesn’t-make-senses are those that come and tie you up to the time just near the deadlines, a steady endeavor to make things all right, but actually you cannot do something really effective and you’ll fail the mission successfully! Sorry, you’re out of luck! Therefore in order not to fall into such pitfalls you would have to take software engineering practical concepts and best practices into consideration and start off planning it as soon as a project enters into the open project lists. In this paper we are going to evaluate different methods and best practices of Unit Testing. We will also cover the Unit Testing framework, NUnit.

Unit Testing

An early point right after coding a small functionality of a program, a function for instance, is the right time for testing that part to ensure it works as expected, and this is why it is called Unit Testing; testing a particular unit which is usually a function. Unit Testing (UT) prevents deadline pitfalls and reduces the amount of time you would typically use for debugging the software. By Using UT, the lower level modules are fully tested before propagating to other parts of the programs. One benefit of running unit tests is that if a problem appears in a code as project progresses, then we would be able to easily find out where the problem is actually originating from. UT can prevent extra costs of maintenance and system extension by helping the developer finding out bugs and assure the module reliability.

Major benefits:

• Very testable units
• Automatic project documentation (Using automation frameworks that support it)
• Accelerates bug detections
• Accelerates integration
• Detects problems early in development cycle
• Reduces development costs
• Facilitates Regression Testing
• UTs can be used to verify that the implementation adheres to the design

Before going on, let’s review some concepts related to testing.

Unit	For the purposes of testing, a unit is the smallest piece of code, usually a method. Because usually tests can be costly due to the low performance in cross reference or cross boundary, therefore the code inside a unit should prohibit calling outside of the class containing it as much as possible.
Test	Test Is a code that is written to test whether all the test-cases passes or not.
Test Case	Test Cases (TC) contain all the possible cases that a test can be tested with. There is a whole different between Test and TCs, each Test is written to check a specific required functionality, and TCs are fed as an input to that Test to make sure all possible cases are handled. Parameterized Tests can accept TCs as input.
Test Stubs	Test Stubs provide local (non-production) data for the application for the purposes of testing in development/test phases. Test Stubs are inside the application and are replaced with real server data in production code. Method Stubs are the interfaces that provide local data to the application through methods.
Test Harness	Is a batch of Tests with their related TCs that is prepared to test a specific part(s) of a software and makes it possible for the Automation Framework monitoring the behavior of the software under different conditions –for instance, high loads- and different Test Cases. Automation Frameworks also make it possible to automate the testing process –for an instance, scheduling the job at nights- and also to provide a decent report of the tests. Unlike Test Stubs, Test Harness is not tied to the application and is apart from it and is used to simulate the data end points or services that a testing environment needs, such as a PayPal test web services, remote objects and services.
Mock Object	Is a simulated object and has the same behavior of the real object so that it can be used to test a specific functionality. MOs are useful for many cases -for instance, some situations that are difficult to happen naturally, like simulating the temperature exceptions in nuclear infrastructures, or a network response from a client.
Regression Testing	Typically throughout the development phases, maintenance, or the development of new updates and patches, you’ll have to modify the modules, libraries, and etc. Regression Testing is used in such situations to recheck and retest all the parts to make sure things work smoothly and that the new or updated features won’t break other parts.

It is recommended to keep the unit as small as possible, a generic/primitive function. Why do you have to keep the unit as small as possible? Because some classes may call other classes, therefore if you define a unit that contains multiple references of classes, then performance problems would occur. As a result, a unit should be as generic and as small as possible and it should not have references to other classes; its scope just has to be limited to its own class. Also, consider writing tests efficiently and optimized because they’re called in each build. In Integration Testing there is no limitation and you can cross reference a class, or even outer bound processes.

Integration Testing

Integration Testing is done after Unit Testing succeeds. Earlier we said you have to keep the testing units as small as possible and they should not cross their class boundary. So what if you want to test a functionality that spans multiple classes or crosses multiple processes or networks? You have to use Integration Testing. In IT, a set of Unit Tests are combined and tested as a group based on a Test Plan. Test Cases for IT are designed in a way to ensure the correctness of inter-unit relations and interactions. IT can also be used to verify the reliability and performance of the units involved.

When to UT?

When a new function is added to a module or class, then you can start implementing the UT part of it at the same time, but the UT part of the whole system can be implemented beforehand –like in TDD methodology- depending on software design strategies of the project. Note that you must never postpone writing UTs to later times because you would pay much later for that -maybe double or triple times more cost than what can be spent right now! So make sure you do write UTs for a function before proceeding to start working on other modules. The best thing suggested here is to write the UT right after writing the smallest unit of code that in your case might be a function.

It should be noted that right after the UT for a function passes, it is important that all of the other UTs pass too. This ensures software integrity and prevents collateral damages (collateral damage happens when one part causes damage to other parts); the methods might be interrelated somehow- so that no other parts of the code would fail right after adding the new function, or updating its functionality.

Who is responsible for UT?

The first and the only one who is responsible to make sure that all UTs pass and that the functions work as expected is the software developer. Software developer can also run some integration tests to make sure the desired combination of methods interact as expected. After this step, Tester groups test what is done in the first step and rerun unit tests with new and random sets of test cases, and also prepare a Test Plan and related test cases to perform some kind of integration test to check the validity of component interactions. Finally, Quality Assurance group run some kind of integration tests to verify the project’s defined qualities -for an instance, performance analysis.

Short-Running and Long-Running Tests

Types of tests, based on time consumption is categorized into long-running and short-running tests. Short-running ones would execute normally as it doesn’t need extra resources such as I/O, CPU or other factors, but longer-running tests need much time to complete comparing to short ones, it might be due to system interrupts, I/O or CPU intensive operations or other factors. Usually the frameworks should have an option to classify and separate shorter tests from longer tests to bind related attributes –for an instance, scheduling options- in order to setup the tests.

Does It Take Too Much Time?

To answer this question you have to ask yourself how often you had to review a part of code and were wondering why the code does not work as expected. Or that, how much time you spent on a code base to debug and track code variables. This might not show off well in small projects but would definitely show itself for relatively medium to large-sized projects. But of course sometimes you have some complex functions in small projects, so you have to use UT for that too or you’ll end up broken functionality or fragile library that may fail in some circumstances.

TDD

The TDD technique -Test Driven Development- is a useful methodology to widen up your view in testing, particularly in Unit Testing. TDD is a test-driven development process, and as the name implies the focus is all on test/development cycles; Therefore TDD deals with the repetition of very short development cycles. The developer starts off writing a test to check the expectations and the desired functionality of the current code being tested. Then if the test fails on the code (usually the first tests fail because the production code is not written yet for that part), then he writes the needed amount of production code to pass the related test. Then all the tests relating to this new code are tested to make sure that the new functionality does not break existing functionalities of other functions or modules that are directly related to each other. Finally, the code is cleaned up and improved -or technically refactored- to comply with project’s defined quality and standards. And this process is repeated over and over as new functionalities are added/updated. Kent Beck, who developed the TDD technique, mentioned that TDD encourages simple designs and inspires confidence. The following diagram shows the TDD technique:

In TDD, tests are written before implementing the production code, maybe along with design/requirement phase, but totally before any attempt to make any production code of a new feature.

Continuous Integration

CI practices cope well with Unit Testing and it is highly recommended to be used with. CI practices focus on recommending practices for team members to prevent integration problems. Some projects need frequent daily integrations and if not managed adequately, will make substantial issues for the whole team while they are merging and modifying resources in the project’s common workspace (mainline, baseline, repository) on the source code server that is shared among team members. CI introduces practices to help prevent such problems in integration which are known as “Integration Hell” or “Merge Hell”. In the team environment, CI server which is the mainline and repository of the project or can also be an integration server –but it is recommended that two servers can be dedicated to do integration and source code repository separately.

Best Practices:

• Make a successful build of your work after each step that is defined by your project manager (the sequence of builds in dependent scenarios, the priorities) -or at the end of the working hour, and make sure all Unit Tests and Integration Tests pass successfully. Each developer should get the latest version of the project from the repository server at the start of the working day.
• Perform frequent builds (including tests), and if they pass then commit changes. This way the early integration bugs can be detected soon for you and each of the other developers. On the other hand, it encourages the developer to write clean, simple, modular and standard code, and makes it possible for quick reverts if anything goes wrong with any new piece of code. It also makes the code quickly available for tester and QA groups, or for other purposes like application demo and etc.
• Code/task space isolation for each developer; each developer should work on an independent part of the code, the more isolation; the better conflicts can be prevented.
• Limit the number of developers working on dependent codes. If two or more team members are working on a same code space, then setup a discipline and a sequence of work and steps and divide tasks in a way to prevent conflicts.
• Consider a code repository to maintain revisions, versions –usually through a revision/version control system/server. Therefore, if a new minor version –a small step- of the project fails and nothing else seems to be working to resolve some errors, the project can get rolled back to the last-known-good version.
• Consider automating the builds and integrations using scripts in automation frameworks.
• The builds should be quickly available to testers and QA team, the sooner the better.
• For the purposes of testing consider a clone of the production server if local environment doesn’t meet the testing needs.
• Automate the whole processes as much as you can; automation can reduce the costs.
• Scripts should be written such that right after each build it automates the unit testing and integration tests.
• It’s also a good idea to consider a report repository including the result of latest builds. The report should include correctness and performance related data.

UT of Legacy Codes

The reality is that you may not have control over every bit of the project codes, some codes might be handed down from another company. Then how to deal with these legacy codes in UT? The following table shows the scenarios and recommends some solutions:

Source	Description
Available	You can easily manage the code and write UTs for them. Note: Most of the libraries made by interpreter languages like .NET have the option to refactor the code -if not obfuscated- in this case you can take the code out and refactor them into class libraries to have more control on them and ultimately write desired UTs for them.
Unavailable	The source code is unavailable, therefore not manageable UT is just enabled on them using wrappers and is just to make sure if it works as expected or not, but you cannot alter the code if UTs fail!

In case the source code is unavailable, like having unmanaged dlls or obfuscated managed dlls, or a web-service end point reference or a service from a SOA based system, you would easily write tests and test them to make sure if they’re working as expected, but as mentioned earlier you will not have access to source code so in case one of the functions does not work as what you think then you’ll have to consider a replacement module or functions for those parts.

UT in Action

The simplest method of UT would be using a custom Assert method.

Step 1: Define the UT method:

How to use it?



Step 2: Use the defined UT method inside the UT code:

In the following example, you expect that the function’s input must be 'false' when a value larger than the 40 centigrade is passed to the method; therefore fails the assertion. The SetupMaxTemperature method returns 'true' if everything is OK. You would define the test to evaluate the argument as bellow:

And then?

Step 3: Write enough test cases to ensure the code works as intended:

TestMaxTemperature(38);

TestMaxTemperature(21);

TestMaxTemperature(42);    // Yields Error! Assertion Failed 

Test Case data are not from main and live data from the production server, but are usually prepared by a local database or substitute data from method stubs or mock objects.

Instead of the message ‘Assertion failed’ you could have written your own message or you can alter the UT method as:

Note that you are not limited to just one UT method to use; you can make many UT methods with different names or arguments for different scenarios/fixtures. The goal is only to make sure things are going well as expected. The examples above demonstrated the simple usage of UT in testing the project's code. UTs can be much more detailed and complicated depending on the scenario.

You can have a set of TCs and organize them into an object:

Before moving on to the next part, let's take a look at the following important practices:

• Use Test Runners to test the project’s code –so that you don’t need to run the application.
• Do not mix NUnit testing codes inside the production code; UTs should be separate in another assembly project referencing and using the production code in order to test it.
• There is a whole difference between debugging and testing since the way you do them differs significantly; debugging is just used after you find out your unit tests yield exceptions. So, do not include any test statement inside the production code and let it be just a space for debugging, nothing else. As mentioned earlier, unit testing should take place outside the current project in an assembly that references the current project.
• Have a good object-oriented design of UTs and categorize them into their relevant groups through namespace or class names. This way you could much easily access and remember them and it is also a good documentation itself and you can call it a self-documentary UT library.
• We mentioned earlier that the Unit Tests should check that the implementation adheres to the design and requirements of the software. Try to match the implementation with the requirement and design documents of the project.
• One of the important things in Unit Testing that should be considered well is the boundary checking and conformance; that includes checking the range, format, order, and etc.
• Try to test the method using test cases that can potentially break and fail the test.
• Run tests with different environmental factors, Network, RAM, Disk, Security, Proxy, Firewall, Access Permissions, and High Loads.
• QA team should perform a performance check to see if methods are running efficiently.
• Testers should have a good knowledge of syntax and specification of the language which is used to implement and run tests.
• Implement a mock object and its relevant test stubs for a production object that has nondeterministic behavior or provides unpredictable results, or a condition that is hard to happen in real situation. For example, network delays can somehow represent nondeterministic behavior of the network, or a complex system like an aircraft needs to be simulated. 
• Consider using a mock framework to analyze the system’s behavior. This kind of framework could be used by a Tester team to verify and confirm the codes and project’s developing libraries, thus improving the software’s quality. But bear in mind that using a mock framework also may introduce some extra costs (amount of code and other relevant complexities), so before deciding to mock an object think of a workaround and that if it can be tested using a quite simpler way. Therefore only implement mock objects for complex and detailed objects that really need it.
• Use Code Coverage frameworks to mark and notify you of uncovered codes (the production codes that have no relevant unit tests yet) and consequently a report on how much codes are covered in percentage. The amount of code coverage for production codes of the project varies from project to project; some projects need a little bit of coverage just to make sure they’re doing their job using expected input data while some others need a high and detailed code coverage. Choosing between two would need a deep consideration and accuracy in project planning and requirement and a trade-off should be made. 
• Unit Testing should focus on testing behaviors than testing every single part of the project. Remember the Unit Testing was made to minimize the software development phase; the tests should not be planned and written in a way that make it worse, try to make a trade-off as usual. 
• Write the tests in a way that is automatic and that it can be run by someone else, testers, QA.
• The unit test project must be defined, designed, developed, and maintained complying with software engineering principles and is as important as the project itself. 
• Consider configuring a source code management system for the UT project on the server.
• On the UT project, consider unit test code isolation (maybe file-level, or class-level) for every team member so that they implement and run their own tests and this prevents test conflicts. Also there might be some other methods of code-space isolation and what you will eventually employ depends on UT project’s planning and design. You might also consider a separate code space for testers and QA.
• How often should you run UTs? After writing a new method, after a compile, fixing a bug, checking in/out with the source control server, and automated scripts to run the tests periodically (using automation frameworks) –usually at nights, but it depends on the project’s size and complexity.

NUnit 

NUnit is a Unit Testing framework for .NET languages.




Configuration

You’ll install and setup NUnit through Visual Studio 2012(latest update) or VS 2013+ and its NuGet feature. In Visual Studio, Tools menu, choose Library Package Manager and then Package Manager Console. Then, The Package Manager Console panel will be displayed at the bottom of the screen. In the panel, choose the target working project from the drop down and leave other options as their defaults. Now, to enter the install script, visit NUnit official website and copy the NUnit installation script of the needed version. The following link shows the latest stable version, but if you’re looking for a beta version or looking for older versions, try the version history at the bottom of the web page.

The Package Manager Console’s prompt starts with ‘PM>’. Finally paste the script in the package console and hit Enter key. The following script downloads and sets up the latest stable version at the time being:

PM> Install-Package NUnit -Version 2.6.3

You should get the following message if NUnit installs successfully:

Successfully installed 'NUnit 2.6.3'.
Successfully added 'NUnit 2.6.3' to %project-name%.

Note: The same script should be executed for each target project and it will setup all the required things automatically. But some may prefer add referencing the required libraries.

Once you installed NUnit, you also need to install a NUnit Test Runner which is a plugin for VS in order to run the tests. This plugin is installed only once because it’s installed in VS IDE, not in project’s repository. From Tools menu, choose Extensions and Updates. Go to Visual Studio Gallery node under Updates tab -it’s recommended to perform the updates in here- and check if NUnit Test Adapter is in the list, if not, look for it in the VS Gallery web site.

Note: Make sure that you have the latest update of Visual Studio 2012 installed or use VS2013+. Restart Visual Studio after the installation.

NUnit Elements

Test-Fixture

The first element is the TestFixture attribute. This attribute is used on a class signifying that it’s a container for all the tests inside the class. Each fixture actually defines a testing scenario and you might plan different scenarios to test a behavior. The next is the Test attribute and is used on a method and make that method testable. Each method can have multiple assertions. We’ll be running tests using Test Cases to make sure the method is actually behaving as we expected. The code bellow shows the simplest form of a NUnit test. 

Note that for all the NUnit tests, the statement 'using NUnit.Framework;' should be declared so we expect you do declare it in your code and we’ll not add it to every code snippet in this writing.

The Floor() is a simple function with no parameter. How can you test the code? Is it possible to run the tests without running the application? Sure, You’ll test them one by one without running the application to make sure of their validity. Test Runners allow you to test units without running the application. A TR is usually a plugin that is installed on the host IDE, and after the plugin installation you’ll find the TR feature within a panel inside the IDE. Here is how the test panel looks like in Visual Studio:

Now that you understand how to use the test tool in VS and its relation with the test codes, let’s move more into NUnit features.

Test-Case

TCs are used to feed test data to the test methods. Imagine that we wanted to test a method using 3 test cases. This is how it is done using the TestCase attribute:

Assertions

Assertions help check the validity of a unit and if the defined condition inside the Assert method is not met, then they would show a default or a custom exception message.

Classic Assertions: These are simple assertions we have dealt with up to now: Assert.That(Math.Floor(3.1) == 3); 

Constraint Assertions: These new types of assertions have almost no difference than classic ones, but their output assertion message varies depending on the applied constraint type.

These two classic and constraint assertions are equal:

Assert.That(Math.Floor(3.1) == 3);   ó   Assert.AreEqual(Math.Floor(3.1), 3);

List of common classic assertions and their signatures:

Assert.AreEqual(expected, actual [, string message])	Assert.AreEqual(expected, actual, tolerance [, string message])
Assert.Less(x, y)	Assert.Greater(x,y)
Assert.GreaterOrEqual(x, y)	Assert.LessOrEqual(x,y)
Assert.IsNull(object [, string message])	Assert.IsNotNull(object [, string message])
Assert.AreSame(expected, actual [, string message])	Assert.IsTrue(bool condition [, string message])
Assert.IsFalse(bool condition [, string message])	Assert.Fail([string message])
FileAssert.AreEqual(FileInfo expected, FileInfo actual)	FileAssert.AreEqual(String pathToExpected, String pathToActual)

List of common constraint assertions and their signatures:

Assert.That(actual, Is.EqualTo(expected))	Assert.That(actual, Is.EqualTo(expected).Within(tolerance))
Assert.That(actual, Is.Not.EqualTo(expected))	Assert.That(actual, Is.AtMost(expected))
Assert.That(expected, Is.Null)	Assert.That(expected, Is.Empty)
Assert.That(actual, Is.AtLeast(expected))	Assert.That(actual, Is.InstanceOfType(expected))
Assert.That(actual, Has.Length(expected))	Assert.That(actual, Text.Matches(expected))

The list of assertions are self-explanatory but some of them need some explanation:

1. Assert.AreEqual(expected, actual, tolerance [, string message]) : The tolerance is the error tolerance of floating-point representation.
2. Assert.AreSame(expected, actual [, string message]) : This method checks if the memory references of two objects are equal.
3. Assert.Fail([string message]) : This method will break and fails the code and is used to mark somewhere in the code that should not be reached.
4. Assert.That(actual, Is.EqualTo(expected).Within(tolerance)) : The Within() method defines the tolerance of the floating-point error.
5. Assert.That(actual, Text.Matches(expected)) : Tests that the expected Regular Expresion string matches the actual string.

Custom Assertions

How to define Custom Assertions?

First define a class that contains a group of related static assert methods. Then you can use existing NUnit assertion inside those methods. Then use these static assert methods inside NUnit test methods instead of NUnit asserts:

And this is how you use it:

The Ignore Attribute

Sometimes some tests need more time; or you have other parts to work on and intend to continue a test at a later time. The Ignore attribute can be set on a test method so that the Test Runner would not execute that test method.

Categories

NUnit supports categorizing tests by providing the Category attribute on the methods being tested. This way you would be able to exclude or include the tests for execution by the Test Runner. For example, you separate shorter-running from longer-running tests, or categorize them by fixture or behavior, or categorizing those test belong to the active part of the project you’re working on. You can assign multiple categories for a test and it means that a test may belong to more than one categories.

Execution Time

QA team should be aware of performance considerations. How can you measure execution time? This can be accomplished by defining a timer and assert the elapsed time:

As you see here, the SetUp attribute is used on an initialization method. The method that has this attribute is called when the class fixture is loaded into the memory. This method should be used for initialization statements; as here the class variable timer is initialized to an instance of Stopwatch class so that we do not have to redefine it in every method of the class.

Conclusion

This paper started with the software engineering aspects of Unit Testing, introduced Test-Driven Development technique and Continuous Integration as well as their best practices and considerations, and ended up by uncovering NUnit framework, explained the related configuration, NUnit elements from TestFixtures to Custom Assertions and useful attributes such as Category and Ignore. Unit Testing is all about the right guess around the problem or the standard approach to test a unit in different conditions using different test cases. One of the important factors -or I should say the most important one- that makes a project successful is the subtle and delicate trade-off the project manager makes; the code coverage, test details, integration and UT automation, development methodology, UT frequency, and etc. And finally, UT and Integration Automation should not be neglected.