Concurrent Use Of Test Resources

Description

A major feature of NCrunch is its ability to run tests in parallel across multiple processes. For compatibility reasons, this feature is disabled by default.

Parallel execution provides considerable benefits in the form of much shorter test cycle times and faster response to codebase changes. However, its use imposes additional constraints on the tests being executed by NCrunch.

When tests are executed in parallel using NCrunch, they will always be executed in separate processes. The intention of this is to prevent tests from interfering with each other through concurrent access to shared state (such as singletons, static members, etc). Because of this, you can be certain that tests processed by NCrunch will never experience concurrency problems provided they do not access resources outside their host process (for example, the file system or a network socket).

Note that NCrunch can still execute multiple tests within a single process, but it will only do so sequentially. If you require a test to be run in complete isolation with its host process torn down after execution, you should use IsolatedAttribute.

IMPORTANT: It is a very common misconception that NCrunch will run tests concurrently across different threads inside the same test process. The current architecture of NCrunch makes this impossible.

Possible Problems

Where tests make use of resources outside their host process, NCrunch must be made aware of the resources involved, or these resources should be carefully arranged so as to avoid their overlapped use. Failure to do this can result in the intermittent failure of the tests involved.

Consider the following test code:

public class Fixture
{
    [Test]
    public void TestWritesToFile1()
    {
        if (File.Exists("File.txt"))
            File.Delete("File.txt");

        using (var writer = new StreamWriter("File.txt"))
        {
            for (int i = 0; i < 10000000; i++)
                writer.Write("a"); 
        }
    }

    [Test]
    public void TestWritesToFile2()
    {
        if (File.Exists("File.txt"))
            File.Delete("File.txt");

        using (var writer = new StreamWriter("File.txt"))
        {
            for (int i = 0; i < 10000000; i++)
                writer.Write("b");
        }
    }
}

With the Max number of processing threads set to a value higher than 1, and the Allow parallel execution option set to true, NCrunch will attempt to run both the above tests in parallel with each other. This will almost certainly result in the failure of one of the two tests due to an IOException as both tests try to write to the same file at the same time.

Solutions

Resource Usage Attributes

The easiest solution to this problem is to make NCrunch aware that both tests are using a resource outside the test process. This can be done by adorning the tests with the ExclusivelyUsesAttribute. In this situation, the attribute could be placed either on both tests or the entire fixture with the result being effectively the same; NCrunch will not run the tests in parallel with each other.

Another option is to use the SerialAttribute, although this is only recommended as a last resort, as the serial attribute broadly places very tight constraints around parallel execution for the entire test pipeline.

Test Concurrency Resilience

Where possible, the best solution is to engineer the tests in such a way that they never access the same file. A clean approach to this is ensuring each test works within its own transient directory that is cleaned up when the test is torn down. For example:

public class Fixture
{
    private string originalDirectory;
    private string temporaryTestDirectory;

    [SetUp]
    public void SetupTestDirectory()
    {
        temporaryTestDirectory = Path.GetRandomFileName();
        Directory.CreateDirectory(temporaryTestDirectory);

        originalDirectory = Directory.GetCurrentDirectory();
        Directory.SetCurrentDirectory(temporaryTestDirectory);
    }

    [TearDown]
    public void TearDownTestDirectory()
    {
        Directory.SetCurrentDirectory(originalDirectory);
        if (Directory.Exists(temporaryTestDirectory))
            Directory.Delete(temporaryTestDirectory, true);
    }

    [Test]
    public void TestWritesToFile1()
    {
        if (File.Exists("File.txt"))
            File.Delete("File.txt");

        using (var writer = new StreamWriter("File.txt"))
        {
            for (int i = 0; i < 10000000; i++)
                writer.Write("a"); 
        }
    }

    [Test]
    public void TestWritesToFile2()
    {
        if (File.Exists("File.txt"))
            File.Delete("File.txt");

        using (var writer = new StreamWriter("File.txt"))
        {
            for (int i = 0; i < 10000000; i++)
                writer.Write("b");
        }
    }
}

The random directory generation in the test code above makes it nearly impossible for both tests to access the same file at the same time. Similar concepts can also be applied to other resources. For example, integration tests that make use of socket ports can be written to allocate the port number randomly - greatly reducing the risk of tests clashing with each other.

In situations where random generation is considered untidy or too inconsistent, an alternative is to instead rely on the ID of the currently executing test runner process. For example:

[SetUp]
public void SetupTestDirectory()
{
    // Name our temporary directory according to
    // our current process ID
    temporaryTestDirectory 
        = Process.GetCurrentProcess().Id.ToString();

    if (Directory.Exists(temporaryTestDirectory) == false)
        Directory.CreateDirectory(temporaryTestDirectory);

    originalDirectory = Directory.GetCurrentDirectory();
    Directory.SetCurrentDirectory(temporaryTestDirectory);
}

As NCrunch will never execute tests concurrently within the same process, the test runner process ID is guaranteed to be unique for each test being executed concurrently. This gives an effective way to separate resources used by these tests.