NCrunch Console Tool

Summary

A console version of NCrunch is available for teams that need to make use of NCrunch features outside an IDE. For example, in order to use NCrunch on a team build server.

The tool is limited to a single end-to-end build and test run over a solution and does not include continuous behaviour. This makes it ideal for using NCrunch's parallel execution and distributed processing features as a part of any batch process.

The tool is available through a separate installer. This makes it possible to use it on a machine that does not have Visual Studio installed. Note that SDKs are often still required for the tool to be able to build your projects.

Licensing

As with NCrunch's IDE plugins, the console tool requires an NCrunch license in order to work. However, because NCrunch is licensed per user (not per machine), there is no need for already licensed users or teams to purchase an additional license in order to use it. As long as one member of a team is already a licensed NCrunch user, the team is free to install the NCrunch console tool on their build server using this team member's license code.

The console tool can work with both NCrunch Named User and Company Seat licenses. It can also be used with a 30-day evaluation license.

The goal of the licensing for this tool is simply to make sure that any team using the tool already holds NCrunch licenses. If no team member has an NCrunch license, a license must be purchased to use this tool (outside its 30-day evaluation).

Because many continuous integration servers do not run under the desktop user profile, the console tool will store its licensing information at machine level. This is different from the NCrunch IDE plugin, where licenses are stored under the user profile. In order to use the tool, you must first specify your license with the following command:

ncrunch.exe /license

Note that this can also be used to turn on the 30-day evaluation license.

You can also specify license details using the command line. For example:

ncrunch.exe /license "Joe Bloggs" "1234-1234-1234-1234-1234-1234-1234-1234-1234"

Features

The console tool uses the NCrunch core engine in its entirety. This means it is capable of loading the projects inside a solution, parsing them, analysing them, creating workspaces, building, instrumenting output assemblies and executing tests. It is essentially a headless version of NCrunch. On completion of one full engine cycle, the console tool will write a series of reports to disk then self-terminate.

Useful features include:

• Workspacing - Work is performed away from the source file locations, allowing the source code to also be used by other processes.
• Project build - The console tool builds the projects itself. This can remove the need to have a separate MSBuild step inside your CI.
• Smart test execution - All tests that have not been ignored by NCrunch configuration will be executed by the tool in priority order, as controlled by the Pipeline Optimisation Priority configuration setting.
• Parallel execution - If NCrunch configuration allows it, the tool will build projects and run tests in parallel in the same manner as the NCrunch VS plugin.
• Distributed processing - Where grid servers exist inside the tool's configuration, it will use them to run builds and tests on its behalf. This can greatly accelerate a CI system.
• Results and metrics reporting - The reports generated by the console tool can be very useful for assessing the status of a solution, including its test code coverage levels.
• Impact detection - By adjusting the Tests To Execute Automatically setting, it's possible to make the tool only run impacted tests.
• JetBrains' Team City integration - The tool has some level of integration with Team City, allowing test results to be read directly in Team City reports.

Because the tool operates with the same constraints as the NCrunch VS plugin, it is also a useful on a CI server to ensure that team members not using NCrunch have not broken the build for team members that are using NCrunch.

Usage

The tool is controlled via command-line parameters. For a detailed list of available options, simply execute the tool with no parameters specified.

It's important to consider that any server using the tool must have installed any SDKs and components that would normally be required to build a solution and run its tests. The NCrunch console tool is subject to the same constraints as any team build server in terms of software requirements. You may or may not need to install Visual Studio or the Windows SDK depending upon the nature of your solution.

The tool will obtain its solution and project level configuration from the NCrunch configuration files inside the codebase. If you experience problems with the tool on a CI server not working the same as the VS plugin, ensure that you have these configuration files checked into your source control system.

For a full setup and usage guide, see the console tool usage guide.

Reports

After an execution run, the tool will write the following reports into the results output directory:

• AllResults.html - This is an extensive HTML report including build and test results along with test coverage information. It is designed entirely for human reading of engine results. The tests themselves will also write their output to file underneath the 'TestOutput' directory. To read this output, you will need to follow the links inside the HTML report.
• TimeLine.html - This is a detailed HTML report giving a full timeline of all tasks executed during the run, including build, analysis and test tasks. Task processors are separated into horizontal lanes and the tasks are shown as bars projected vertically over time. This is a useful way to identify bottlenecks in the run or to help understand how best to optimise your configuration for improved throughput.
• RawCoverageResults.xml - This is an XML file containing NCrunch's raw test coverage data. It can be useful if you are seeking to interpret the results using other tools.
• RawResults.xml - This is an XML file containing the raw build and test results as output from the NCrunch engine. It is less legible than the HTML report, but easier to read using other tools.
• TestResultsInNUnitFormat.xml - This XML file is in the same structure as NUnit's standard test results export file. It can be used to import data into other tools. Note that this format is emulated by NCrunch and as such the data may differ slightly when compared with normal NUnit results, but the file should be similar enough that it can be used with other tools that support NUnit results files. You do not need to be using NCrunch with NUnit to export data in this format - it will export simplified data from any test frameworks you are using.
• Partcover.xml - This file contains NCrunch's code coverage data exported in Partcover format. This file can be used for importing code coverage data into tools like NDepend.

Process Exit/Return Codes

The exit codes returned by the console tool will vary depending upon the result of the execution run.