Testing

Nipy uses the Numpy test framework which is based on nose. If you plan to do much development you should familiarize yourself with nose and read through the numpy testing guidelines.

Writing tests

Test files

The numpy testing framework and nipy extensions are imported with one line in your test module:

from nipy.testing import *

This imports all the assert_* functions you need like assert_equal, assert_raises, assert_array_almost_equal etc..., numpy’s rand function, and the numpy test decorators: knownfailure, slow, skipif, etc...

Please name your test file with the test_ prefix followed by the module name it tests. This makes it obvious for other developers which modules are tested, where to add tests, etc... An example test file and module pairing:

nipy/core/reference/coordinate_system.py
nipy/core/reference/tests/test_coordinate_system.py

All tests go in a test subdirectory for each package.

Temporary files

If you need to create a temporary file during your testing, you could use one of these three methods, in order of convenience:

1. StringIO

   StringIO creates an in memory file-like object. The memory buffer is freed when the file is closed. This is the preferred method for temporary files in tests.

2. nibabel.tmpdirs.InTemporaryDirectory context manager.

   This is a convenient way of putting you into a temporary directory so you can save anything you like into the current directory, and feel fine about it after. Like this:

   from ..tmpdirs import InTemporaryDirectory
   
   with InTemporaryDirectory():
       f = open('myfile', 'wt')
       f.write('Anything at all')
       f.close()
   

   One thing to be careful of is that you may need to delete objects holding onto the file before you exit the with statement, otherwise Windows may refuse to delete the file.

3. tempfile.mkstemp

   This will create a temporary file which can be used during testing. There are parameters for specifying the filename prefix and suffix.

   Note

   The tempfile module includes a convenience function NamedTemporaryFile which deletes the file automatically when it is closed. However, whether the files can be opened a second time varies across platforms and there are problems using this function on Windows.

   Example:

   from tempfile import mkstemp
   try:
       fd, name = mkstemp(suffix='.nii.gz')
       tmpfile = open(name)
       save_image(fake_image, tmpfile.name)
       tmpfile.close()
   finally:
       os.unlink(name)  # This deletes the temp file
   

Please don’t just create a file in the test directory and then remove it with a call to os.remove. For various reasons, sometimes os.remove doesn’t get called and temp files get left around.

Many tests in one test function

To keep tests organized, it’s best to have one test function correspond to one class method or module-level function. Often though, you need many individual tests to thoroughly cover (100% coverage) the method/function. This calls for a generator function. Use a yield statement to run each individual test, independent from the other tests. This prevents the case where the first test fails and as a result the following tests don’t get run.

This test function executes four independent tests:

def test_index():
    cs = CoordinateSystem('ijk')
    yield assert_equal, cs.index('i'), 0
    yield assert_equal, cs.index('j'), 1
    yield assert_equal, cs.index('k'), 2
    yield assert_raises, ValueError, cs.index, 'x'

Suppress warnings on test output

In order to reduce noise when running the tests, consider suppressing warnings in your test modules. This can be done in the module-level setup and teardown functions:

import warnings
...

def setup():
    # Suppress warnings during tests to reduce noise
    warnings.simplefilter("ignore")

def teardown():
    # Clear list of warning filters
    warnings.resetwarnings()

Running tests

Running the full test suite

For our tests, we have collected a set of fmri imaging data which are required for the tests to run. To do this, download the latest example data and template package files from NIPY data packages. See Optional data packages.

Running individual tests

You can also run nose from the command line with a variety of options. To test an individual module:

nosetests test_image.py

To test an individual function:

nosetests test_module:test_function

To test a class:

nosetests test_module:TestClass

To test a class method:

nosetests test_module:TestClass.test_method

Verbose mode (-v option) will print out the function names as they are executed. Standard output is normally supressed by nose, to see any print statements you must include the -s option. In order to get a “full verbose” output, call nose like this:

nosetests -sv test_module.py

To include doctests in the nose test:

nosetests -sv --with-doctest test_module.py

For details on all the command line options:

nosetests --help

Coverage Testing

Coverage testing is a technique used to see how much of the code is exercised by the unit tests. It is important to remember that a high level of coverage is a necessary but not sufficient condition for having effective tests. Coverage testing can be useful for identifying whole functions or classes which are not tested, or for finding certain conditions which are never tested.

This is an excellent task for nose - the automated test runner we are using. Nose can run the python coverage tester. First make sure you have the coverage tester installed on your system. Download the tarball from the link, extract and install python setup.py install. Or on Ubuntu you can install from apt-get: sudo apt-get install python-coverage.

Run nose with coverage testing arguments:

nosetests -sv --with-coverage path_to_code

For example, this command:

nosetests -sv --with-coverage test_coordinate_map.py

will report the following:

Name                                            Stmts   Exec  Cover   Missing
-----------------------------------------------------------------------------
nipy                                       21     14    66%   70-74, 88-89
nipy.core                                   4      4   100%
nipy.core.reference                         8      8   100%
nipy.core.reference.array_coords          100     90    90%   133-134, 148-151, 220, 222, 235, 242
nipy.core.reference.coordinate_map        188    187    99%   738
nipy.core.reference.coordinate_system      61     61   100%
nipy.core.reference.slices                 34     34   100%
nipy.core.transforms                        0      0   100%
nipy.core.transforms.affines               14     14   100%

The coverage report will cover any python source module imported after the start of the test. This can be noisy and difficult to focus on the specific module for which you are writing nosetests. For instance, the above report also included coverage of most of numpy. To focus the coverage report, you can provide nose with the specific package you would like output from using the --cover-package. For example, in writing tests for the coordinate_map module:

nosetests --with-coverage --cover-package=nipy.core.reference.coordinate_map test_coordinate_map.py

Since that’s a lot to type, I wrote a tool called sneeze to that simplifies coverage testing with nose.

Sneeze

Sneeze runs nose with coverage testing and reports only the package the test module is testing. It requires the test module follow a simple naming convention:

1. Prefix test_
2. The package name you are testing
3. Suffix .py

For example, the test module for the coordinate_map module is named test_coordinate_map.py. Then testing coverage is as simple as:

sneeze.py test_coordinate_map.py

Sneeze is included in the tools directory in the nipy source. Simply run the setup.py to install sneeze in your local bin directory.