A guide to making a nipy release

A guide for developers who are doing a nipy release

Release tools

There are some release utilities that come with nibabel. nibabel should install these as the nisext package, and the testing stuff is understandably in the testers module of that package. nipy has Makefile targets for their use. The relevant targets are:

make check-version-info
make check-files
make sdist-tests

The first installs the code from a git archive, from the repository, and for in-place use, and runs the get_info() function to confirm that installation is working and information parameters are set correctly.

The second (sdist-tests) makes an sdist source distribution archive, installs it to a temporary directory, and runs the tests of that install.

If you have a version of nibabel trunk past February 11th 2011, there will also be a functional make target:

make bdist-egg-tests

This builds an egg (which is a zip file), hatches it (unzips the egg) and runs the tests from the resulting directory.

Release checklist

• Review the open list of nipy issues. Check whether there are outstanding issues that can be closed, and whether there are any issues that should delay the release. Label them !

• Review and update the release notes. Review and update the Changelog file. Get a partial list of contributors with something like:

  git log 0.2.0.. | grep '^Author' | cut -d' ' -f 2- | sort | uniq

  where 0.2.0 was the last release tag name.

  Then manually go over git shortlog 0.2.0.. to make sure the release notes are as complete as possible and that every contributor was recognized.

• Use the opportunity to update the .mailmap file if there are any duplicate authors listed from git shortlog -ns.

• Add any new authors to the AUTHORS file. Add any new entries to the THANKS file.

• Check the copyright years in doc/conf.py and LICENSE

• Check the examples in python 2 and python 3, by running something like:

  cd ..
  ./nipy/tools/run_log_examples.py nipy/examples --log-path=~/tmp/eg_logs

  in a python 2 and python 3 virtualenv. Review the output in (e.g.) ~/tmp/eg_logs. The output file summary.txt will have the pass file printout that the run_log_examples.py script puts onto stdout while running.

• Check the long_description in nipy/info.py. Check it matches the README in the root directory, maybe with vim diffthis command.

• Do a final check on the nipy buildbot

• If you have travis-ci building set up you might want to push the code in its current state to a branch that will build, e.g:

  git branch -D pre-release-test # in case branch already exists
  git co -b pre-release-test
  git push origin pre-release-test

• Make sure all the .c generated files are up to date with Cython sources with:

  ./tools/nicythize

Release checking - buildbots

• Check all the buildbots pass

• Run the builder and review the possibly green output from http://nipy.bic.berkeley.edu/builders/nipy-release-checks

  This runs all of:

  make distclean
  python -m compileall .
  make sdist-tests
  make bdist-egg-tests
  make check-version-info
  make check-files

• You need to review the outputs for errors; at the moment this buildbot builder does not check whether these tests passed or failed.

• make bdist-egg-tests may well fail because of a problem with the script tests; if you have a recent (>= Jan 15 2013) nibabel nisext package, you could try instead doing:

  python -c 'from nisext.testers import bdist_egg_tests; bdist_egg_tests("nipy", label="not slow and not script_test")'

  Eventually we should update the bdist-egg-tests makefile target.

• make check-version-info checks how the commit hash is stored in the installed files. You should see something like this:

  {'sys_version': '2.6.6 (r266:84374, Aug 31 2010, 11:00:51) \n[GCC 4.0.1 (Apple Inc. build 5493)]', 'commit_source': 'archive substitution', 'np_version': '1.5.0', 'commit_hash': '25b4125', 'pkg_path': '/var/folders/jg/jgfZ12ZXHwGSFKD85xLpLk+++TI/-Tmp-/tmpGPiD3E/pylib/nipy', 'sys_executable': '/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python', 'sys_platform': 'darwin'}
  /var/folders/jg/jgfZ12ZXHwGSFKD85xLpLk+++TI/-Tmp-/tmpGPiD3E/pylib/nipy/__init__.pyc
  {'sys_version': '2.6.6 (r266:84374, Aug 31 2010, 11:00:51) \n[GCC 4.0.1 (Apple Inc. build 5493)]', 'commit_source': 'installation', 'np_version': '1.5.0', 'commit_hash': '25b4125', 'pkg_path': '/var/folders/jg/jgfZ12ZXHwGSFKD85xLpLk+++TI/-Tmp-/tmpGPiD3E/pylib/nipy', 'sys_executable': '/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python', 'sys_platform': 'darwin'}
  /Users/mb312/dev_trees/nipy/nipy/__init__.pyc
  {'sys_version': '2.6.6 (r266:84374, Aug 31 2010, 11:00:51) \n[GCC 4.0.1 (Apple Inc. build 5493)]', 'commit_source': 'repository', 'np_version': '1.5.0', 'commit_hash': '25b4125', 'pkg_path': '/Users/mb312/dev_trees/nipy/nipy', 'sys_executable': '/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python', 'sys_platform': 'darwin'}

• make check-files checks if the source distribution is picking up all the library and script files. Look for output at the end about missed files, such as:

  Missed script files:  /Users/mb312/dev_trees/nipy/bin/nib-dicomfs, /Users/mb312/dev_trees/nipy/bin/nifti1_diagnose.py

  Fix setup.py to carry across any files that should be in the distribution.

• Check the documentation doctests pass from http://nipy.bic.berkeley.edu/builders/nipy-doc-builder

• You may have virtualenvs for different python versions. Check the tests pass for different configurations. If you have pytox and a network connection, and lots of pythons installed, you might be able to do:

  tox
  

  and get tests for python 2.5, 2.6, 2.7, 3.2. I (MB) have my own set of virtualenvs installed and I’ve set them up to run with:

  tox -e python25,python26,python27,python32,np-1.2.1

  The trick was only to define these testenv sections in tox.ini.

  These two above run with:

  make tox-fresh
  make tox-stale

  respectively.

  The long-hand not-tox way looks like this:

  workon python26
  make sdist-tests
  deactivate

  etc for the different virtualenvs.

Doing the release

• The release should now be ready.

• Edit nipy/info.py to set _version_extra to ''; commit. Then:

  make source-release

• Once everything looks good, you are ready to upload the source release to PyPi. See setuptools intro. Make sure you have a file \$HOME/.pypirc, of form:

  [distutils]
  index-servers =
      pypi
  
  [pypi]
  username:your.pypi.username
  password:your-password
  
  [server-login]
  username:your.pypi.username
  password:your-password

• Once everything looks good, upload the source release to PyPi. See setuptools intro:

  python setup.py register
  python setup.py sdist --formats=gztar,zip upload

• Trigger binary builds for Windows from the buildbots. See builders nipy-bdist32-26, nipy-bdist32-27, nipy-bdist32-32. The exe builds will appear in http://nipy.bic.berkeley.edu/nipy-dist . Download the builds and upload to pypi.

• Trigger binary builds for OSX from the buildbots nipy-bdist-mpkg-2.6, nipy-bdist-mpkg-2.7, nipy-bdist-mpkg-3.3. egg and mpkg builds will appear in http://nipy.bic.berkeley.edu/nipy-dist . Download the eggs and upload to pypi.

• Download the mpkg builds, maybe with:

  scp -r buildbot@nipy.bic.berkeley.edu:nibotmi/public_html/nipy-dist/*.mpkg .

  Make sure you have github bdist_mpkg installed, for the root user. For each mpkg directory, run:

  sudo reown_mpkg nipy-0.3.0.dev-py2.6-macosx10.6.mpkg root admin
  zip -r nipy-0.3.0.dev-py2.6-macosx10.6.mpkg.zip nipy-0.3.0.dev-py2.6-macosx10.6.mpkg

  Upload the mpkg.zip files. (At the moment, these don’t seem to store the scripts - needs more work)

• Tag the release with tag of form 0.3.0:

  git tag -am 'Second main release' 0.3.0

• Now the version number is OK, push the docs to sourceforge with:

  cd doc
  make upload-stable-web-mysfusername

  where mysfusername is obviously your own sourceforge username.

• Set up maintenance / development branches

  If this is this is a full release you need to set up two branches, one for further substantial development (often called ‘trunk’) and another for maintenance releases.

  • Branch to maintenance:

    git co -b maint/0.2.x

    Set _version_extra back to .dev and bump _version_micro by 1. Thus the maintenance series will have version numbers like - say - ‘0.2.1.dev’ until the next maintenance release - say ‘0.2.1’. Commit. Don’t forget to push upstream with something like:

    git push upstream maint/0.2.x --set-upstream

  • Start next development series:

    git co main-master

    then restore .dev to _version_extra, and bump _version_minor by 1. Thus the development series (‘trunk’) will have a version number here of ‘0.3.0.dev’ and the next full release will be ‘0.3.0’.

  • Merge -s ours the version number changes from the maint release, e.g:

    git merge -s ours maint/0.3.x

    This marks the version number changes commit as merged, so we can merge any changes we need from the maintenance branch without merge conflicts.

  If this is just a maintenance release from maint/0.2.x or similar, just tag and set the version number to - say - 0.2.1.dev.

• Push tags:

  git push --tags

• Announce to the mailing lists.