Getting started with the TVIPS tools

The tiff2smv and tvips2smv utilities convert sequences of related electron diffraction images, or sweeps, recorded by Tietz Video and Image Processing Systems (TVIPS) cameras to the Super Marty View (SMV) format. Output SMV images are directly suitable for further processing in suites such as DIALS, MOSFLM, or XDS. tiff2smv supports TIFF images written by either EMMENU or EM_Player, as long as their pixel values can be losslessly represented as 16-bit unsigned integers. tvips2smv supports all TVIPS set files written by EMMENU.

Using the programs

$ tiff2smv -v foo/*.tiff

This will read all the files whose names end in .tiff from the directory foo. tiff2smv will provide verbose diagnostics on standard output, but will not write any converted images.

$ tiff2smv -o /tmp/bar_##.img foo/*.tiff

Convert all the .tiff files in directory foo to SMV format. The converted images will be written to /tmp/bar_01.img, /tmp/bar_02.img, etc. If there are more than one hundred images the hundred-first converted image will overwrite /tmp/bar_01.img.

$ tvips2smv -a -d 3600 -r 0.3 -v -o /tmp/bar_###.img foo_00?.tvips

Convert images stored in TVIPS set files to SMV format. Depending on the size of the dataset and the version of EMMENU used to record it, there will be one or more of these files for each dataset, and they have to be listed on the command line in order of increasing sequence number; using shell wildcards such as ? and * will usually ensure this is the case. The template name for the output files, t_###.img, allows up to 1000 images (i.e. /tmp/bar_000.img, /tmp/bar_001.img, ..., /tmp/bar_999.img) to be written before paths are recycled, and output files are overwritten. The distance in the output files will be set to 3600 mm and the oscillation ranges and start angles for each image will correspond to a tilt rate of 0.3 degrees per second.

There are command line options to set parameters such as the sample-detector distance or the beam center. Full details are available in the man pages (man tvips2smv).

Availability

The software is copyright Howard Hughes Medical Institute, and made available under a permissive Internet Software Consortium license. Binary distributions for a limited set of systems are available from http://www.ccpem.ac.uk/downloads. Source distribution tarballs are available for download from http://cryoem.janelia.org/downloads. Once downloaded, these archives can be unpacked using

$ tar -xvzf tvips-tools-X.Y.Z.tar.gz

where X, Y, and Z are the major, minor, and patch version numbers, which change from release to release according to the rules of semantic versioning.

Compiling and installing the programs

Dependencies

The TVIPS tools depend on LibTIFF, version 4.0.0 or later. Both the headers as well as the shared library are required. To build the sources C- and C++-compilers are needed as well as the CMake utility, version 3.2 or later, and its dependencies.

All steps should complete without errors or warnings. If they do not, please file a report on the gonenlab issue tracker.

Unix-like systems

Assuming the sources are available and all the prerequisites are in place, the TVIPS tools are configured in a fresh build directory with

$ mkdir build
$ cd build
$ cmake ../tvips-tools-X.Y.Z -DCMAKE_INSTALL_PREFIX:PATH=<install path prefix>

The name of the build directory is arbitrary; the (relative) path to the source directory, which contains the CMakeLists.txt file (../tvips-tools-X.Y.Z in the example above), needs to be adjusted correspondingly. If LibTIFF is installed in a non-standard location, the cmake invocation may need to be amended with -DTIFF_INCLUDE_DIR=<LibTIFF install path prefix>/include and -DTIFF_LIBRARY=<LibTIFF install path prefix>/lib/libtiff.so. <install path prefix> denotes the desired destination location of the compiled binaries and the accompanying documentation.

Once the build system has been generated, compile the sources using

$ make

Optionally, run the test suite

$ make test

Finally, to install the binaries and the documentation, do

$ make install

MinGW on Microsoft Windows

The TVIPS tools can be compiled on Microsoft Windows using the Minimalist GNU for Windows development environment. CMake, and optionally Git, need to be installed and in PATH. For the compiled binaries to be portable to systems lacking a MinGW installation, the TVIPS tools and all dynamically linked dependencies must be compiled with the -static-libgcc linker flag. This does not necessarily violate the GNU General Public License due to the GCC Runtime Library Exception.

Since both MinGW's LibTIFF and the binaries available from GnuWin are too old, a custom installation is required. With source code downloaded from http://download.osgeo.org/libtiff, LibTIFF can be installed from Command Prompt (cmd.exe) using

> mkdir tiff-4.0.6-build
> cd tiff-4.0.6-build
> cmake -G "MinGW Makefiles" -DCMAKE_INSTALL_PREFIX=<LibTIFF install path prefix> -DCMAKE_SHARED_LINKER_FLAGS=-static-libgcc ..\tiff-4.0.6
> mingw32-make
> mingw32-make install

where <LibTIFF install path prefix> denotes the desired destination location of the LibTIFF library. The build system for the TVIPS tools must be made aware of the installation prefix of the LibTIFF library prior to compilation

> mkdir build
> cd build
> cmake -G "MinGW Makefiles" -DCMAKE_INSTALL_PREFIX:PATH=<install path prefix>  -DTIFF_INCLUDE_DIR="<LibTIFF install path>\include" -DTIFF_LIBRARY="<LibTIFF install path>\bin\libtiff.dll" ..\tvips-X.Y.Z

where <install path prefix> denotes the desired destination location of the compiled binaries and the accompanying documentation. To run the compiled tools, the run-time linker needs to be able to locate libtiff.dll; this can be accomplished by copying it to the installation directory.

> mingw32-make
> mingw32-make install
> cp -p <LibTIFF install path>\libtiff.dll <install path prefix>\bin

To optionally run the test suite, a copy of libtiff.dll is needed in the build directory as well.

> cp -p <LibTIFF install path>\libtiff.dll .
> mingw32-make test

Developers

Developers wishing to contribute source code will need access to the live repository, which requires the Git utility, a user account at GitHub, and membership in the gonenlab group. This step may require some interaction with the present developers in the MicroED team; sending an e-mail to microed@hhmi.org is a good way to start. To obtain a local copy of the source tree, the GitHub account first has to be set up for SSH access, which means generating an SSH key and adding it to the GitHub account. Then

$ git clone git@github.com:JaneliaSciComp/gonenlab.git mygonenlab

This clones the master branch of the remote GitHub repository to a local directory. The directory name is arbitrary; mygonenlab is just an example, and will be used in what follows. The build system currently requires git to be in PATH, otherwise the compiled tools will not report the correct information when invoked with the -V option.

To keep the sources up-to-date as other developers push their edits, do

$ cd mygonenlab
$ git pull

Local modifications to the source code may give rise to conflicts in this step. Such conflicts have to be resolved manually.

Building the documentation

Releases include pre-compiled documentation, in machine-independent man and PDF formats. Developers who are working on a local clone of the GitHub repository and wish to have the documentation installed will have to build it separately. This requires Libxml2, Apache FOP, and the DocBook XSL stylesheets. To build the man-pages, do

$ make man

To generate the documentation in portable document format (PDF), do

$ make pdf

To generate the documentation in HyperText Markup Language (HTML), do

$ make html

Rolling a distribution

Building a tape archive for distribution requires an uncluttered source tree. This is because CMake's packaging mechanism works by collecting all files in the source directory. Developers are therefore encouraged to keep their source and build directories separate. Then

$ make dist

This step also requires pandoc to convert the GitHub-flavored markdown files LICENSE.md and README.md to plain text.

If the source directory is unchanged since it was last tagged, this creates a file called tvips-tools-X.Y.Z.tar.gz in the build directory. Otherwise, the name of the distribution tarball will be tvips-tools-branch-date.tar.gz, where branch is the name of the current branch and date is the current date in the form %Y%m%d (see strftime(3)).