Getting started with the TVIPS tools

The tiff2smv and tvips2smv utilities convert sequences of related electron diffraction images, or MicroED sweeps, recorded by Tietz Video and Image Processing Systems (TVIPS) cameras to the Super Marty View (SMV) format. Diffraction data must be recorded in rolling shutter mode under continuous rotation of the stage. Output SMV images are directly suitable for further processing in suites such as DIALS, MOSFLM, or XDS. tiff2smv supports TIFF images written by either EMMENU or EM_Player, as long as their pixel values can be losslessly represented as 16-bit unsigned integers. tvips2smv supports all TVIPS set files written by EMMENU.

Using the programs

$ tiff2smv -v foo/*.tiff

This will read all the files whose names end in .tiff from the directory foo. tiff2smv will provide verbose diagnostics on standard output, but will not write any converted images.

$ tiff2smv -o "/tmp/bar_##.img" foo/*.tiff

Convert all the .tiff files in directory foo to SMV format. The template name for the output files, t_##.img, allows for up to 100 images (i.e. /tmp/bar_01.img, /tmp/bar_02.img, ..., /tmp/bar_99.img, /tmp/bar_00.img) to be written before paths are recycled and the the hundred-first converted image overwrites /tmp/bar_01.img. tiff2smv will prompt before overwriting files.

$ tvips2smv -S :16 -d 3600 -r 0.3 -v -o "/tmp/bar_###.img" foo_00?.tvips

Convert the 15 first images stored in TVIPS set files to SMV format. Depending on the size of the dataset and the version of EMMENU used to record it, there can be more than one of these files for each dataset, and they have to be listed on the command line in order of increasing sequence number; using shell wildcards such as ? and * will usually ensure this is the case. The distance in the output files will be set to 3600 mm and the oscillation ranges and start angles for each image will correspond to a tilt rate of 0.3° per second.

Full details are available in the man pages (man tiff2smv).

Availability

The software is copyright Howard Hughes Medical Institute, and made available under a permissive Internet Software Consortium license. Binary distributions for a limited set of systems are available from http://www.ccpem.ac.uk/downloads. Source distribution tarballs are available for download from http://cryoem.ucla.edu/downloads.

Compiling and installing the programs

Dependencies

The TVIPS tools depend on LibTIFF, version 4.0.0 or later. To build the sources C- and C++-compilers, flex (version 2.5.34 or later), and CMake (version 3.4 or later) are required. All steps should complete without errors or warnings. If they do not, please file a report on the gonenlab issue tracker.

Unix-like systems

Source archives of the TVIPS tools can be unpacked using

$ tar -xvzf tvips-tools-X.Y.Z.tar.gz

where X, Y, and Z are the major, minor, and patch version numbers, which change from release to release according to the rules of semantic versioning. The sources are configured in a fresh build directory with

$ mkdir build
$ cd build
$ cmake ../tvips-tools-X.Y.Z -DCMAKE_INSTALL_PREFIX:PATH=${prefix}

The name of the build directory is arbitrary; the (relative) path to the source directory, which contains the CMakeLists.txt file (../tvips-tools-X.Y.Z in the example above), needs to be adjusted correspondingly. ${prefix} denotes the desired destination location of the compiled binaries and the accompanying documentation. If LibTIFF is installed in a non-standard location, ${LibTIFF_prefix}, the cmake invocation may need to be amended with -DTIFF_INCLUDE_DIR=${LibTIFF_prefix}/include and -DTIFF_LIBRARY=${LibTIFF_prefix}/lib/libtiff.so.

Once the build system has been generated, compile the sources using

$ make

Optionally, run the test suite

$ make test

Finally, to install the binaries and the documentation, do

$ make install

MinGW on Microsoft Windows

The TVIPS tools can be compiled on Microsoft Windows using the Minimalist GNU for Windows (MinGW) development environment. MinGW does not provide flex, but a Windows-port can be obtained from Win flex-bison. Since MinGW does also not provide LibTIFF and the binaries available from GnuWin32 are too old, a custom installation is required. With source code downloaded from http://download.osgeo.org/libtiff, LibTIFF can be installed from Command Prompt (cmd.exe) using

> unzip tiff-4.0.9.zip
> mkdir tiff-4.0.9-build
> cd tiff-4.0.9-build
> cmake -G "MinGW Makefiles" -DCMAKE_INSTALL_PREFIX=%LibTIFF_prefix% ..\tiff-4.0.9
> mingw32-make
> mingw32-make test
> mingw32-make install

where %LibTIFF_prefix% denotes the desired location of the LibTIFF installation. To build a static LibTIFF without ZIP support, the CMake invocation can be amended with -DBUILD_SHARED_LIBS=OFF -Dzlib=OFF.

The build system for the TVIPS tools must be made aware of the installation prefix of the LibTIFF library prior to compilation

> tar -xvzf tvips-tools-X.Y.Z.tar.gz
> mkdir build
> cd build
> cmake -G "MinGW Makefiles" -DCMAKE_INSTALL_PREFIX:PATH=%prefix%  -DFLEX_EXECUTABLE="%win_flex_bison_prefix%\win_flex" -DTIFF_INCLUDE_DIR="%LibTIFF_prefix%\include" -DTIFF_LIBRARY="%LibTIFF_prefix%\bin\libtiff.dll" ..\tvips-X.Y.Z
> mingw32-make
> mingw32-make test
> mingw32-make install

where %prefix% denotes the desired destination location of the compiled binaries and the accompanying documentation, and X, Y, and Z are the major, minor, and patch version numbers of the TVIPS tools. %win_flex_bison_prefix% is the path to where the Win flex-bison was extracted.

To run a dynamically linked executable, the runtime linker needs to locate libtiff.dll, libgcc_s_dw2-1.dll, libiconv-2.dll, and pthreadGC-3.dll. On Microsoft Windows, this can be accomplished by placing a copy of these dynamic-link libraries in the same directory as the compiled executables. The first library is found in %LibTIFF_prefix%\bin as defined above; the remaining libraries are part of the MinGW installation and located in C:\MinGW\bin by default. This does not necessarily violate the GNU General Public License due to the GCC Runtime Library Exception.

To build statically linked TVIPS tools, add -DCMAKE_EXE_LINKER_FLAGS=-static and replace bin/libtiff.dll with lib/libtiff.a in the definition of TIFF_LIBRARY.

The test suite should pass for dynamically and statically linked TVIPS tools.

Developers

Developers wishing to contribute source code will need access to the live repository, which requires the Git utility, a user account at GitHub, and membership in the gonenlab group. The last step may require some interaction with the present developers in the MicroED team; sending an e-mail to microed@hhmi.org is a good way to start. To obtain a local copy of the source tree, the GitHub account first has to be set up for SSH access, which means generating an SSH key and adding it to the GitHub account. Then

$ git clone git@github.com:JaneliaSciComp/gonenlab.git mygonenlab

This clones the master branch of the remote GitHub repository to a local directory. The directory name is arbitrary; mygonenlab is just an example, and will be used in what follows. The build system currently requires git to be in PATH, otherwise the compiled tools will report incorrect information when invoked with the -V or --version option.

To keep the sources up-to-date as other developers push their changes, do

$ cd mygonenlab
$ git pull

Local modifications to the source code may give rise to conflicts in this step. Such conflicts have to be resolved manually.

Building the documentation

Releases include pre-compiled documentation, in machine-independent man and portable document format (PDF). Developers who are working on a local clone of the GitHub repository and wish to have the documentation installed will have to build it separately. This requires Libxml2, Apache FOP, and the DocBook XSL stylesheets. To build the man-pages, do

$ make man

To generate the PDF documentation, do

$ make pdf

To generate the documentation in HyperText Markup Language (HTML), do

$ make html

Rolling a distribution

Building a tape archive for distribution requires an uncluttered source tree. This is because CMake’s packaging mechanism works by collecting all files in the source directory. Unlike the GNU Autotools, CMake does not provide a distclean target; developers are therefore encouraged to keep their source and build directories separate. Then

$ make dist

This step also requires pandoc to convert the GitHub-flavored markdown files LICENSE.md and README.md to plain text.

If the source directory is unchanged since it was last tagged, this creates a file called tvips-tools-X.Y.Z.tar.gz in the build directory. Otherwise, the name of the distribution tarball will be tvips-tools-branch-date.tar.gz, where branch is the name of the current branch and date is the current date in the form %Y%m%d (see strftime(3)).