================================================================
EMDB-SFF TK (SFFTK)
================================================================

.. contents:: Table of Contents

----------------------------------------------------------------
About
----------------------------------------------------------------

SFFTK
================================================================
The EMDB-SFF Toolkit (SFFTK) is a set of utilities to generate, edit and display segmentation data contained in EMDB Segmentation File Format (SFF) files. 
EMDB-SFF is the result of a community effort to unify representation of segmentations derived from various segmentation tools.
The toolkit is designed as a set of subcommands that perform the various functions:

.. code:: bash

	pkorir@pkorir-tarakimu:trunk $ python sff -h
	usage: sff [-h] EMDB-SFF tools ...
	
	The EMDB-SFF Toolkit (sfftk)
	
	optional arguments:
	  -h, --help      show this help message and exit
	
	Tools:
	  The EMDB-SFF Toolkit (sfftk) provides the following tools:
	
	  EMDB-SFF tools
	    convert       converts from/to EMDB-SFF
	    updateschema  update schemas (emdb_sff.py or roi.py) using generateDS.py
	    list          list various entities
	    createroi     create ROIs and write to file
	    attachroi     attach ROIs to image
	    deleteroi     delete ROIs associated with image
	    config        manage configs
	    view          view file summary
	    view3d        render 3D model
	    tests         run unit tests

The various utilities are designed as packages for reusability.

EMDB-SFF
================================================================
The EMDB-SFF has two principal roles:

1. An open and flexible format for storing segmentations
2. A means to capture annotation of segmentations

The current version of the file format is 0.5.8 pending discussion and feedback from the EM community.

As a unifying representation, EMDB-SFF will facilitate display of segmentations alongside associated reconstructed volumes.
EMDB-SFF incorporates four generic segmentation representations:

- 3D volumes
- contours
- meshes (surfaces)
- shape primitives (ellipsoid, cuboid, cylinder, cone)

Funding
================================================================

TBD


----------------------------------------------------------------
Obtaining SFFTK
----------------------------------------------------------------

TBD

----------------------------------------------------------------
Installing SFFTK
----------------------------------------------------------------

TBD

----------------------------------------------------------------
Purpose
----------------------------------------------------------------

SFFTK is designed to:

- convert application specific segmentation file formats (AS-SFF) into EMDB-SFF;
- convert EMDB-SFF files into AS-SFF (where possible);
- display segmentations contained in EMDB-SFF files for inspection;
- prepare OMERO ROIs from segmentations;
- attach generated OMERO ROIs to corresponding images hosted in OMERO;
- annotate segmentations using biological ontologies;
- view segmentation metadata;


----------------------------------------------------------------
Audience
----------------------------------------------------------------

SFFTK is aimed at 3D electron microscopy (3DEM) practitioners.

----------------------------------------------------------------
Input Data
----------------------------------------------------------------

SFFTK accepts the following data types:

- several AS-SFFs (see Supported Formats below for the up-to-date listing)
- ROI files: XML files formatted with the ROI schema (link);
- EMDB-SFF files: XML files formatted with the EMDB_SFF schema;  


----------------------------------------------------------------
Supported Formats
----------------------------------------------------------------

(extension in alphabetic order)

- Amira Mesh (.am)
- EMDB Map masks (.map)
- IMOD (.mod)
- ROI (.roi)*
- Segger (.seg)
- EMDB-SFF (.sff)
- Amira HxSurface (.surf)

*internal file

----------------------------------------------------------------
Dependencies
----------------------------------------------------------------

(alphabetic order of Python import name)

SFFTK has the following dependencies:

- generateDS
- h5py
- javabridge
- matplotlib
- numpy
- omero
- python-bioformats
- scipy
- simpleparser
- skimage for generating contours (the find_contours() function)
- The Visualisation Toolkit (VTK)

----------------------------------------------------------------
Features
----------------------------------------------------------------


Generic Options
================================================================
Some options work on most subcommands. Run `sff <subcommand> -h` to verify available options.

-v/--verbose
-h/--help

Converting Files to EMDB-SFF
================================================================
Supported AS-SFF files are converted to EMDB-SFF files using the `convert` subcommand:

.. code:: bash

	$ sff convert <file.ext>

where `ext` is a supported extension.  

Example:

.. code:: bash

	# verbosely convert an IMOD file into EMDB-SFF and write the output to file.sff
	$ python sff convert -v sff/test_data/test_data.mod -o file.sff
	Thu Jul 14 16:43:04 2016	Created XMLData object from schema <schema.emdb_sff.segmentation object at 0x102a4f610>
	Thu Jul 14 16:43:04 2016	Reading in IMOD file from sff/test_data/test_data.mod file
	Thu Jul 14 16:43:04 2016	Writing out XML to file file.sff

Viewing Segmentations
================================================================
Viewing raw data
----------------------------------------------------------------
The raw data can be viewed through the `view` subcommand:

.. code:: bash

	$ sff view <file.ext>
	
Example:

.. code:: bash

	# view metadata from an Amira Mesh file
	$ sff view sff/test_data/test_data.am

	*******************************************************************************
	Amira Mesh file
	Version:              2.1
	Format:               BINARY-LITTLE-ENDIAN
	*******************************************************************************
	Materials:  
	            
	Mitochondria_
	Id                    4
	Color                 [1.0, 1.0, 0.0]
	
	Inside      
	Id                    2
	Color                 [0.64, 0.0, 0.8]
	
	Mitochondria
	Id                    3
	Color                 [0.0, 1.0, 0.0]
	
	NE          
	Id                    6
	Color                 [1.0, 0.0, 0.0]
	
	mitochondria__
	Id                    5
	Color                 [0.0, 0.125, 1.0]
	
	Exterior    
	Id                    1
	
	            
	Mesh                  200 images each of 971 X 862 pixels (3 colors)
	            
	*******************************************************************************
	

Viewing 3D Segmentations
----------------------------------------------------------------

.. code:: bash

	$ sff view3d <file.sff>
	
Options:

  -n, --normals-off     do not to use normals if present [default: False]
  -a, --no-orientation-axes
                        do not display orientation axes (bottom right of
                        viewport) [default: True]
  -A, --all-contours    show all contours [default: False]
  -X, --x-contours      show x contours [default: False]
  -Y, --y-contours      show y contours [default: False]
  -Z, --z-contours      show z contours [default: False]
  -F, --full-screen     show models in full-screen mode [default: False]
  -v, --verbose         verbose output
  -d PRIMARY_DESCRIPTOR, --primary-descriptor PRIMARY_DESCRIPTOR
                        set the primary descriptor to this value [valid
                        values: threeDVolume, contourList, meshList,
                        shapePrimitive]
  -f, --fill-holes      attempt to fill holes in mesh [default: False]
  -s, --smooth          attempt to smoothen mesh [default: False]
  -w, --wireframe       use wireframe representation [default: False]


Creating ROIs
================================================================
ROIs are 2D shapes (open or closed) that are overlaid on the original images to mark out regions of interest. 
A 3D volume can have ROIs drawn in any or all of the three planes: xy-plane (perpendicular to the z axis), xz, and yz. 
In order for ROIs to coincide with the image planes they are drawn at integral values of x, y and z. 
For example, a volume which extends in the x direction from x=37.6 to x=398.4 will have ROIs drawn in the yz plane at x values x=38.0, x=39.0, ..., x=398.0 (only at whole values of x). 
We refer to these as xContours and in the corresponding ROI file will have each contour with a fixed value of x. The same can be said of yContours and zContours.

.. code:: xml
	
	<?xml version="1.0" encoding="utf-8"?>
	<ROI>
	    <segment id="0">
	        <colour>
	            <red>0.282352954149246</red>
	            <green>1.</green>
	            <blue>0.639215707778931</blue>
	            <alpha>0.24</alpha>
	        </colour>
	        <xContours>
	            <contour>
	                <p id="0" x="831." y="4755.7890625" z="150."/>
	                <p id="1" x="831." y="4754.0390625" z="149.6956787109375"/>
	                <p id="2" x="831." y="4609.23681640625" z="127.560592651367188"/>
	                <p id="3" x="831." y="4479.2861328125" z="110.098487854003906"/>
	                <p id="4" x="831." y="4415.46240234375" z="101.988166809082031"/>
	                <p id="5" x="831." y="4413.79833984375" z="101.807365417480469"/>
	                <p id="6" x="831." y="4377.51806640625" z="99.088821411132812"/>
	                <p id="7" x="831." y="4290.314453125" z="89."/>
	                <p id="8" x="831." y="4217.826171875" z="89."/>
	                <p id="9" x="831." y="4117.93896484375" z="89."/>
					...
					                <p id="91" x="831." y="4847.10888671875" z="420.5286865234375"/>
	                <p id="92" x="831." y="4847.63134765625" z="417.99993896484375"/>
	                <p id="93" x="831." y="4836.73583984375" z="385.825164794921875"/>
	                <p id="94" x="831." y="4796.35302734375" z="267.99993896484375"/>
	                <p id="95" x="831." y="4791.98583984375" z="250.470703125"/>
	                <p id="96" x="831." y="4755.7890625" z="150."/>
	            </contour>
	            ...

We can create ROIs to be attached to an OMERO-stored image using the `createroi` subcommand from either an EMDB-SFF or AS-SFF file. 

.. code:: bash

	$ sff createroi -v file.sff
	Fri Jul 15 10:35:18 2016	Reading SFF file file.sff...
	Fri Jul 15 10:35:21 2016	Writing out ROI file to file.roi...
	$
	$  sff createroi -v sff/test_data/test_data.mod -o file.roi
	Fri Jul 15 10:35:48 2016	Intermediate conversion from sff/test_data/test_data.mod to tmp-1468575348.11.sff...
	Fri Jul 15 10:35:57 2016	Created XMLData object from schema <schema.emdb_sff.segmentation object at 0x1197d4cd0>
	Fri Jul 15 10:35:57 2016	Reading in IMOD file from sff/test_data/test_data.mod file
	Fri Jul 15 10:35:57 2016	Writing out XML to file tmp-1468575348.11.sff
	Fri Jul 15 10:36:01 2016	Writing out ROI file to file.roi...


Configurations (TBD)
================================================================
Most subcommands act transiently. However, subcommands which require persistent data must save configurations. 
For example, subcommands that require connections to an OMERO instance must at least store the host, port and username (not advisable to store passwords). 
Also, updating the schema needs a persistent record of the location and destination of the schema and schema API, respectively.

.. code:: bash

	$ sff config <config.name>=<config.value>

Example:

.. code:: bash

	$ sff config omero.host=localhost omero.port=4064 omero.user=test 


Working with OMERO
================================================================
Credentials
----------------------------------------------------------------
We need credentials to interact with the OMERO server. These should have been already created.

.. code:: bash

	$ sff list -H <omero.host> -p <omero.port> -U <omero.user> -P <omero.password> [list_options]

To with with an OMERO server, the following credentials are required. There are default values (problem). These may be overridden with stored configs or may be supplied at the shell.  

-H/--host
-p/--port
-U/--user
-P/--password


Listing Available Data
----------------------------------------------------------------

OMERO can store both 3D volumes (as z-stacks of images) and accompanying segmentations (as planar regions-of-interest, ROIs) using the following synopsis:

.. code:: bash

	$ sff list [options]

SFFTK can facilitate the following actions:

1. List all available images

.. code:: bash

	$ sff list --images
	# or 
	$ sff list -I
	********************************************************************************************************************************************
	                                                         *** AVAILABLE IMAGES ***                                                           
	********************************************************************************************************************************************
	Project        Dataset        ImageID ImageName                                    X     Y     Z   pX (μm)   pY (μm)   pZ (μm) #ROIs
	--------------------------------------------------------------------------------------------------------------------------------------------
	Test           ROI-Tests      467     cryotomo_rod1_01_front.rec                 512   150   512  0.001907  0.001907  0.001907     0
	Test           ROI-Tests      466     cryotomo_rod1_01_rightside.rec             512   150   512  0.001907  0.001907  0.001907     3
	Test           ROI-Tests      462     cryotomo_rod1_01_top.rec                   512   512   150  0.001907  0.001907  0.001907     1
	...
	Test           ROI-Tests      456     plastictomo_female_01_top.rec             1796  2038    71  0.002202  0.002202  0.002202     1
	Test           ROI-Tests      301     test2.am                                   862   971   200     15.58     15.58     15.58   282
	Test           test-data      2       emd_1080.map                               100   100   100   0.00027   0.00027   0.00027   249
	Test           test-data      1       mask.mrc                                    10    10    10    0.0001    0.0001    0.0001     0
	--------------------------------------------------------------------------------------------------------------------------------------------
	
A Note on Orienting Images 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Images stored on OMERO must be properly oriented (rotated) in order for ROIs to be meaningfully aligned.
The following rotations are performed for each orientation:

- Front images are associated with xContours i.e. the volume consists of images stacked along the x-axis and viewed down the x-axis at the origin.
- Rightside images belong to yContours
- Top images correspond to zContours 

2. List individual images

.. code:: bash

	$ sff list -I -i 467
	# or 
	$ sff list -I --image-id 467
	********************************************************************************************************************************************
	                                                         *** AVAILABLE IMAGES ***                                                           
	********************************************************************************************************************************************
	Project        Dataset        ImageID ImageName                                    X     Y     Z   pX (μm)   pY (μm)   pZ (μm) #ROIs
	--------------------------------------------------------------------------------------------------------------------------------------------
	Test           ROI-Tests      467     cryotomo_rod1_01_front.rec                 512   150   512  0.001907  0.001907  0.001907     0
	--------------------------------------------------------------------------------------------------------------------------------------------

3. List all available ROIs

.. code:: bash

	$ sff list -R
	# or 
	$ sff list --rois
	********************************************************************************************************************************************
	                                                          *** AVAILABLE ROIS ***                                                            
	********************************************************************************************************************************************
	ROI ID         Project        Dataset        ImageID ImageName                               PrimaryShape
	--------------------------------------------------------------------------------------------------------------------------------------------
	6310           Test           ROI-Tests      466     cryotomo_rod1_01_rightside.rec          <class 'omero.model.PolygonI'>
	6311           Test           ROI-Tests      466     cryotomo_rod1_01_rightside.rec          <class 'omero.model.PolygonI'>
	6312           Test           ROI-Tests      466     cryotomo_rod1_01_rightside.rec          <class 'omero.model.PolygonI'>
	6303           Test           ROI-Tests      462     cryotomo_rod1_01_top.rec                <class 'omero.model.PolygonI'>
	...
	6285           Test           test-data      2       emd_1080.map                            <class 'omero.model.PointI'>
	6286           Test           test-data      2       emd_1080.map                            <class 'omero.model.PolygonI'>
	6287           Test           test-data      2       emd_1080.map                            <class 'omero.model.PolylineI'>
	6289           Test           test-data      2       emd_1080.map                            <class 'omero.model.RectangleI'>
	--------------------------------------------------------------------------------------------------------------------------------------------
 
4. List all ROIs for an image

.. code:: bash

	$ sff list -R -i 466
	********************************************************************************************************************************************
	                                                          *** AVAILABLE ROIS ***                                                            
	********************************************************************************************************************************************
	ROI ID         Project        Dataset        ImageID ImageName                               PrimaryShape
	--------------------------------------------------------------------------------------------------------------------------------------------
	6310           Test           ROI-Tests      466     cryotomo_rod1_01_rightside.rec          <class 'omero.model.PolygonI'>
	6311           Test           ROI-Tests      466     cryotomo_rod1_01_rightside.rec          <class 'omero.model.PolygonI'>
	6312           Test           ROI-Tests      466     cryotomo_rod1_01_rightside.rec          <class 'omero.model.PolygonI'>
	--------------------------------------------------------------------------------------------------------------------------------------------


Loading and Modifying Data
----------------------------------------------------------------
1. Load ROIs and associate them with an image

ROIs should be associated to an image by image ID (see 4 above). xContours are associated with front images, yContours with rightside images and zContours with top images.
The `attachroi` subcommand has an option to specify the respective images for xContours (`-x` option), yContours (`-y`) and zContours (`-z`). 

.. code:: bash 
	
	$ sff attachroi -v -x 467 -y 466 -z 462 ~/Data/seg/IMOD/ehanssen/cryotomo_rod1_01.roi
	Fri Jul 15 16:27:33 2016	Reading ROIs from /Users/pkorir/Data/seg/IMOD/ehanssen/cryotomo_rod1_01.roi...
	Fri Jul 15 16:27:33 2016	Connecting to OMERO.server at localhost on port 4064 using user "root"
	Fri Jul 15 16:27:33 2016	Attempting to connect to the OMERO.server... SUCCESS!
	Fri Jul 15 16:27:34 2016	Checking whether y image of ID 466 exists... OK
	Fri Jul 15 16:27:34 2016	Parsing /Users/pkorir/Data/seg/IMOD/ehanssen/cryotomo_rod1_01.roi...
	Fri Jul 15 16:27:34 2016	Loading ROI data into OMERO object for y... OK
	Fri Jul 15 16:27:34 2016	Saving ROI to OMERO-server for y... OK
	Fri Jul 15 16:27:35 2016	Checking whether x image of ID 467 exists... OK
	Fri Jul 15 16:27:35 2016	Parsing /Users/pkorir/Data/seg/IMOD/ehanssen/cryotomo_rod1_01.roi...
	Fri Jul 15 16:27:35 2016	Loading ROI data into OMERO object for x... OK
	Fri Jul 15 16:27:35 2016	Saving ROI to OMERO-server for x... OK
	Fri Jul 15 16:27:36 2016	Checking whether z image of ID 462 exists... OK
	Fri Jul 15 16:27:36 2016	Parsing /Users/pkorir/Data/seg/IMOD/ehanssen/cryotomo_rod1_01.roi...
	Fri Jul 15 16:27:36 2016	Loading ROI data into OMERO object for z... OK
	Fri Jul 15 16:27:36 2016	Saving ROI to OMERO-server for z... OK
	Fri Jul 15 16:27:37 2016	Closing the connection...
	Fri Jul 15 16:27:37 2016	Deleting ROIHandler instance... SUCCESS!
	Fri Jul 15 16:27:37 2016	Connection closed.

2. Delete all ROIs associated with an image

.. code:: bash

	$ sff deleteroi -v -i 467
	Fri Jul 15 16:36:39 2016	Connecting to OMERO.server at localhost on port 4064 using user "root"
	Fri Jul 15 16:36:39 2016	Attempting to connect to the OMERO.server... SUCCESS!
	Fri Jul 15 16:36:40 2016	Deleting ROI 6405
	Fri Jul 15 16:36:40 2016	Deleting, please wait . . OK
	Fri Jul 15 16:36:41 2016	Deleting ROI 6408
	Fri Jul 15 16:36:41 2016	Deleting, please wait . . OK
	Fri Jul 15 16:36:43 2016	Deleting ROI 6411
	Fri Jul 15 16:36:43 2016	Deleting, please wait . . OK
	Fri Jul 15 16:36:44 2016	Deleting ROI 6414
	Fri Jul 15 16:36:44 2016	Deleting, please wait . . OK
	Fri Jul 15 16:36:45 2016	Closing the connection...
	Fri Jul 15 16:36:45 2016	Deleting ROIHandler instance... SUCCESS!
	Fri Jul 15 16:36:45 2016	Connection closed.

3. Delete a single ROI

.. code:: bash

	$ sff deleteroi -v -r 6404
	Fri Jul 15 16:37:50 2016	Connecting to OMERO.server at localhost on port 4064 using user "root"
	Fri Jul 15 16:37:50 2016	Attempting to connect to the OMERO.server... SUCCESS!
	Fri Jul 15 16:37:51 2016	Deleting ROI 6404
	Fri Jul 15 16:37:51 2016	Deleting, please wait . . OK
	Fri Jul 15 16:37:52 2016	Closing the connection...
	Fri Jul 15 16:37:52 2016	Deleting ROIHandler instance... SUCCESS!
	Fri Jul 15 16:37:52 2016	Connection closed.


Listing Supported AS-SFFs
================================================================

TBD

Annotating Segmentations
================================================================

TBD

----------------------------------------------------------------
Using the SFFTK API
----------------------------------------------------------------

SFFTK provides an API to its functionality for reusability. 

1. Read EMDB-SFF files

.. code:: python
	:number-lines:
	
	from sfftk.readers.sffreader import get_data
	
	# get_data takes an EMDB-SFF file name
	descriptor, segments, colours, alphas = get_data(sff_fn)
	
	# descriptor is one of 'threeDVolume', 'contourList', 'meshList', 'shapePrimitive'
	# segments is a dictionary of contours/meshes
	# colours is a list with as many RGB float-triples as segments
	# alphas is a list with as many floats (in the range 0-1 inclusive) as segments
	
2. Convert supported AS-SFF files to EMDB-SFF

.. code:: python
	:number-lines:
	
	import sys
	from sfftk.converters.seg_to_sff import XMLData
	from sfftk.schema import emdb_sff as XMLSchema
	    
	xd = XMLData(XMLSchema, verbose=args.verbose) 
	
	# convert an IMOD file
	xd.read_imod(mod_file)
	xd.write(sys.stdout)
	
	
3. Generate meshes derived from EMDB-SFF data

.. code:: python
	:number-lines:
	
	import vtk
	from sfftk.readers.sffreader import get_data
	from sfftk.meshtools import get_meshes
	
	# e.g sff_file = sys.argv[1] (import sys first)
	descriptor, segments, colours, alphas = get_data(sff_file)
	
	# a list of vtkPolyData objects
	meshes = get_meshes(segments, colours, alphas, form=descriptor)
	
	# define the renderer        
	ren = vtk.vtkRenderer()
	ren.SetBackground(0.1, 0.2, 0.4)
	
	# populate the renderer with the meshes
	for mesh in meshes:
	    ren = mesh.render(ren)
	
	# render window
	renWin = vtk.vtkRenderWindow()
	renWin.AddRenderer(ren)
	
	# render interactor
	iren = vtk.vtkRenderWindowInteractor()
	iren.SetRenderWindow(renWin)
	iren.SetInteractorStyle(vtk.vtkInteractorStyleTrackballCamera())
	
	iren.Initialize()
	iren.Start()

4. Convert EMDB-SFF to supported AS-SFF files

TBD

----------------------------------------------------------------
Extending SFFTK
----------------------------------------------------------------

TBD

----------------------------------------------------------------
Developers
----------------------------------------------------------------

Paul K. Korir, PhD

----------------------------------------------------------------
Contact
----------------------------------------------------------------

For questions, comments and/or bug reports, please write to:

pkorir [THE @ SIGN] ebi [FIRST DOT] ac [ANOTHER DOT] uk
paul.korir [THE @ SIGN] gmail [ONLY ONE DOT] com
