• Introduction
• Key Concepts
• The Main SurfaceEditor Window
• The Spline/Objectmap Control
• 4D Surface Editing
• Delineating a surface: a step-by-step recipe

Introduction

The surfaceeditor applications provide powerful functionality for interactive segmentation of different anatomical structures. There are two distinct modes of operation. The surface tools allow the interactive outlining of surfaces which are represented as "stacks" of curves, one curve per image slice. The objectmap tools allow for the labeling of voxels groups as belonging to different types of tissue and/or structures.

The software traces its roots to the original Surface Editor developed in 1995-1998 specifically for the segmentation of the bounding surfaces ofthe the left ventricle of the heart (endocardium and epicardium) using then Silicon Graphics exclusive software libraries (Open Inventor). It has since been used for a number of applications including segmentation of neuroanatomical structure, region of interest definition of functional neuroimaging studies, cardiac segmentation, abdominal structure segmentation -- including fat quantification -- etc. The rest of this manual describes the core functionality of the surface editor tools. There are two related tools: the surface editor tool which includes both surface and objectmap tools for 3D images, and the 4D surfaceeditor tool which has only the surface editing tools but includes additional functionality for cardiac image analysis. Most of this manual is concerned with the 3D version of the tools. The last section addresses functionality specific to the 4D version.

All pictures in this manual are for the Windows version of Surface Editor, although the user interface is identical (other than for cosmetic differences) to the other versions. Much like the rest of BioImage Suite there are versions available also for Linux, Mac OS X 10.4+, Solaris 10, IRIX 6.5 and FreeBSD 6.0.

Key Concepts

The SurfaceEditor tools operate in an edit in 2D display in 3D paradigm. Editing is performed in a slice-by-slice mode using the slice editor tool (Figure above right) and the 3D overall picture (i.e. 3D surface/3D objectmap) is displayed in the main Surface Editor window (Figure above left). The main window is updated either manually using the "Update" button in the editor or automatically when the slice is changed if the "Auto Upd" checkbutton is in the "on" state. The Surface Editor window consists of the same core BioImage Suite orthogonal viewer described in the viewer pages, but has additional functionality related to the interactive segmentation tasks that will be described next.

Note that editing in the slice editor is performed on the the XY slice of the underlying image, i.e. the axial slice if the image was acquired in an axial orientation, or the coronal slice if it was acquired in a coronal orientation. If a different 2D slice orientation is preferred, the image will need to be re-oriented or resliced (if oblique slices are required) using the Image Processing tool.

The Surface Editor has the ability to create surfaces from binary images. In this way segmentations generated using other tools such as itk-SNAP, which are typically stored as binary images, can be imported into the Surface Editor for more detailed manipulation. See the Creating Surfaces from Binary Images for more details.

The Main Surface Editor Window

The main Surface Editor window is shown to left in Figure 2. There are 5 parts to the user interface.

A: The Menu Bar:

This contains the usual options of loading/saving the underlying images, image processing for smoothing, reslicing/cropping images, etc. In addition, the objectmap menu has options specific to the surface editor. Please note that the File Menu only contains operations for loading/saving the underlying image and not the surfaces/objectmaps. Separate functionality is provided to do this.

B. The Tab Selection:

This selects which of three sets of controls appears on the right most pane of the window. These are the Image tab which has the usual orthogonal viewer options for manipulating how the image is displayed (e.g. single slice, orthogonal slices, volume rendering etc.), the Surface tab which contains most core surface operation and the Surface+ tab contains additional surface operations. The "Surface" tab (shown) is divided into two parts, namely:

C: Current-stack properties:

Options for manipulating the current surface (how to display this e.g. open surface and fast surface) and the all-important definition of surface extent, i.e. on which slices does the structure appear. This is the first thing that needs to be set when creating a new surface.

D: Current-Stack Options:

These are options for resampling the current stack, flipping the x-axis, changing the color and loading and saving. On the bottom of the window

E: Editor Slice:

contains functionality for selecting the current slice on which the spline/objectmap editor operates and a "button" for opening the Editor.

---

The "Image" "Surface" and "Surface+" tabs in more detail

Much of the functionality of the surface editor is bound within the notebook tabs in the rightmost pane. The Image tab is identical to the standard orthogonal viewer and the reader is directed to the "viewers" documentation on the Bioimage Suite web-page for more information.
The Surface tab contains most of the functionality needed for manipulating surfaces in a global sense. The surface editor can be used to outline a number of surfaces at a time (typically 4; but can be set to as high as 50 using the user preferences editor). These can all be shown or hid using the Show All/Hide All buttons in the "Global Properties" frame at the top.

The "Spline Stack Properties" frames has the following options. First the "Current Stack" option menu enables the selection of the current surface (e.g. 1 to 4) for editing/manipulation. All subsequent operations operate on this surface. The "Draw Mode" option has mostly cosmetic effect. Typically the surfaces are cylinder-like with open tops and bottoms, i.e. "Open Surface". The other options in this drop-menu enable the visual closing of either the top or the bottom or both ends of the surface. This adds computational cost to the surface updating and is best used sparingly. The "Display Mode" option has three choices "Fast Surface" which produces a fast but relatively low quality surface rendering, "Full Surface" which produces a slower but higher quality rendering and "Hide" which completely hides the surface. The "Bot Trim:" and "Top Trim:" controls enable the user to set the critical bottom slice and top slice extends of the surface. If the structure of interest appears only in e.g. slices 24-32, then bottom trim should be set to 24 and top trim to 32. Failure to do this will result in either additional or missing curves in the surface with erroneous results.

The "Operations on Current Stack Frame" contains options for global manipulation of the surface. The first line consists of a row of buttons that resample the surface. For example, "x0.5" resamples all curves to have half the number of control points (as discussed later each curve is parameterized as a b-spline), and similarly x0.8, x.1.25 and x2. The "s2" and "s5" buttons re-parameterize the surface to have different degrees of smoothness and result in non-uniform b-spline. Finally the "FlipX" button performs a left-right mirroring of the surface which is useful in some applications.  The "Color" button brings up a dialog box for selecting the color of the surface. Clicking the "Cylinder" button resets the surface to a cylinder of default location and size. The "Volume" button computes the volume of the surface by adding up the areas of the individual curves and multiplying by the image slice thickness. The VOI Prop tab performs VOI (volume-of-interest) analysis of the area inside the surface. The two thresholds can be set to restrict the analysis to only voxels in that range. The "Load" and "Save" buttons enable the loading and saving of surfaces in the custom ".sur" format. Finally the Copy and Paste options enable the copying of the surface to the clipboards and its pasting (presumably to another surface).

The Surface+ tab contains some additional functionality. The "Current Stack" drop menu mirrors the one in the "Spline Stack Properties" frame and can be used to set the current surface. The Load Set and Save Set options can be used to load/save all the surfaces in the editor in a composite ".multisur" format file. (Note that the .multisur file simply contains pointers to an additional number of individual surface files. E.g. if one saved a group of 4 surfaces to mysurfaces.multisur, the following files will be created: mysurfaces.multisur, musurfaces.multisur_1.sur, musurfaces.multisur_2.sur, musurfaces.multisur_3.sur and musurfaces.multisur_4.sur). The Autosave and Backup checkbuttons if enabled turn on the autosave features of the software (leave them on!). The Shift and Scale button brings up a legacy control for re-scaling surfaces generated with older versions of Surface Editor which did not store proper slice thickness information. The Cylinders button brings up a control for creating a custom cylinder by specifying both radii, slice extents, curve centroids and number of control points. The Export buttons brings up a control for exporting surfaces either in the custom ".tstack" format or in the more common ".vtk" format (as defined by vtkPolyDataReader/Writer). The user can specify the sampling of the spline surfaces in mm as well as the format and a number of smoothing iterations to be applied prior to exporting.

Creating Surfaces from Binary Images

A recent addition is the Extract From Binary Image button which allows for the creating of a surface (.sur) from a binary pre-segmented image. In this way segmentations generated using other tools such as itk-SNAP, which are typically stored as binary images, can be imported into the Surface Editor for more detailed manipulation. Pressing the Extract From Binary Image button, brings up a helper dialog (shown in Figure 4) which allows the user to specify four options: (i) Surface Smoothing: this determines how many control points will be used to parameterize the extracted surface, (ii) Image smoothing: which is the standard deviation of the kernel that is used to smooth the binary image prior to surface extraction and (iii) and (iv) bottom and top slice to limit the extraction range. The extraction routine automatically performs connectivity analysis on the underlying image to eliminate small holes etc. and will return only those consecutive slices for which the surface extraction has succeeded.

Objectmaps and the Objectmap Menu

In addition to surfaces whose manipulation is controlled using the "Surface" and "Surface+" tabs described above, the Surface Editor can create and modify another class of structure called "objectmaps". Objectmaps are essentially images (and are saved in the standard Mayo/Analyze .hdr/.img image file format) in which each voxel has an object label (e.g. 1-10) signifying its membership in a class/structure. While surface editing has many advantages for outlining smooth anatomical structures, objectmap descriptions are often needed for applications such as tissue labeling (e.g. muscle vs fat). By default once a new image is loaded (using the File Menu), an empty objectmap of the same dimensions is created. This can be loaded/saved using the "Load Obejctmap" and "Save Objectmap" options under the objectmap menu. A new empty objectmap can be created using the "New Objectmap" button. The volume of different structures in the objectmap, as well as a VOI analysis of the underlying image, is obtained using the "VOI Volume/Stats Computation" option.  The objectmap can be manipulated using the mathematical morphology tools (under Segmentation menu, Math_Morphology) by copying it over using the "Store in Morphology Mask" option, if this is desired.

Objectmaps can be created either from the current displayed image in the viewer using the "Create From Data/Grab from Current Displayed Image" button or from the surfaces (i.e. convert surfaces to objectmaps) using the "Surface Objectmap Tool" control shown above that is accessed using the "Create from Data/Create from Surfaces" option. The default setup for this control (using the "Create!") button enables the user to specify which surfaces to use, and value inside each surface as well as the background. The Distance Map function creates a "2D Distance Map" based on the surfaces which is useful in some applications. The "Mask" button creates a mask where only the parts of the image inside the selected surface are left, the rest are blanked. The opposite (i.e. only parts of the image outside the selected surfaces) is accomplished using the "Inverse" button.

The (Objectmap | Fat Objectmap Operations) menu choice brings up two different customized options for application specific needs to be described in more detail in the next version of the manual.

The Spline/Objectmap Control

The local editing of surfaces/objectmaps is done within the Spline/Objectmap tool shown at right in Figure 6. This is accessed using the "Objectmap/Spline Editor" button in the bottom left corner of the main SurfaceEditor window. This extremely powerful control enables the manipulation of individual curves by moving b-spline control points, the drawing of new curves by simply clicking points which are then converted to b-splines for further editing, as well as the manipulation of objectmaps using paint-brush like operations.

Editing Splines

In the viewer portion (i.e. the left part of the image in Figure 6), we show a single spline. If the viewer "edit" mode is turned on, manual mode off and the mode in the Fill Control is set to "Spline" (these are all the default settings), then the spline can be manipulated by moving individual control points by clicking and dragging them using the left mouse button. Holding down the shift-key while and left mouse button allows you to move the entire spline. Clicking on the curve with the left mouse button while holding down the "Control" key adds a new control point half way between the closest two control points. In this way, a fully customizable spline curve can be generated on each slice of an image, thus creating a user-defined surface.

The Bottom Row -- selecting slice and objectmap transparency.

Given the absence of a menu in this control, most core functionality is present in the bottom row "buttonbar" shown above. The "Slice" control determines which slice is currently being edited. This is mirrored in the slice control in the bottom row of the main Surface Editor window. The "Mask" control sets the relative opacity of the objectmap with respect to the image. If set to 0 the objectmap is completely transparent (effectively hidden) whereas if set to 100 the objectmap becomes completely opaque and hides the underlying image. The next label "200(56,83,62)" shows information about the last clicked voxel in the image with, in this case, image coordinates (56,83,62) and intensity=200.00. The "SaveTiff" button enables the saving of the contens of the viewer (i.e. a screen grab) as a tiff/jpeg file (depending on the suffix). The "Zoom" arrow buttons enabling zooming in and out (this is also accomplished by dragging with the middle mouse button). The "ColorMap" button brings up the standard colormap editor (as described in the viewers documentation) and the "Reset" button resets the viewer to default zoom/location levels. The "Interp" check button turns on/off standard anti-aliasing interpolation in the image display.

The Spline Controls &ndash manipulating splines

The "Spline Control" shown on the left has functionality for manipulating individual curves. We again note that in the SurfaceEditor each surface is made up of a single curve on each XY slice (within the bounds set by the bottom trim and top trim controls). These curves are manipulated using the spline control shown here.

The "Global Properties" frame has options for turning "Edit Mode" on and off, enabling and disabling "Manual Mode" – described in detail below – showing and hiding the actual control points and showing and hiding all splines. The (-) button is used to flip the relative depth positioning of the splines and the image. It is primarily needed when tracing on Sagital images. If the splines are not visible a single click of the (-) button should fix the problem,

Individual spline properties are set using the options in the "Spline Properties" frame. The "Current Spline:" drop menu selects the current spline to edit/manipulate. Spline 1 is the spline corresponding to "Surface 1" in the main Surface Editor window, "Spline 2" is part of "Surface 2" etc. The spline can be hidden by unchecking the display button. The "Size:" optionmenu controls the size of the control points.

The "Operations On Current Spline" frame mirrors functionality in the "Operations on Current Stack" frame in the "Surface" tab of the main surface editor. The "Snake" and "Update Snake" options enable the use of interactive snake-like segmentation methods to be described in more detail in the next version of this manual.

The Copy/Paste options are the real workhorse options of this program. Given the general smoothness of most anatomical structures, the shape of a surface on slice 51 is very similar to that in slice 50. Hence the curve can be copied from slice 50, the slice number incremented to 51 and then the curve pasted giving an excellent initial position for further editing. The "Undo" button reverts to the version of the curve that is currently stored in the surface editor (i.e. prior to editing). Past flip performs an x-flip prior to pasting.

The Update and "Auto Upd" buttons control the communication between the spline/objectmap editor and the surface editor. On pressing update the versions of the curves and the objectmap slice (see below) as edited in the spline/objectmap editor are copied over to the main SurfaceEditor, and the 3D display in the Surface Editor is updated to reflect this. The "Auto Upd" checkbutton enables automatic updating upon slice change. The "Update" button automatically copies the current spline and places it in the clipboard so that it can be pasted directly.

Manual Mode: In certain cases the initial shape to be traced is very complex and cannot easily be captured by moving the control points of an elliptical spline to fit this. The use of   "Manual mode" enables the use of the more traditional (and less flexible for subsequent editing purposes) method of clicking individual points on the boundary to create a polygonal curve which is then converted to a b-spline for editing. To use manual mode first enable this by selecting the "Manual Mode" checkbutton towards the top right corner of the "Global Properties" frame.

Next, click individual points using the left mouse button to form a polygon (there is no need to close this, the fitting process automatically does this for you -- fact the polygon is best left open), as shown in the case of the thalamus (see figure on the left.) Once the polygon is mostly closed, turn off "Manual Mode" by clicking on the "Manual mode" checkbutton as before. Once this is done a pop-up dialog asks whether you want to replace the current spline with one fitted to the current set of manually clicked points. If you answer "no", the current manually clicked points are deleted. If you answer "yes" a spline is fitted to the polygonal points which can then be further edited.

The Fill Controls - manipulating objectmaps

The "fill control" which is shown on the left (and is located in the bottom right part of the Spline Objectmap Editor), enables the manipulation of object- maps by appropriately coloring in image voxels (by using the left mouse button). To switch from curve editing mode to image painting/filling mode, use the "Mode" drop menu (shown in the picture below) to switch from "Spline" to one of "Brush-1" to "Brush-9". The number after the word "Brush", e.g. 1 or 9, refers to the size of the paintbrush used to paint the image.

In the object map the image can be painted with values 1-9. The value "0" is used as an eraser, i.e. use this to remove voxels from given structures. The objects 1-9 are shown in the image in the colors under the respective buttons (e.g. object 1 will be shown in red. This can be changed using the "Edit Color" button. The resulting color selections can be loaded and saved using the "Load" and "Save" buttons on the bottom row of the "Fill Control" frame. The "Restore" button can be used to reset the color selection to its default settings.

While simply selecting a paintbrush size and color can be used to edit the objectmap, this control provides additional functionality to speed up the process. The two text boxes shown in white with values 0.0000 and 300.0000 (which can be edited naturally) contain upper and lower thresholds which can be used to constrain the painting process.

To paint in only voxels within intensity range 200-300, set the thresholds to 200 and 300 respectively and enable the "Use Threshld" checkbutton. If we wish to only color in voxels in this range but which are also connected to the location under the mouse we can also enable the "Connect" button. The use of the thresholds and connect button together with a large size paint brush can speed up manual segmentation significantly. Finally the "Use Mask" checkbutton switches the thresholding/connecting operations to use the objectmap itself as opposed to the underlying image -- this can be useful for maintaining boundaries between adjacent objects.

The "Fill In" and "Fill Out" buttons use the threshold (but not connect) settings together with the current spline to paint the image. Fill In will fill the inside of a spline with a certain value, whereas "Fill Out" will fill the region outside the spline with the selected value. If threshold settings are set, these are also used.

4D Surface Editing Functionality -- Cardiac Image Analysis

The 4D Surface Editor package (which predates the 3D version) is designed specifically for cardiac image analysis. As such it has additional functionality for this purpose, and it also provides graphical user interfaces to parts of the shape-based deformation analysis methods [Papademetris et al. TMI 2002].


The rest of this description assumes familiarity with the 3D editor and simply focuses on the above 5 points.

Movie Controls (Cine Mode)

The movie control (shown in Figure 12, left) is accessed using the "Movie Control" button in the "Image" tab. It has functionality for playing cine-loops of the contents of the viewer (e.g. image, surfaces, tstacks, strain maps etc.). It has two distinct play modes "Complete" which is equivalent to simply incrementing the frame and "Fast" which uses cached versions all frames prior to playing to achieve higher performance at the loss of interaction duing movie mode. The caching is perfomed by clicking the "Prepare" button.

The "Multi" and "Segment" Tabs

The "Multi" tab is closely related to the "Surface+" tab in 3D Surface Editor. The key difference is that the Load All/Save All functionality referers to the loading/saving  of temporal sequences of surfaces instead of surface sets. In addition the "Export" button provides access to a more advanced export facility, for exporting temporal sequences of surfaces.

The "Segment"  tab provides facilities for spatial and temporal interpolation. In the "Spatial" tab the user specifies the top and the bottom slice and curves are filled inbetween.  In the temporal interpolation, the assumption is that the user will first trace the ED and ES surfaces and then the interpolation algorithm can be used to generate good candidate segmentations by simple interpolations.

The "Auto" and "Batch" modes refer to automated segmentation using the snake algorith, This is work in progress and should be used sparingly.

Changes in the menu and controls

The 4D viewer has a number of different menu choices and associated control windows from the those of the 3D viewer.

The Menu

The Cardiac menu shows how to access the "tstack" control and the "Abaqus post control". The tstack control shown below has functionality for manipulating tstack (Triangulated Stack files). These are created from the surface .sur files and represent an explicit surface representation (in terms of points and triangles) as opposed to the implicit cubic polynomial representation of surfaces stored in the .sur files.

The Tstack Control

The .tstack files can be generated using the "Export" button in the "Multi" tab of the main 4D surface editor window. These generates a series of files which can be Loaded/Saved using the Load/Save options in the T-stcack control. Much of the rest of the functionality of the T-stack control mirrors closely that for the SplineStack control described earlier. One key difference is the presence of the "Color Mode" menu.  This selects the function used to create a color scale for the surface. These measures are based on the different combinations of the two principal curvatures (e.g. Gaussian, Mean, 1st principal, 2nd principal, bending energy and shape index). These colorscales are aids to visualization.

Computing Curvatures

The curvatures can be computed using the "Computing Curvatures" control (shown in the top right) which is accessed using the "Curvatures" button. There are two paramters to be set. The "scale" parameter controls the size of the neighborhood used for curvature computation and the "smooth iterations" the number of smoothing iterations applied prior to curvature computation.
Once curvatures are computed the can be used as an input to the shape tracking algorithm. This is done in the "Shape-based ... tracking control" which is accessed from the "Shape-Tracking" menu on the T-stack control. The options are involved are too technical for a brief overview of this kind. The same applies to the solid control which creates a finite-element mesh between  selected endo-cardial and epi-cardial surfaces. The functionality captures in these controls (curvatures, shape-based tracking, create solid) represents most of the steps in the shape-based tracking deformation algortithm of [Papademetris et al , TMI 2002]. The one step missing is the finite element analysis step itself which requires the presence of the Abaqus Finite Element package.

The output of the Abaqus FEM package can be loaded into the "Abaqus Post Control" to visualize and quantify regional myocardial strains. More information on this will be included in the next version of this manual.

The Spline Editor Control (4D Surface Editing)

Fig. 16   The Spline Editor is similar in spirit to the more advanced Spline/Objectmap editor used in the 3D Surface Editor. It does not have any "objectmap" functionality. The one additional feature is the "Ghost Display" option which enable the display of the previous and/or next curve(s) in the temporal sequence (the ghosts of the past and the future) to enable the easier construction of smooth temporal sequences.

---

Delineating a surface: a step-by-step recipe.

---

About this manual.