Differences between revisions 22 and 190 (spanning 168 versions)
Revision 22 as of 2005-08-26 13:57:09
Size: 15146
Comment:
Revision 190 as of 2023-09-13 15:46:41
Size: 3992
Editor: JacksonNolan
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#pragma section-numbers on #acl LcnGroup:read,write,delete,revert All:read
Line 3: Line 3:
[[Navigation(children)]] = FreeSurfer Dev Guide =
Line 5: Line 5:
'''Index'''  * Visit the BuildGuide for instructions on building and installing freesurfer manually.
 * Visit the GitHub page for an introduction to the github workflow.
 * Visit the GitAnnex page for detailed instructions on using git annex for storing and retrieving large data files in the repository.
Line 7: Line 9:
[[TableOfContents]] == File Size Limitations ==
Line 9: Line 11:
=== CVS Checkout === Any files '''larger than 50MB''' should be stored in the GitAnnex, following the hyperlinked instructions, and properly linked to your utility.
Line 11: Line 13:
You can checkout the FreeSurfer source code from the NMR center using local CVS access or remotely by using SSH as the CVS remote connection method. == Adding a New C Program ==
Line 13: Line 15:
==== Local CVS Access ====

The CVS repository is /space/repo/1/dev. Use this as your CVSROOT. You can either set it as an environment variable:
If you'd like to add a new program to the tree, you should create a new subdirectory with the title of your tool. As an example, let's create a new c++ program called `mri_process`. First, we'll create a top-level subdirectory that contains our new c++ file and an empty `CMakeLists.txt` file:
Line 18: Line 18:
setenv CVSROOT /space/repo/1/dev freesurfer/
    mri_process/
        CMakeLists.txt
        mri_process.cpp
Line 21: Line 24:
or specify it in the checkout command with the -d option. Note that the CVS root is cached in a CVS checkout directory, so if you choose to use the -d method, you will only have to do it once during your first checkout.

Check out the code with the CVS checkout command. The archive name is dev.
In order to configure our new code, we should add the following to the empty `CMakeLists.txt` file.
Line 26: Line 27:
cvs checkout -P dev project(mri_process)

include_directories(${FS_INCLUDE_DIRS})

add_executable(mri_process mri_process.cpp)
target_link_libraries(mri_process utils)

install(TARGETS mri_process DESTINATION bin)
Line 29: Line 37:
or This will compile `mri_process.cpp`, link it against the `utils` freesurfer library, and copy the executable to the `$FREESURFER_HOME/bin` directory during install. To include this subdirectory in the main freesurfer build, make sure to modify the top-level `CMakeLists.txt` by adding `mri_process` to the long list of included directories at the bottom of the file. Now, after reconfiguring your build, you can run `make` in the `mri_process` directory of your build tree to successfully compile the new program. If you're having trouble configuring and building freesurfer, be sure to visit the BuildGuide for step-by-step instructions.

== Adding a New Python Program ==

Adding a new python program is similar to adding a new c++ program, with an additional step of creating a bash wrapper to be installed in `$FREESURFER_HOME/bin`.
 * Note: Make sure all paths in the Bash wrapper that reference files in the FreeSurfer file tree are defined relative to `$FREESURFER_HOME`, to ensure they are found regardless of the install location

As an example, let’s add a program called `mri_process` to the FreeSurfer tree, where the main functionality is in a python script called `mri_process.py`, and has a dependency on a model called `process_model.h5`.

First, we’ll create a directory containing the scripts, dependencies, and `CMakeLists.txt`:
Line 32: Line 49:
cvs -d /space/repo/1/dev checkout -P dev freesurfer/
 mri_process/
  CMakeLists.txt
  mri_process.sh # shell wrapper to install in bin/
  process_model.h5 # model dependency stored in the annex
  python/ # dir containing the python code
   mri_process.py # python script called by shell wrapper
Line 35: Line 58:
This will copy the entire archive to your directory, creating a directory called dev/. The -P option will purge the old directories that have been removed from the CVS repository. Now we’ll need to populate the empty `CMakeLists.txt` file. We need to achieve the following in this file:
 1. Give the utility a project name
 2. Link the python scripts to the location they are referenced by the Bash wrapper
 3. Install the wrapper script into `$FREESURFER_HOME/bin`
 4. Link any models to the proper location to be accessible to the python code
Line 37: Line 64:
Generally it is handy to define an environment variable called FSDEV which is set to your Freesurfer development directory. Be sure also to define your FREESURFER_HOME environment variable set to the intended Freesurfer installation directory. An example `CMakeLists.txt`:
{{{
project(mri_process)
Line 39: Line 68:
==== Remote CVS Access ==== install_symlinks(python/mri_process.py TYPE files DESTINATION python/scripts)
Line 41: Line 70:
Tell CVS to use SSH to access the archive by setting the following environment variable: install_configured(mri_process.sh DESTINATION bin)
Line 43: Line 72:
{{{
setenv CVS_RSH ssh
install_symlinks(process_model.h5 TYPE files DESTINATION models)
Line 47: Line 75:
Use the following string as your CVS root:

{{{
setenv CVSROOT :ext:USER@MACHINE.nmr.mgh.harvard.edu:/space/repo/1/dev
}}}

Where USER is your username and MACHINE is one of the NMR machines visible to the outside, i.e. gate, entry, or door. Then use the CVS commands normally.

Note that using this method makes an SSH connection for every CVS command, and you will be required to enter your password every time. You may want to look into a utility to automatically authenticate SSH connections, such as SSH agent. See:

http://mah.everybody.org/docs/ssh

When working in a CVS checkout directory, cvs knows where the CVSROOT is for that directory. However, it doesn't know what CVS_RSH method to use, so if you are grabbing files remotely, you may want to set this in your login file.

=== Running configure ===

Prior to running the configure script, the following is necessary to create the platform specific tools required by configure (execute in the dev/ directory):
{{{
  ./setup_configure
}}}
This script runs a set of commands (aclocal, libtoolize, automake, autoconf) that creates the platform specific files for configure and puts them in the 'dev/config' directory.

Now you need to configure your building paramters for your machine:
{{{
  ./configure
}}}
This runs the configure script which checks a bunch of local settings and libraries that are needed. You will probably need to supply some extra options:

{{{--prefix=~/work/freesurfer}}}

The prefix option determines where your build will be installed when you use the 'make install' command. This is your private copy of FreeSurfer, not a release for others, so put it in your own directory space.

{{{--with-mni-dir=/usr/pubsw/packages/mni/current --with-tcl-dir=/usr/pubsw/packages/qt/current}}}

These options specify where certain packages are. These are often
needed so configure can find libraries installed in /usr/pubsw. Note
that you should have /usr/pubsw mounted on your machine. This is
normal for NMR computers.

Freesurfer builds against the following set of open-sourced libraries,
thus, these must be installed prior to building:

  GSL
  JPEG
  TIFF
  Tcl 8.4
  Tk 8.4
  Tix 8.4
  BLT
  GLut
  Qt 3.3.4

Freesurfer also builds against certain packages of the MINC toolkit.
A special bundle of the source code for the packages require by
Freesurfer has been put together and is available on the download site:

ftp://surfer.nmr.mgh.harvard.edu/pub/dist/

Look for mni.srcbuild.tar.gz, and mni.srcbuild.README.txt describes the
build process.
{{{
  CFLAGS=-I/sw/include
  CXXFLAGS=-I/sw/include
  LDFLAGS=-L/sw/lib
}}}
The above are needed on Macs on which software is installed in /sw by
fink or FinkCommander. Note that you can also use these options to
specify other compile-line flags. Note that these options aren't
preceded by dashes.

{{{--with-tixwish=/usr/sw/bin/tixwish8.1.8.4}}}

This one is also often needed for Macs.

A sample configure line for the Mac:
{{{
  ./configure --prefix=~/home/freesurfer \
        CFLAGS=-I/sw/include \
        CXXFLAGS=-I/sw/include \
        LDFLAGS=-L/sw/lib \
        --with-mni-dir=/usr/local/mni \
        --with-tixwish=/usr/sw/bin/tixwish8.1.8.4
}}}

=== Building ===

You can now run 'make', edit and build code, etc. The first thing you should do is 'make' to build libraries like libutils. (Note that all libs will be built to your local checkout directory.) To just build libutils, cd into utils/ and run make.

You can now compile individual programs or all of them at once. To compile all of them, just run 'make' from dev/. Binaries will automatically be placed in their individual subdirectories.

However you may just want to compile one of them at a time. To do so, cd to the directory of the program you want and use 'make' to compile it:
{{{
  cd mri_info
  make
}}}
This creates mri_info in the mri_info/ directory.

==== Adding a new binary to the tree ====

Assuming that you have a source file {{{MYPROG.c}}} that compiles into MYPROG and want to add it to the FreeSurfer tree:

1) Make the directory in dev and copy the source file there. Name the directory MYPROG and the source file {{{MYPROG.c}}}.

{{{
mkdir dev/MYPROG
cp MYPROG.c dev/MYPROG
}}}

2) Tell the autotools to build your program when you type {{{make}}} from the top dir.

a) Modify {{{dev/configure.in}}} to add {{{MYPROG/Makefile}}} to the list at the end of the file in the definition of {{{AC_OUTPUT}}}. Be sure to add a backslash at the end of line:

{{{
AC_OUTPUT( \
... other files ...
MYPROG/Makefile \
)
}}}

b) Modify {{{dev/Makefile.am}}} to add {{{MYPROG}}} to the {{{SUBDIRS}}} definition. (You can also alternatively it to the end of {{{MRISUBDIRS}}} or {{{MRISSUBDIRS}}} if more appropriate.)

{{{
SUBDIRS= ... other directories ... MYPROG
}}}

c) Copy {{{dev/dummy/Makefile.am}}} into {{{MYPROG/}}} and customize it, replacing 'dummy' with 'MYPROG'. Be sure to change:

{{{
bin_PROGRAMS = MYPROG
}}}

d) Copy in the additional testing file {{{dev/dummy/myown.c}}}. You can customize it for your test program later.

3) Run {{{automake}}} from {{{dev/}}}. You should get no errors. If you do, make sure you followed the above instructions properly. Also try the AutoconfTroubleshooting page. Verify that this stepped work by checking if {{{MYPROG/Makefile.in}}} was created.

4) Run {{{autoconf}}} to generate a new {{{configure}}} script that now includes your new MYPROG directory.

5) Run {{{./configure}}} with the parameters you previously used. To check these out, run {{{head config.log}}} from {{{dev/}}}. The output should include the {{{./configure}}} line you used. Copy it, but leave out the {{{--no-create --no-recursion}}} options if present.

{{{
[dev/]$ head config.log
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by Freesurfer configure 0.1, which was
generated by GNU Autoconf 2.57. Invocation command line was

  $ ./configure --with-mni-dir=/usr/pubsw/packages/mni/current --prefix=/home/kteich/freesurfer/dev --no-create --no-recursion

## --------- ##
## Platform. ##
[dev/]$ ./configure --with-mni-dir=/usr/pubsw/packages/mni/current --prefix=/home/kteich/freesurfer/dev
}}}

Note: Do not just copy this example, use what's in your own {{{config.log}}} file!

6) Run {{{make}}} and verify that your binary program {{{MYPROG/MYPROG}}} was created.

7) Check in your changes.

{{{
[dev/] cvs ci -m "Added MYPROG" configure configure.in Makefile.am Makefile.in
[dev/] cvs add MYPROG
[dev/] cd MYPROG
[MYPROG/] cvs add Makefile.am Makefile.in MYPROG.c myown.c
[MYPROG/] cvs commit -m "First checkin." Makefile.am Makefile.in MYPROG.c myown.c
}}}

=== Installing ===

To install all binaries and support files into your private FreeSurfer, use 'make install' from the toplevel dev/ directory. This will make a directory called freesurfer/ in the directory specified by the --prefix option to configure, above. Note that if you do not specify this location, it will try to install to /usr/local, which you probably don't have permission to do. Even if you do, i.e. you are
installing on a laptop, it's generally better to specify a prefix of /usr/local/freesurfer to keep everything in the same place.

Note that you can also run 'make release'. 'make install' makes and installs the NMR internal version of FreeSurfer, while 'make release' makes the public version which omits some stuff.

The first time you run 'make install', it will take a while to copy all the big data files to the new installation. Subsequent 'make installs' will only copy the changed files.

If you only want to install a single binary, run 'make install' from a subdirectory. For example, running 'make install' from the scuba/ directory will copy the scuba binary and its support script files to the proper locations. Running 'make install' from scripts/ will copy all the necessary scripts to the right location.

==== autoconf Troubleshooting ====

Here's a list of common problems and solutions for autotools problems:

[AutoconfTroubleshooting]

==== Making changes live ====

The current setup allows you to have your own private installation of FreeSurfer using {{{configure --prefix=/your/private/directory}}}. This lets you use {{{make install}}} to easily install the entire FreeSurfer environment into a local copy.

The purpose of having your own private installation of FreeSurfer is to use it as a testbed to make sure that (1) your updates, whatever they be, work in freesurfer/ setup that is just like a real one, and (2) your updates are properly installed using 'make install' and 'make release'. Once you are sure of these two things, check in your updated files, including any Makefile.am and Makefile.in changes, etc, and the nightly build script will take care of the rest.

The nightly build stuff automatically updates and installs the dev/ distribution on all seven build platforms. The next morning, just check if your changes are in any /usr/local/freesurfer/dev; if not, notify me. If they are in one /usr/local/freesurfer, they will be in the rest of them. (If any build/installs fail, I am notified, and will do the build/install manually for that day.)

Things not to do:

- Don't test your local setup by manually copying files into your local freesurfer/. Otherwise you can't be sure that the automated build will correctly install your files with 'make install' and 'make release'. The only way files should get into your freesurfer/ distro as with 'make install' and 'make release'.

- Don't try to install into /usr/local/freesurfer yourself (i.e. by setting your .configure --prefix=... to target it). It's set up so the build scripts can do their job automatically. Also, you would only be installing on one of the seven platforms. It's a pain to do all three builds on all build machines and verify it all worked - that's what the nightly build script is for, so let it do it for you.

If you absolutely need a 'hot fix' right away, and need to make changes live in any of the /usr/local/freesurfers before the nightly build stuff goes, let KevinTeich know and he'll do it.

==== How the nightly build works ====

First a note on the directory structure. In {{{/space/freesurfer/build}}} there is a directory for each build machine. Each builds a platform in {{{/space/freesurfer}}}, i.e. {{{/space/freesurfer/rh9}}}. Currently these are as follows:

|| '''Build machine''' || '''Platform''' ||
|| fishie || centos4.0 ||
|| icelin || centos4.0_x86_64 ||
|| kani || rh9 ||
|| martinos01 || rh7.3 ||
|| minnehaha || fc2 ||
|| storm || tiger ||

Inside each {{{/space/freesurfer/build/MACHINE}}} is {{{trunk/}}} and {{{stable/}}}, as well as directories for local Qt installs, and on Tiger, MNI, and files for automated building. {{{configure_options.txt}}} contain command-line arguments for {{{configure}}} for that platform, and {{{source_before_building.csh}}} (optional) contains environment variables that must be set (mainly things like CPPFLAGS that are used by configure but can't be passed in to configure via the command line from a cronjob). The trunk and stable directories contain cvs checkouts.

In {{{/space/freesurfer/build/scripts}}} is {{{build_dev.csh}}} and {{{build_betapub.csh}}}. Each build machine runs {{{build_dev.csh}}} nightly. It basically goes to {{{/space/freesurfer/build/MACHINE/trunk/dev}}} and does the following:

{{{
cvs update -d
make distclean
rm -rf autom4te.cache
aclocal
autoconf
automake
./configure `cat ${BUILD_DIR}/configure_options.txt` --prefix=/usr/local/freesurfer/dev
make
rm -rf freesurfer/bin_old
mv freesurfer/bin freesurfer/bin_old
make install
}}}

The {{{build_betapub.csh}}} works in {{{/space/freesurfer/build/MACHINE/stable/dev}}} and also does:

{{{
make release prefix=/usr/local/freesurfer/pub
}}}

All the autotools stuff gaurantees that enough of the Makefiles will be regenerated to ensure that new directories will be added to all Makefiles. This wouldn't normally be done with a simple {{{./config.status}}} because {{{configure}}} may need to be regenerated to know about new directories.

Currently (as of 4/8/2005) the build_betapub.csh script is not run regularly. This is to minimize interruption to the freesurfer/beta build.

Note that all machines can run these scripts at any time from the command line. This lets maintainers (KevinTeich and NickSchmansky) explicity do a build and install on any machine if a 'hot fix' is needed.

Additionally, a developer can go to each machine's {{{/space/freesurfer/build/MACHINE/trunk/dev}}}, update code, and do a {{{make install}}} from an individual binary directory if a hot fix is needed for a single binary before the nightly build. This is the quick and dirty way of doing an update for a single binary without waiting for the nightly build. For example, to update tkmedit, the developer would, for each build machine:

{{{
cd /space/freesurfer/build/MACHINE/trunk/dev/tkmedit
cvs update
make install
}}}

This would install the files to {{{/usr/local/freesurfer/dev}}}. However, the developer must know about file dependencies. In this case, the script files that tkmedit may also need to be updated, so the developer would also have to {{{cvs update}}} and {{{make install}}} from {{{dev/scripts}}}.

=== RPM ===

See RpmInfo

=== Medical Image Format FAQ ===
[http://www.dclunie.com/medical-image-faq/html Medical Image Format FAQ]
Finally, we need to add this directory to the list of included directories in the `CMakeLists.txt` located at the top level of the FreeSurfer file tree. To add your new utility to your build of FreeSurfer, reconfigure your build, and run make in the `mri_process` directory. You can also rebuild all of FreeSurfer with your new utility if needed. More information on building Freesurfer is located in the BuildGuide.

FreeSurfer Dev Guide

  • Visit the BuildGuide for instructions on building and installing freesurfer manually.

  • Visit the GitHub page for an introduction to the github workflow.

  • Visit the GitAnnex page for detailed instructions on using git annex for storing and retrieving large data files in the repository.

File Size Limitations

Any files larger than 50MB should be stored in the GitAnnex, following the hyperlinked instructions, and properly linked to your utility.

Adding a New C Program

If you'd like to add a new program to the tree, you should create a new subdirectory with the title of your tool. As an example, let's create a new c++ program called mri_process. First, we'll create a top-level subdirectory that contains our new c++ file and an empty CMakeLists.txt file:

freesurfer/
    mri_process/
        CMakeLists.txt
        mri_process.cpp

In order to configure our new code, we should add the following to the empty CMakeLists.txt file.

project(mri_process)

include_directories(${FS_INCLUDE_DIRS})

add_executable(mri_process mri_process.cpp)
target_link_libraries(mri_process utils)

install(TARGETS mri_process DESTINATION bin)

This will compile mri_process.cpp, link it against the utils freesurfer library, and copy the executable to the $FREESURFER_HOME/bin directory during install. To include this subdirectory in the main freesurfer build, make sure to modify the top-level CMakeLists.txt by adding mri_process to the long list of included directories at the bottom of the file. Now, after reconfiguring your build, you can run make in the mri_process directory of your build tree to successfully compile the new program. If you're having trouble configuring and building freesurfer, be sure to visit the BuildGuide for step-by-step instructions.

Adding a New Python Program

Adding a new python program is similar to adding a new c++ program, with an additional step of creating a bash wrapper to be installed in $FREESURFER_HOME/bin.

  • Note: Make sure all paths in the Bash wrapper that reference files in the FreeSurfer file tree are defined relative to $FREESURFER_HOME, to ensure they are found regardless of the install location

As an example, let’s add a program called mri_process to the FreeSurfer tree, where the main functionality is in a python script called mri_process.py, and has a dependency on a model called process_model.h5.

First, we’ll create a directory containing the scripts, dependencies, and CMakeLists.txt:

freesurfer/
        mri_process/
                CMakeLists.txt
                mri_process.sh          # shell wrapper to install in bin/
                process_model.h5        # model dependency stored in the annex
                python/                 # dir containing the python code
                        mri_process.py  # python script called by shell wrapper

Now we’ll need to populate the empty CMakeLists.txt file. We need to achieve the following in this file:

  1. Give the utility a project name
  2. Link the python scripts to the location they are referenced by the Bash wrapper
  3. Install the wrapper script into $FREESURFER_HOME/bin

  4. Link any models to the proper location to be accessible to the python code

An example CMakeLists.txt:

project(mri_process)

install_symlinks(python/mri_process.py TYPE files DESTINATION python/scripts)

install_configured(mri_process.sh DESTINATION bin)

install_symlinks(process_model.h5 TYPE files DESTINATION models)

Finally, we need to add this directory to the list of included directories in the CMakeLists.txt located at the top level of the FreeSurfer file tree. To add your new utility to your build of FreeSurfer, reconfigure your build, and run make in the mri_process directory. You can also rebuild all of FreeSurfer with your new utility if needed. More information on building Freesurfer is located in the BuildGuide.

DevelopersGuide (last edited 2023-09-13 15:46:41 by JacksonNolan)