Differences between revisions 1 and 8 (spanning 7 versions)
⇤ ← Revision 1 as of 2017-04-12 16:54:56 →
Size: 2175
Editor: ZekeKaufman
Comment:
← Revision 8 as of 2017-05-12 17:20:37 → ⇥
Size: 3090
Editor: ZekeKaufman
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#acl LcnGroup:read,write,delete,revert #acl LcnGroup:read,write,delete,revert All:read
Line 11: Line 11:
 {{{ {{{
Line 20: Line 20:
 }}} }}}
Line 27: Line 27:
git annex add <filename>
git commit -a -m "Added new file"
git annex copy --to datasrc
Line 30: Line 32:
== Tagging a data file == == Getting a data file ==
Line 32: Line 34:
The data files have been broken down into the following 3 categories, and it is essential that data files get the proper tag(s) so that our servers and diskspace is not overwhelmed when only a known subset of the data is required.: To retrieve the contents of a data file:

{{{
git fetch datasrc (maybe could do 'git annex sync')
git annex get mri_em_register/testdata.tar.gz
}}}

Retrieve everything under the current directory (not recommended)
{{{
git annex get .
}}}

== Modifying a data file ==

To modify the contents of a data file, first unlock it (which eliminates the symlink), than modify it, then re-add to the annex:
{{{
git annex unlock mri_em_register/testdata.tar.gz
<modify contents of tar file>
git annex add mri_em_register/testdata.tar.gz
git commit -am "New test data"
git push
git annex copy --to datasrc
}}}

== Tagging ==

Git -annex provides the ability to to tag data files. Freesurfer utilizes tags so that subsets of the data can be retrieved without having to download everything. The data files have been broken down into the following 3 categories:
Line 38: Line 66:
To list the metadata of a given file: It is essential that data files get the proper tag(s) so that our servers and diskspace is not overwhelmed when only a known subset of the data is required.

=== Display metadata ===

To show all the metadata associated with a file:
Line 44: Line 76:
To assign a tag to an existing datafile: === Assign metadata ===

To assign a tag to an existing datafile.
Line 53: Line 88:
git annex sync
Line 55: Line 91:
To list all files with a given tag: === List all files with a given tag ===
Line 60: Line 96:
To download all datafiles with a given tag: === Retrieve all files with a given tag ===
Line 62: Line 98:
Get only the data files required for build time checks (1.9 GB)
{{{
git annex get --metadata fstags=makecheck .
}}}
Line 63: Line 103:
Get only the data files required for local installation (4.3 GB)
Line 64: Line 105:
## Get only the data files required for build time checks (1.9 GB)
git annex get --metadata fstags=makecheck .

## Get only the data files required for local installation (4.3 GB)
Line 69: Line 106:

## Just give me everything! Not Recommended (6.8 GB)
git annex get .

Contents

  1. Initial Setup
  2. Adding a data file
  3. Getting a data file
  4. Modifying a data file
  5. Tagging
    1. Display metadata
    2. Assign metadata
    3. List all files with a given tag
    4. Retrieve all files with a given tag

This page describes how to deal with adding and tagging data files in the freesurfer source code repository.

Initial Setup

Based in the information included in the Freesurfer_github page, the remotes of your freesurfer repo working directory should look something like: 

git remote -v 
 
  datasrc       file:///space/freesurfer/repo/annex.git (fetch)
  datasrc       file:///space/freesurfer/repo/annex.git (push)
  origin        git@github.com:zkaufman/freesurfer.git (fetch)
  origin        git@github.com:zkaufman/freesurfer.git (push)
  upstream      git@github.com:freesurfer/freesurfer.git (fetch)
  upstream      git@github.com:freesurfer/freesurfer.git (push)

Adding a data file

The following example assumes we want to add a data file called 'testdata.tar.gz' to the 'distribution' directory: 

git annex add <filename>
git commit -a -m "Added new file"
git annex copy --to datasrc

Getting a data file

To retrieve the contents of a data file: 

git fetch datasrc (maybe could do 'git annex sync')
git annex get mri_em_register/testdata.tar.gz

Retrieve everything under the current directory (not recommended) 

git annex get .

Modifying a data file

To modify the contents of a data file, first unlock it (which eliminates the symlink), than modify it, then re-add to the annex: 

git annex unlock mri_em_register/testdata.tar.gz
<modify contents of tar file>
git annex add mri_em_register/testdata.tar.gz
git commit -am "New test data"
git push
git annex copy --to datasrc

Tagging

Git -annex provides the ability to to tag data files. Freesurfer utilizes tags so that subsets of the data can be retrieved without having to download everything. The data files have been broken down into the following 3 categories:

  1. Those being required for build time checks (tagged makecheck)

  2. Those required for a local installation (tagged makeinstall)

  3. Everything else (untagged)

It is essential that data files get the proper tag(s) so that our servers and diskspace is not overwhelmed when only a known subset of the data is required.

Display metadata

To show all the metadata associated with a file: 

git annex metadata mri_em_register/testdata.tar.gz

Assign metadata

To assign a tag to an existing datafile. 

git annex metadata mri_em_register/testdata.tar.gz -s fstags=makecheck
git annex sync

We can also append tags: 

git annex metadata mri_em_register/testdata.tar.gz -s fstags+=makeinstall
git annex sync

List all files with a given tag

git annex find --metadata fstags=makecheck

Retrieve all files with a given tag

Get only the data files required for build time checks (1.9 GB) 

git annex get --metadata fstags=makecheck .

Get only the data files required for local installation (4.3 GB) 

git annex get --metadata fstags=makeinstall .