Brain Innovation

support portal

Navigation

BrainVoyager Available Tools Matlab Tools Transparent file access in NeuroElf (BVQXTools)

Transparent file access in NeuroElf (BVQXTools)

Memory problems and solutions

Since performing a (voxel or vertex based) group statistics may require a huge amount of available memory (say 3GB for 30 VTCs with 100MB each), it is often necessary to perform an analysis bit by bit. 

One way to solve this is to split an analysis up into temporally independent calculations (e.g. GLM estimation for separate studies). While this will work in some cases, there are situations where the time course data of all studies must be present in working memory to perform a calculus. However, it will (hopefully) only rarely occur that both all spatial and temporal information must be loaded in memory at the same time.

To simplify the access of large numerical arrays within files in chunks, the @transio class has been introduced into the toolbox.

Working mechanism

For each file type (e.g. VTCs) a threshold can be set (which by default is set to positive infinity). If for any given member variable of a file (in the case of a VTC the VTCData member) this threshold is exceeded, instead of reading the array into memory, a transio-object reference is created instead and the according number of bytes is skipped during the read.

Later, during access to that array, all read and write operations are passed to two functions that perform these operations directly on disk, efficiently blocking even complex indexing schemes into as few as possible read operations on disk.

Enabling transio access
To enable transio access for VTCs, you would give the BVQXfile class the new threshold:

BVQXfile(0, 'transiosize', 'vtc', 65536);

With this command, every VTC that is opened afterwards will contain transio objects instead of large data arrays for fields that surpass the 64kB threshold.

To set this threshold for all (binary) files, leave out the filetype:

BVQXfile(0, 'transiosize', 5e5);

This would set the threshold to roughly 500kb for all filetypes

Advantages

There are quite a few advantages that might be useful in several different cases:

  • opening time of large files reduces dramatically (e.g. to check whether a VTC file has the correct PRT file linked)
  • highly efficient file access, allowing even complex index operations in a reasonable time
  • highly efficient memory consumption by only storing required information

Syntax

The general syntax is quite straight forward. Indexing follows MATLAB rules with one major exception: indices might not be reused! Thus, if you want to access the same position within an array (or file) more than once, you must (at least currently) do so by first accessing unique positions and then indexing into the resulting array.

Here is just one example:

% enable transio for VTCs
BVQXfile(0, 'transiosize', 'vtc', 65536);
% open VTC, but do not load the VTCData...
vtc = BVQXfile('S1_R1.vtc');

% show memory consumtion
whos

This might then give...

Name    Size    Bytes     Class
vtc     1x1     26527     BVQXfile object

Grand total is 2488 elements using 26527 bytes

A useful way to employ transio objects could then be:

% create a mask in memory
mask = false(vdsize(2:4));
% get VTC size (overloaded correctly)
vdsize = size(vtc.VTCData);

% iterate over BV Z (TAL X) axis
for sc = 1:vdsize(4)

% and find voxels where mean is > 80
mask(:, :, sc) = squeeze(mean(vtc.VTCData(:, :, :, sc))) > 80;
end

% now extract only voxels of interest
maskedtc = vtc.VTCData(:, mask);

To yet receive the entire array, one can of course use the full-colon notation, but if the size is unknown (e.g. for a GLM map, which in the case of surfaces is VxP, vertices-by-predictors), another function, resolve(tiobj);, which has been introduced in v0.7b.

Limitations
WARNING: Also, for the current implementation, write operations are directly performed within the file that was opened. So, use any write operations into variables with extreme care!

For obvious reasons, it is impossible to grow or shrink a transio object within a file. Although it would be possible to implement this (by first copying the array into a temporary file and then doing the operations there), this would certainly require a chunk of memory at least by the size of the entire array to efficiently perform these operations.

Hence, for the time being, call such as

% growing a VTC in time dim
vtc.VTCData(end+1:end+50, :, :, :) = 0;
% or removing BV system Z slices 21:end
vtc.VTCData(:, :, :, 21:end) = [];

will not work!!

You are here: HomeBrainVoyagerAvailable ToolsMatlab Tools ≫ Transparent file access in NeuroElf (BVQXTools)