By default, Matlab matrices must be fully loaded into memory. This can make allocating and working with huge matrices a pain, especially if you only really need access to a small portion of the matrix at a time. memmapfile allows the data for a matrix to be stored on disk, but you can't access the matrix transparently in functions that don't expect a memmapfile object without reading in the whole matrix. MappedTensor is a matlab class that looks like a simple matlab tensor, with all the data stored on disk.

A few extra niceties over memmapfile are included, such as built-in per-slice access; fast addition, subtraction, multiplication and division by scalars; fast negation; permutation; complex support.

Tensor data is automatically allocated on disk in a temporary file, which is removed when all referencing objects are cleared. Existing binary files can also be accessed. MappedTensor is a handle class, which means that assigning an existing mapped tensor to another variable will not make a copy, but both variables will point to the same data. Changing the data in one variable will change both variables.

MappedTensor internally uses mex functions, which need to be compiled the first time MappedTensor is used. If compilation fails then slower, non-mex versions will be used.

Download MappedTensor
Unzip the @MappedTensor directory to somewhere on the Matlab path. The @ ampersand symbol is important, as it signals to Matlab that this is a class directory. Then type:

addpath /path/to/Mappedtensor

Note: MappedTensor provides an accelerated MEX function for performing file reads and writes. MappedTensor will attempt to compile this function when a MappedTensor variable is first created. This requires mex to be configured correctly for your system. If compilation fails, then a slower pure Matlab version will be used.

M = MappedTensor([dim1 dim2...], ...) M = MappedTensor(dim1, dim2, ...) M = MappedTensor(dim, ...) allocates a MappedTensor object to store an array with specified size [d1 d2...]. When only one dimension value is specified, the actual dimension is for a square array [d d].

M = MappedTensor(FILENAME, [d1 d2...], 'FORMAT', class, ...) constructs a MappedTensor object that re-uses an existing map file FILENAME for an array with dimensions [d1 d2...]. The full size, class, and offset of the file must
be known and specified in advance. This file will not be removed when all handle references are destroyed.

M = MappedTensor(ARRAY) constructs a MappedTensor object that maps a numeric array ARRAY into a temporary file. The array must be 2D or more (not scalar, nor vector that would conflict with a dimension setting). This syntax implies to already allocate the initial array, which limits the size of the MappedTensor. For large arrays, it is more efficient to pre-allocate the object with specified dimensions or the 'Size' property and then set the content, per chunks.

M = MappedTensor(..., PROP1, VALUE1, PROP2, VALUE2, ...) constructs a MappedTensor object, and sets the properties of that object that are named in the argument list (PROP1, PROP2, etc.) to the given values (VALUE1, VALUE2, etc.). All property name arguments must be quoted strings (e.g., 'writable'). Any properties that are not specified are given their default values.

Note: When a variable containing a MappedTensor object goes out of scope or is otherwise cleared, the memory map is automatically closed. You may also call the DELETE method to force clear the object.

All properties can be accessed with syntax e.g. M.property. All these properties can also be set when building the tensor.

We detail below the use of these properties, especially to set an initial tensor.

Format of the contents of the mapped region. Format specifies that the mapped data is to be accessed as a single vector of type specified by Format's value. Supported char arrays are 'int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', 'uint32', 'uint64', 'single', and 'double'. Complex arrays are supported. Sparse arrays are not supported. You can change later the storage class of the object with the CAST method, however this is usually not recommended.

Number of bytes from the start of the file to the start of the mapped region. Offset 0 represents the start of the file. This allows to skip over the beginning of an (existing) binary file, by throwing away the specified number of header bytes. You can use methdos FREAD and FWRITE to read this header region.

Access level which determines whether or not Data property (see below) may be assigned to. This property can be changed after object creation.

When false, the associated file is kept when the object is cleared. Such files can be further reused. When the object is created from an array, Temporary is true. When creating from an existing map file Temporary is false. You can change this property after creation. When saving an object, the Temporary state is set to false. This property can be changed after object creation.

If not specified, the machine-native format will be used.

Array to assign to the mapped object. This property can be changed after object creation. You can also set the Data with syntax:

set(M, 'Data', array)
M(:)            = whole_array;
M([ 1 3 5... ]) = slice; 

Contains the name of the file being mapped. You can also get the mapped file with FILEPARTS.

Contains the name of the file being mapped (complex part). You can also get the mapped file with FILEPARTS.

Directory where the mapped file(s) should stored. The default path is e.g. TMPDIR or /tmp. You may also use /dev/shm on Linux systems to map the file into memory. This can be very efficient in terms of I/O, and an be coupled with tensor compression with the pack method.

Specified array dimension and class is used to preallocate a new object. Note that sparse arrays are not supported.

Vector which specifies the size of the mapped array. This is the same as specifying dimensions as first arguments (see above).

All the properties above may also be accessed after the MappedTensor object has been created with the GET method. For example,

set(M, 'Writable', true); % or M.Writable = true;

changes the Writable property of M to true.

The LOAD method allows to lazy import binary data sets with syntax

m = load(MappedTensor, 'filename');

with the following data formats.

The MappedTensor array can be used in most cases just as a normal Matlab array, as many class methods have been defined to match the usual behaviour.

You may access the array with indices as in M(I,J,..). The full tensor content is retrieved with M(:) as a column, or M(:,:) as pages, and finally as M.Data to get the raw shaped array.

Most standard Matlab operators just work transparently with MAPPEDTENSOR. You may use single objects, and even array of tensors for a vectorized processing, such as in:

m=MappedTensor(rand(100)); n=copyobj(m); p=2*[m n];

These objects contain a reference to the actual data. Defining n=m actually access the same data. To make a copy, use the COPYOBJ method.

Transparent casting to other classes is supported in O(1) time. Note that due to transparent casting and tranparent O(1) scaling, rounding may occur in a different class to the returned data, and therefore may not match Matlab rounding precisely. If this is an issue, index the tensor and then scale the returned values rather than rely on O(1) scaling of the entire tensor.

To work efficiently on very large arrays, it is recommended to employ the ARRAYFUN method, which applies a function FUN along a given dimension. This is done transparently for many unary and binary operators (with ARRAYFUN2).

The NUMEL method returns 1 on a single object, and the number of elements in vectors of objects. To get the number of elements in a single object, use NUMEL2(M) or PROD(SIZE(M)). This behaviour allows most methods to be vectorized on sequences on tensors.

If you need to handle many such tensors, it may be a good idea to compress them with pack(m) while you are not using them. This can be done for instance right after loading content. Decompression is performed transparently while you access the tensor array. Think about re-compressing afterwards to save disk/memory. Compression is usually extremely efficient on data with low randomness.

An efficient processing pipeline could be:

• load tensors
• compress them with pack
• do whatever you need (extraction is performed automatically)
• recompress as soon as possible with pack

A list of available methods is shown below.

   % To create a mapped file for a given input array:
   % A temporary file is created to hold the data.
   m = MappedTensor(rand(100,100,100));

   % To reuse a previously existing mapped file:
   m = MappedTensor('records.dat', [100 100 100], ...
         'format','double', 'writable', true);
   m(:) = rand(100, 100, 100);  % assign new data
   m(1:2:end) = 0;

This work was published in Frontiers in Neuroinformatics: DR Muir and BM Kampa. 2015. FocusStack and StimServer: A new open source MATLAB toolchain for visual stimulation and analysis of two-photon calcium neuronal imaging data, Frontiers in Neuroinformatics 8 85. DOI: 10.3389/fninf.2014.00085. Please cite our publication in lieu of thanks, if you use this code.

This version of the code has been heavily revamped by [email protected]. Please cite the following publication:

• E. Farhi et al., J. Neut. Res., 17 (2013) 5. DOI: 10.3233/JNR-130001