NETCAL: An interactive platform for large-scale, NETwork and population dynamics analysis of CALcium imaging recordings

NETCAL is a MATLAB-built, dedicated software platform to record, manage and analyze high-speed high-resolution calcium imaging experiments. Its ease of use, interactive graphical interface and exhaustive documentation is aimed to wet-lab researchers, but it will also meet the needs of any experienced data scientist through its plugin and scripting system. We have developed a large set of tools and incorporated state-of-the-art algorithms and toolboxes for large-scale analysis of network and population dynamics. Analyses include: automated cell detection (both static and dynamic); trace and population sorting through machine learning, clustering and pattern recognition; bursting dynamics; spike detection; network inference (from functional networks to causal relations); and many more. Several of these tools are also available in real-time, e.g. cells and spikes can be monitored during the actual recording, giving the researcher extensive feedback on the progress of the experiment.

• You can find here the latest installation instructions.
• Install additional MATLAB toolboxes: Go to the installDependencies folder and execute the mltbx files (or drag and drop in MATLAB). NOTE: In MATLAB r2017a+ you might need to download the GUI Layout Toolbox from the Add On Explorer yourself.
• Change default MAT-Files version: Go to MATLAB Preferences -> General -> MAT-Files to v7.3
• To be able to export figures to pdf and eps you need ghostscript installed.
• To be able to use any of the python modules you need python installed and available to MATLAB

• Matlab 9.1+ (2015a or higher) - 2017a or higher recommended

• Bioinformatics Toolbox (4.5.1) [bioinfo]
• Curve Fitting Toolbox (3.5.1) [curve fit]
• Econometrics Toolbox (3.2) [econ]
• Global Optimization Toolbox (3.3.1) [globaloptim]
• System Identification Toolbox (9.2) [ident]
• Image Processing Toolbox (9.2) [images]
• Neural Network Toolbox (8.3) [nnet]
• Statistics and Machine Learning Toolbox (10.0) [stats]
• Computer Vision System Toolbox (6.2) [vision]

• viewROI now accepts power spectrum images
• peak detector to detect peaks in traces (will be used to refine pattern-based classifiers)
• module to detect dying cells (those with step-like changes in fluorescence levels)
• view traces available in batch mode
• Several new features on the KCl analyis module
• Spike-based burst detection method
• Power-spectrum based preprocessing (obtain different average images based on power-spectrum bands)
• More spike features (spikes in bursts)
• More options for MLspike inference
• Cross-correlation measures
• New raster plots with average activity on top
• Options to multiply or divide all frames using an external file (to correct uneven light fields)

• Updated CaImAn repository
• Updated CellSort repository and fully integrated it within NETCAL (GUI pending)
• Schmitt trigger now admits point to point thresholds
• Substantial speed up on automatic ROI detection (quick_dev and split modules)

• Renamed brick montage to avoid conflicts
• Some files where not being copied when importing/cloning exsting .EXP files
• Some statistics were not being correctly exported

• Several consistency checks on trace fixer

• There is an ongoing bug where several pipeline functions are not being correctly loaded from their json files

• Open beta release

• There is an ongoing bug where several pipeline functions are not being correctly loaded from their json files

• Clean slate preparing for OpenBeta release

• Introduced Development mode to distinguish between dev builds and the open beta
• Preparing repository for open beta release

• Increased menus responsiveness when switching between experiments
• Added error handling for incomplete video files

• Moved several functions into the new development folders (will only be avilable on the private/testing repository)

• Links to old trunk versions in the repository

• Option to define the type of averages when computing the statistics (mean or median)

• Option to change bar group order in statistics plots (both normal and treatments)

• Cell-array-type options can now (hopefully) be correctly saved and loaded from pipeline files

• Complete rework of the optionsWindow GUI (again)

• Issue when data tables inside options were not being correctly saved

• Small bug fixes

• Option to remove a background image on every frame-acquisition operation
• Metadata is now displayed on the experiment info tab
• External files to load patch clamp data
• External files to correctly read xml for future graphpad output

• Some bug fixes on the plot statistics options

• Treatment statistics for bursts

• Treatment statistics now fully working for both population and spike statistics

• Complete treatment statistics class. Implemented population statistics as first test

• Currenlty the set of labels inside the projectPipleine are not saved when moving through the options

• Many pipeline statistics plots merged and integrated into a cohesive structure. Currently working: populations statistics, fluorescence bursts, spike statistics, permutation entropy/complexity
• Detailed warning incase the pipeline cannot be saved
• optionsWindow substructures working much better

• boxPlot to accept multiple colors

• DEBUG tag for pipeline functions. These functions won't be uplodaded to the production version but will appear on the pipeline
• Unified grid-like ROI addition. With option to reset or append new GRIDs

• Pipeline functions now are sorted by name and children.
• baseOptions class now accepts structures
• optionsWindow help class generation accepts structures

• camControl new experiments were not being updated on the projectTree
• producedFields was missing on spikeDetectConflicts

• Bug when the ROI screen could not be opened

• Tononi complexity measure

• Drag and drop a project onto the experiment list to load it
• Option to append a ROI file to a preexisting ROI list on the viewROI menu

• Issues with acute protocols and PCA analysis

• Full rework on how to plot 1D statistics using a new class

• viewRecording and viewROI menus now have a minimum width and height to deal with very one-dimensional recordings
• Redefined several observables for the acute protocols
• Number of (total/selected) ROIs now appear in the viewROI and viewTraces window

• Experiment info window was not always being updated after changes
• Sometimes new movies could not be generated from the viewRecording menu

• Spike statistics for inter-experiment treatments (icomplete)
• Spike statistics now allows grouping by any kind of labels
• complexity-entropy curves with global spike trains
• Dissimilarity and complexity measures on the spike featurse (trivial implementation, not really useful as of now)

• Several features to the acute stimulation protocols
• PCA Analysis with features from stimulation protocols

• Some .avi files were not correctly being displayed on the viewRecording window

• Entropy, ISI fano factor and ISI coefficient of variation as new spike features
• Analysis and statistics for acute stimulation protocols related to fluorescence transients (KCl for now)
• New experiment list tree (now with drag and drop functionality)
• MLspike now has options
• Experiments can have multiple (comma separated) labels. Also option to modify or add labels on the experiment list
• Options menu to change default/global options on a project
• Current pipeline functions can be copied by pressing control while dragging the function

• Experiment and project pipelines fused into one
• Changelog only displayed on succesful updates in a new window
• Trees should have a consistent look now
• Project menu is now called File
• Switching between experiment mode and pipeline is now much faster

• xlwrite support - it was a pain and was buggy for large exports. Use CSV instead

• Bug when moving functions/experiments on very large trees
• Lots of small fixes

• Graphad export support
• Complexity measures

• Removed debug lines on plotSpikeFeatures

• traceFixer now also works on smooth traces, new correction method works much better on single traces.
• Procedures for the analysis of acute stimulation protocols

• Lots of small bugs
• traceFixer should also work with smoothed traces now

• Another issue when loading BTF files

• Updated OASIS to the latest remote commit. New models available

• Issue when opening the stream on BTF files

• Support for BTF files

• Manual option for the new trace fixer
• Trace fixer options are now also stored

• OASIS integration
• pipeline: plotPopulationStatistics
• Lots of undocumented changes (sorry!)

• New option to remove conflictive spike (those that pattern matching detected as something else)
• Valid patterns are now stored in a separate file to speed-up loading times

• New try/catch mechanism for log messages

• Pipelines: Run & Check buttons below the pipelines current function list
• Pipelines: Right click on any function in the available functions list to display its help
• Pipelines: project info is now also displayed on the pipeline modes
• Plot spike statistics function on the pipeline
• Added new options to the pattern count classification

• Small bug fixes

• Relative threshold for foopsi spike infernece (works much better)
• Option to delete all patterns before a new import
• New options class for raster plots

• Lots of small bug fixes

• Pipeline functions now support any number of parents and nested parents. Use colons to separate nested groups in the file header
• Pipeline functions for pattern-based group classification
• Pipeline functions for HCG detection
• Automatic detection and correction of corrupted HIS files
• Several new options to plot and export qCEC curves

• Reestructured pipeline functions

• Pipeline functions for burst detection
• Pipeline functions to load external pattern file
• View traces: when simultaneously viewing movie and traces, ROIs can be sorted by distance from a given point (by right clicking)
• camControl: same roi sorting functionality as in view traces
• camControl: better webcam support
• Font sizes can be changed from the main preferences menu (experimental)

• Completely redesigned burst analysis menu to fully implement pattern-based burst detection

• Several bug fixes

• Several bugs on the view traces menu due to the new fast loading of the traces

• Youtube channel! Subscribe to stay up to date https://www.youtube.com/channel/UCQ8j9oytnMe34Ros_xV4x-w
• View traces: now when creating the HCG, two new groups will appear (one excluding the largest HCG and another excluding all HCG members)
• Improved burst statistics export
• Improved traces export

• Many small bug fixes

• Denoiser to clean and generate new recordings
• Denoised movies are compatible with most primary functions, i.e., roi selection, trace extraction, smoothing...
• Speedup most functions by only loading the required traces and big fields at any given time
• Data cursors can now be enabled on many windows
• Percentile-based preprocessing
• viewTraces: option to sort ROIs by distance from pointer when visualizing movie and traces simultaneously

• New smoothing option (v2), simplified and more accurate (in theory)

• Serval bug fixes

• Compatibility for RIKEN VDAQ files
• Updater now skips user modified files that contain % MODIFIED in the header

• View traces: ROI index now appears on the view positions menu
• qCEC: Option to show aggregated results by label

• Small bug fixes

• Trace/event/manual training: You can select multiple training elements by pressing shift
• View Traces: You can move through the traces both with the arrowkeys and WASD. You can also move through training groups by pressing the appropiate number key
• View Traces: You should be able to move through the traces with the mousewheel

• Several bug fixes

• Patterns: Impot/export feature
• Highly correlated groups: automatic search of optimal threshold

• Several bug fixes

• Bug when saving the times after cuting the traces
• Added option to rebase time

• Completed q-CEC pipeline functions. Added option to work on the raw traces instead

• More bug fixes

• Weird bug on burstAnalysis

• Error on PCA k-means

• Automatic event classification
• q-complexity-entropy measure and sorting option
• Added pattern-based burst detection

• Option to color the PCA results by experiment, label or cluster index
• Added subset option to export spikes
• Added option to plot raster in the pipeline
• Speedup of experiment saves

• loadExperiment with old quick_dev formats
• Loading experiment where the traces file could not be located anymore
• Several bugs with the new progressbar

• Several bugs with the new progressbar

• New robust smooth traces option

• Completely redesigned progressbar, now it doesn't get stuck anymore while performing disk writes. There still might be compatibility issues with the old one. Hence the change in version major.
• Completely redesigned event-based trace classification - almost production-ready
• New default 'ask' option for updates
• Started integration of mlspike
• Black colormap

• Considerable speedup on experiment saves - now traces and big folders are only saved if they are modified

• Lots of small bug fixes

• camControl .bin files are fully supported

• Improved trace export feature

• New button to load ROI for compatibility issues

• Reverted the use of isstring due to compatibility uses with prior matlab versions

• Experiments are now backed up after every save and one hour apart, so previous versions can be recovered
• Full export of burst statistics populations

• camControl now can handle importing all previous ROI files
• Issue with the trace smoothing where the spline could overshoot at the end of the recordings

• Removing xls support on several export optionsk, since csv works much better and can directly be imported into excel

• Export spikes pipeline

• Issue where the pipeline was being incorrectly saved (or not saved at all)
• Improved pbar reliability
• Improved trace loader

• Pipeline option to export burst features

• Zoom level is now preserved when browsing through traces
• Added alpha-channel slider when viewing the positions of the current traces
• Dual monitor setups are better supported when it comes to window positioning
• Option to clear the current log
• Option to import experiment from the workspace

• Several camControl issues when loading and saving ROIs

• Missing

• Pipeline option to export fluroescence traces features

• Small bugs on camControl

• Plugin to import experiment from workspace

• small bugs on camControl

• small bugs on camControl

• Java-based multithreading for file writing

• Comptability issues with new newline function
• Spurious java warnings

• Started the detailed documentation for every function
• macvideo support on camControl

• Old glia analysis now conforms to the new call system

• Options to deal with incomplete .HIS files
• Full integration of new generation camControl files
• Option to add and edit information in the experiments (metadata)
• viewTraces now conserves zoom while changing modes
• new automatic ROI detection mode (combines the previous two)
• Ability to generate .bin files with a raid-like structure
• Several new pipeline functions
• camControl now supports any winvideo-based camera
• Direct option to integrate experiments within MATLAB workspace

• Several bugs with json files in windows systems
• Disabled zero-padding on fft smoothing
• Lots of small bug fixes
• camControl is now a lot more robust

• Option to resize all ROI
• Full integration of camControl within netCal. Now recorded videos will directly appear onto the current project

• Moved several buttons from viewROI to the menubar

• Several errors when trying to load incomplete experiment files

• camControl can now record experiments movies and traces simultaneouslly. Almost ready for production

• Several functions to camControl

• New issue when loading old exported ROI files

• Issue with overlapping paths due to oasis dependencies

• The viewRecording plugin now remembers the last used folder
• Oasis spike inference (only in training for now)
• Barebones cam control (camControlRedux)

• Bug with some HIS files where the last frames would appear out of place

• Option to reset ROI IDs
• Ability to compare experiments with different number of ROIs (will only compare equal IDs)

• Bug when exporting ROI lists that were moved around

• Pipeline modes for plotting and exporting spectrograms
• Ability to preload experiment options in the pipeline functions

• Spectrogram to the viewTraces menu
• Analysis of avalanches exponents
• Automatic ROI deletion when moving all ROIs simultaneously

• Removed hardcoded robuboost trainer when only 2 populations existed
• Changed from pcolor to imagesc on fluorescence rasters for 1:1 representation

• Several avalanche analysis functions

• Drift of spike times when the frame rate was fractional
• Baseline in oopsi

• Several fixes to the new pbar structure

• New options on experiment comparison

• New option for batch experiment comparison

• Several bugs fixes

• Several bugs with the new pbar

• Redesigned progressbar and its calls standarizing implementation
• Several new pipeline functions
• Parallel (asynchronous) running of the pipeline (1 experiment per thread)
• Experiment-specific validation of function options initial values
• Transition plots now have trasnparency and count type options
• Transition plots now have a minimum bar size to avoid issues with nearly empty populations

• Several bugfixes

• More options for automatic import of multiple experiments
• Option to directly import CRCNS datastes (MEA)
• More pipeline functions

• Faster load of multiple experiments

• Memory leak on spike feature extraction for very large spike trains
• Bug when displaying spikes form non-fluorescence experiments

• Much better handling of the options window in pipeline mode

• Second pipeline mode for directly managing projects.
• Experiment pipeline is for functions that run on experiments individually.
• Project pipeline is for functions that run on all the checked experiments at once.

• Several functions to the pipeline

• Automatic sorting of the function list based on their dependencies

• Bug with the update preventing creation of new folders

• Color coding for progress in pipeline mode

• Pipeline mode fully working. But most functions still need to be passed to the new format
• Several standalone functions for the new pipeline mode

• Typo on refineROI mode

• Option on PCA analysis to import an old PCA basis

• Option on PCA analysis to import an old PCA file

• ROI with wrong colors on the ppt reports
• Improved the speed of grid-like ROI detection
• Added warnings on spike feature extraction

• Several trace features were not working as intended

• Unsupervised trace classifier based on fuzzy clustering
• Several new features for trace classification
• More detailed error messages

• Error when a population was not defined in the aggregated statistics

• SaraReport now allows statistics on glia events
• Much more detailed error repot messages

• Very specific bugs on spike selection

• Spike/event features based on schmitt trigger inference
• Export options for both spikes and spikea features in the view spikes menu
• Option to remove the average (or percentile) trace from the recording

• Bugs on the ppt report
• Bug on trace classifier when using other experiments
• Bug when importing multiple experiments with wrong extension capitalization

• Bugs on the trace selection in the spikes window

• Bugs on the new options window
• Bugs on the trace selection

• Enabled pipeline mode. Menus and function selection working. Still a lot to go.

• Several bug fixes in trace and group selection
• Several bug fixed in ppt report

• PPT report should be working again

• Automatic fluorescence processing now works for both single and batch experiment modes
• Basis for the new pipeline mode

• Bug when loading smoothed traces that were already in numeric mode
• Several bugs on the trace selection and ordering process. Simplified the functions

• Real time spike detection on camControl (via foopsi)
• Standalone version of viewRecording can be called via the plugins menu

• Improvements on camControl

• Improvements on camControl

• Improvements on camControl

• auto levels now works much better. It is iterative and persistent

• Several bugs on view traces menu for subpopulations
• Spike features menu was not working
• Consistency check on windows systems was not working properly

• camControl plugin to control Hamamatsu cameras

• viewRecording as a standalone function

• Issue with the udpater in windows

• Foopsi model for spike inference fully working
• Automatic ROI detection as a new analysis menu (working towards pipeline model)

• Swapped around the way the options for spike inference training work

• Bug with the output of spike inference training
• Burst detection was not working with ill-defined groups
• The recording image was not displaying correctly after changing the file handle path

• Bug when the update required changes in the progressbar

• Menu to check consistency of the files and to force updates

• Issues with the file by file updater

• Completed view groups options by adding the group list on the left.
• Ability to delete, rename and sort groups from the view groups menu.

• Bug when loading ROI

• Bug when the Tree structure was not being loaded correctly from uiExtras package
• Bug when the updater was not creating the appropiate directories

• Bug when the ROI list was an empty cell
• Bug when trying to execute PCA analysis with old groups in the experiments

• Complete redesign of the way experiments are managed. Now there are two main working modes. Single experiment or batch mode. On single experiment you can proceed as usual, view and analyzing stuff one by one. On batch mode you just check the list of experiments that you want to analyze and each function will be performed on the checked list. Also the averaged statistics will only appear on the batch mode and apply to the checked experiments in the current order. The experiment order and labels can now be changed by right clicking on the experiment list.
• Complete redesign of the update function, now it updates file by file instead of downloading a single big zip every time.
• There is no concept of active experiment anymore. Experiments are loaded and saved on demand. Now if you have more than one view window open at any given time, experiments will not be saved on exit unless forced. Now there is an option to force save on each of the viewing windows.
• New view groups window to simply view the defined groups (populations) and do some statistics on them.
• Ability to export the groups generated by PCA and k-means analysis back to the original experiments.
• List of analysis, viewing and statistics functions are now created programatically with the loadModules() function, now it should be much faster to create new menus via plugins.
• Added beeswarm-like option to view distributions
• Several minor undocumented changes

• Several old functions are now incomplete. Namely the automatic processing of multiple functions at the same time.
• notBoxPlot now shows each point with a beeswarm-like structure

• Splash screen
• Logo

• Bug on the progressbar for smoothing traces

• Option to select export file format on preprocessing
• Option to export pca contributions
• Early check for GUI Layout toolbox

• Improvmeents on glia analysis
• New window for spike inference training
• Option to display the spikes features distributions
• Redesigned progressbar - iterative, sequential and circular
• Nonmodal options window
• Options window help automatically generated
• Spikes are now visible on view recording and view glia windows

• Better handling on the positioning of new figures and the first time they become visible
• Rework on automatic processing

• PCA contributions were being incorrectly displayed
• Missing java library declaration for xlsx exports

• Improvements on glia analysis
• Option to compress (or not) export movies

• Bug when selecting populations for the peeling
• Bug when resetting training traces
• Bug with new preprocessing options

• Statistics and plots for the glia events

• Option to view the populations on the view recordings gui

• Update error if netcal.json was not present

• Log panel on all GUI windows
• New projects are also added to the recent projects list
• Preparing menus for spike inference
• Option to train populations with learning traces from other experiments

• Complete rework on the log messaging system
• Complete rework on how the gui handles are passed through windows
• Removed the tabbed structure from the main window, since I haven't found any use for it since the beginning.
• Name of main windows - for better understanding
• Positioning of new figures, so there is less overlap
• Recent projects menu now display as multiple lines if the path is too large

• Aggregated PCA windows
• Possible infinite loop when updating menus
• Recent project order is updated so the latest load always become the last one in the list
• Recent project list is read backwards, so the last one is the top one on the list

• Option for loading recent experiments
• Option for batch burst analysis
• Option to plot tangent transitions in the experiment comparisons plots

• Completed burst viewing on view recording

• Several

• Several new avg trace options in the preprocess menu

• Complete rework on the import/export groups menu - now based on trees

• Major work on glia analysis
• Key bindings for view recording

• Bug when plotting experiment comparisons

• Bug when comparing experiments

• Bug when preprocessing new experiment

• Fixed group imports and exports

• Check for invalid zip files
• Additional external functions
• Added plots to compare population flows between experiments
• Option to extract traces with a median filter
• Added fft trace baseline correction
• Options for real size and image reset on viewROI menu
• Added help menu on the options window

• Complete rework on all GUIs to allow arbitrary groups.
• Complete rework on the experiment comparison
• Complete rework on the multi-experiment statistics
• Complete rework on the experiment and populations selection procedures

• Lots

• Merged ROI preferences options into a class
• Merged automatic ROI procedures into one

• New popup window for experiment selection when there are too many experiments (>25)
• Options class for peeling
• Batch peeling procedure

• Redesigned the experiment lists menus
• Redesigned the peeling functions
• Option to show spikes and burst length in view traces menu, now each burst in its own color
• Several spike features

• Burst length based on fluorescence analysis now is a lot more accurate
• New spike features were not being saved properly

• Skipped

• FCA menu
• View traces: menu to show the baseline (if saved)
• Option to sort traces by FCA order

• View traces: for the smooth traces, the dashed line now shows where the 0 fluorescence level should be
• View traces: redesigned the alignTraces option
• Added several new options for trace smoothing

• Option to see the new fluorescence-based spike burst size on the traces menu

• The spike features are now obtained from the spike inference menu (not inside view spikes)

• Automatic updater was not working

• Bug while importing quick_dev mat files from dcimg recordings
• Gui display on glia functions

• Button in the options window to revert any changes to the default values

• Several bugs with handling the new options classes
• Bugs in the preprocessing window

• Completeley redesigned the smoothing menu

• Compliance with semantic versioning - had to update to directly to 2.1.0 for compatibility issues

• Bug when importing HIS files with unstandard datatypes
• Bug when comparing experiments

• Help section, including acknowledgments and the changelog.
• Started the coding guidelines (see GUIIDELINES.md)
• Enabled the use of Markdown. This should speed up creating help sections and posterior documentation
• Several undocumented changes (check commits)

• Completey redesigned the options class and the optionsWindow menu. Not it has a lot more functionality
• Reestructured several menus
• Base netcal options are stored in an external json file, not a matlab object

• Progressbar can safely be closed (even if was already closed)

• Bunch of unused external dependencies

• Bug selecting points in the PCA plot.

• Work around to export features on the PCA analysis as separate bugs. There is a memory bug when creating too large xls files in there

• Bug when exporting aggreted experiments from the PCA analysis - due to the new trace structure

• Traces of individual cells can be selected on the View Recording menu by right clicking on top of the cell
• Spikes and bursts from the HCG can be seen on the View Recording menu from its menu
• Option to export a movie (and figure) with the current View Recording options

• Export option on the glia menu to match those on the view recording

• Bugs on some progress bars when the number of total iterations was below 100

• Option to do K-Means analysis of spike data with NaN values

• The znorm option in K-Means was not really doing a z-norm (only rescaling)

• New options to select specific features from the spikes
• New feature using the fluorescence trace to define burst length

• Generalized the feature list on several plots

• Current experiment was not being saved before doing batch PCA analysis

• Option to reset all training on the view traces menu

• Bug when selecting traces to train (caused by the new colormap)s

• Colormap selection on all menus now also show the colorbar

• Program now shows release notes of the latest 3 versions

• Spike features options are now passed like any other options
• Added temporal range to the spike features extraction

• Current experiment is no longer stored inside the current project
• Consistency check on finish ROI selection to delete old traces
• Cleanup of project files

• Fixed a bug when cloning current experiment
• Fixed a bug in the aggregated density menu

• Options are now saved both inside the experiment and the project files (not only for the project)
• Created base class for options windows
• Batch smoothing options
• Added K-means options on view PCA
• Added PCA export by features

• Progressbar support for multiple bars and color changes with completion, not randomly
• Changed all options classes to adhere to the new base class
• Merged analysis menus together
• Redesigned the batch process system

• Classifier class is no longer stored in the experiment

• Bug when cancelling export to xls
• No need to Add Gui handles when loading options classes

Undocumented. Check commits