Guide#

This guide focuses on our most common workflow (IF/iDISCO –> LSFM –> atlas registration –> voxel-wise stats –> cluster validation).
These video tutorials cover roughly similar content, but please refer to this guide and command help guides for additional info.
For getting started, test commands using these downsampled images from an example mouse.
- For instructions, see this UNRAVEL_commands.txt

If you are unfamiliar with the terminal, start here:
- Linux in 100 seconds
- Bash in 100 seconds
- Interactive Linux Survival guide
- Refer to common commands and tips in this cheat sheet (click to expand dropdown):

Terminal command cheat sheet

Working Directory

The working directory, or current directory, is the folder in the file system that a user or program is currently accessing.

Default Location: The terminal typically starts in your home directory.
Relative Paths: Commands use paths relative to the working directory.
Changing Directories: Use cd /path/to/directory to change the working directory.
Viewing the Directory: Use pwd to display the current working directory.
Importance: Knowing your working directory is often crucial for executing commands and managing files as intended.

Stopping Commands

Ctrl + C: Interrupt the command

Learning About Commands

Help Option: Many commands offer a --help option to provide a quick overview of usage. For example, ls --help or ls -h.
Manual Pages: Use man <command> to access the manual page for detailed information about a command. For example, man ls provides options and descriptions for the ls command.
which: Use which <command> to locate the executable file for a command. For example, which python shows where the Python interpreter is installed.

Listing Files

ls: List files and directories.
ls -l: List files in long format, including details like permissions, size, and modification date.
ls -a: List all files, including hidden files, which start with a “.” (unravel commands are logged in a hidden file [.command_log.txt]).
ls -1: List files in a single column.

Viewing and Creating Files

cat <filename>: Display the contents of a file.
echo "string" > new.txt: Create a new file named new.txt and write “string” to it.
echo "string" >> new.txt: Append “string” to the existing new.txt file.
touch new.txt: Create a new file that is empty

File/Folder Management

mkdir <dir_name>: Make directory
rm -rf <dir_name>: Delete folder and all contents recursively (be careful where you run this and what patterns you use)
rm <file> or rm <path/file>

Tab completion: Use tab to auto-complete file and directory names.
Up arrow: Scroll through command history. In zsh, type a partial command and press the up arrow to find past commands starting with the typed string.
cd <path>: Change the current directory to <path>. Drag/drop a file or folder into the terminal to paste the path to it. Or select the file/folder, hit Ctrl + C, select the terminal, and press Ctrl + Shift + V.
cd <Tab> or cd <Tab Tab>: To see folders in the working dir that you can cd to (with zsh use Tab and Shift + Tab to cycle –> enter or space)
cd ..: Move up one directory level.
cd ../..: Move up two directory levels.
cd ../dir: Move up one directory and then into dir.
cd -: Switch to the previous directory.
cd: Change to the home directory.
cd ~/: Change to the home directory using the tilde shortcut.

Command History

history: Print the history of terminal commands
!3: Run the third command in the history
cat .command_log.txt: Print the history of unravel commands
cat .command_log.txt | tail -10: Print the last 10 lines
cat .command_log.txt | head -10: Print the first 10 lines

Keyboard Shortcuts

Copy/Paste:
- Ctrl + Shift + C: Copy text
- Ctrl + Shift + V: Paste text.
- Or select text and click the middle wheel on the mouse to copy and paste.
Movement:
- Ctrl + A: Move to the beginning of the line.
- Ctrl + E: Move to the end of the line.
- Ctrl + Arrow: Move word by word.
Deleting:
- Ctrl + W: Delete the word before the cursor (space-separated for bash, additional delimiters for zsh).
- Ctrl + U: Delete from the cursor to the beginning of the line.
- Ctrl + K: Delete from the cursor to the end of the line.
- Ctrl + Y: Paste the last killed text.
Find:
- Ctrl + Shift + F: Open the search bar to find text.

Command Execution and Scripting

Using ; vs. &: ; chains commands to execute sequentially, while & runs a command in the background.
Another way to chain commands is to “pipe” (“|” character) the output from one command (what would be printed to the terminal) so it is used as the input of the next command
- For example, cat <filename> | wc -l: Cat would print the contents of , but here this is used as the input for wc, which counts the number of lines (e.g., rows in a csv)
Parentheses () for multi-line commands: Group commands to execute them in a subshell.

Global Variables

$PWD: Variable with the current working directory path.
$PATH: List of directories that have executable files. Allows for executing scripts without providing the full path (e.g., path/script.py –> script.py). Folders with UNRAVEL scripts don’t need to be added to the $PATH if it installed w/ pip.
See this guide to define common command arguments as global variables in env_var.sh (so they can be loaded in each terminal session)

Batch Processing Via Loops

For loops examples:
- for d in dir* ; do original_dir=$PWD ; cd $d ; pwd ; cd $original_dir ; done
- for s in sample?? ; do cat $s/parameters/metadata.txt ; done
- for d in <space separated list of experiment folder> ; do cd $d ; for s in sample?? ; do echo $s ; done ; done
- for i in 1 2 3 4; do echo $i ; done
- for i in {1..10}; do echo $i ; done
- Parallel processing w/ &: for i in *.nii.gz ; do fslmaths $i -bin ${i::-7}_bin -odt char & done # Binarizes all images, output data type 8-bit, ${i::-7} trims last 7 characters in bash (does not work in zsh) (run echo to see when processes are done).

Variables and Substitution

$PWD: Variable with the current working directory path.
$(): Command substitution to capture output for use in commands (e.g., x=$(for ))

File Searching and Manipulation

find: Search for files in a directory hierarchy.
- find . -name "*.txt": Search for text files matching this pattern.
- find $PWD -name "*.txt" -exec rm {} \;: Find all .txt files and delete them.
grep: Search text using patterns.
- grep -r <string_in_files>: Recursively search for a pattern
- grep -r --exclude-dir=docs <pattern>: Search for a pattern while excluding specific directories.
- grep -r <pattern1> | grep <pattern2>: Search for a pattern filter to preserve lines with a second string
- grep -r <pattern1> | grep -v <pattern2>: Search for a pattern filter to exlude lines with a second string
fzf: A command-line fuzzy finder that allows you to search and filter items interactively, providing a fast and efficient way to locate files, commands, or history entries.

Text Processing

sed: Stream editor for filtering and transforming text.
- sed 's/old/new/g' file: Replace all occurrences of old with new in file (use *.txt for multiple files)

Aliases

Aliases are shortcuts for longer commands or sequences of commands, making it easier and quicker to execute frequently used commands. You can create aliases in your shell configuration file (like .bashrc for Bash or .zshrc for Zsh).

How to Create Aliases:

Open your shell configuration file (.bashrc or .zshrc) with VS Code or nano
```
# For example:
nano ~/.bashrc  # For Bash users
```

Add alias definitions in the format:

alias shortname='full command'

For example:

alias ll='ls -l'
alias ilastik='path/to/ilastik_executable'  # For launching Ilastik via the terminal by running: ilastik
alias i="io_nii_info -i "
alias gs='git status'

Save the file and reload your configuration:
```
source ~/.bashrc  # For Bash users
```

FSL Commands for NIfTI files (.nii.gz extension)

fslmaths: Perform mathematical operations on images.
- add
- sub,
- bin (binarize)
- uthr (Zero out intenisties above upper threshold)
- thr (Zero out intenisties below threshold)
- mas (mask).
- Use -odt at end to set output data type (char for 8-bit and short for unsigned 16-bit). Default is float 32-bit. Pay attention to the max range that each data type can represent (e.g., 255 for 8-bit and 65,535 for 16-bit).
- “.nii.gz” is automatically added
- Examples:
- fslmaths img_1 -bin img_1_bin -odt char # char is for 8-bit
- fslmaths img_2 -mas img_1_bin -odt short # This used img_1_bin as a mask
- fslmaths img_1_bin -mul -1 -add 1 img_1_bin_inv -odt char # This inverts a mask
- fslmaths img_1 -sub img_2 diff # Then check if the images are the same: fslstats diff -R
- fslmaths img_1 -sub img_1 blank # Make an empty image for adding masks of each region
- for i in 1 56 672 ; do fslmaths atlas -thr $i -uthr $i region_${i} -odt short
- for i in region_mask*.nii.gz ; do fslmaths blank -add $i blank ; done ; mv blank.nii.gz all_regions.nii.gz
fslstats: Compute statistics on images (-R, -V, -M)
fslroi: Crop or expand images.
fslcpgeom: Copy header info (e.g., resolution and position in global space for viewing) from one image to another.

Killing a Process

If Ctrl + C doesn’t stop the command, you can kill the process:

Find the Process ID (PID):
- Use ps aux | grep <process_name> to find the PID of the process. Replace <process_name> with the name of the command or process you want to stop.
- Alternatively, use pgrep <process_name> to directly get the PID.
Kill the Process:
- Use kill <PID> to send a termination signal to the process. Replace <PID> with the actual process ID.
- If the process doesn’t stop, force it with kill -9 <PID>.

Command Help Guides#

For help on a command, run this in the terminal:

<command> -h

Help guides often include info on prereqs, inputs, outputs, next steps, usage, and other notes.
Viewing the terminal help provides default values and additional options not covered in this guide.
Provide -v when running commands for verbose mode (to see info about function calls, parameters, etc.)

Syntax related to commands

The symbols < and > indicate placeholders. Replace <command> with the actual command name you want to learn more about.
Square brackets [ ] in command syntax signify optional elements.
Double backticks are used in help guides to indicate a command. (e.g., ``command``)
Asterisks act as wildcards for matching patterns in strings
?? means two digits

To view help on arguments for each command in the online documentation, go to the page for that module/script, scroll to the parse_args() function, and click the link for viewing the source code.

Listing Commands#

# List common commands
unravel_commands -c

# List common commands, aliases, and descriptions:
uc -cad

# List all commands matching a string (e.g., vstats):
uc -ad -f vstats  # Find commands related to voxel-wise stats 

# List all commands and the modules/scripts that they run:
uc -m

Hint

Prefixes group together related commands. Use tab completion in the terminal to quickly view and access sets of commands within each group.

Set Up#

Recommended steps to set up for analysis:

Back Up Raw Data#

For Heifets lab members, we keep one copy of raw data on an external drive and another on a remote server (Dan and Austen have access)

Stitch Z-stacks#

We use ZEN (blue edition) since we have a Zeiss Lightsheet 7

Batch stitching settings

_images/batch_stitching_1.JPG — Select a mid-stack reference slice. Ideally, tissue will be present in each tile of the reference slice.#

Running batch stitching

Drag and drop images to be stitched into this section.
For each image in the list, apply the stitching settings one by one (do not go back to a prior image, as settings will no longer stick).
Select all images (control + A or shift and click)
Click “Check All”
Click “Run Selected”

Open source options for stitching

Make Sample Folders#

Make a folder named after each condition in the experiment folder(s)

. # This root level folder is referred to as the experiment directory (exp dir) 
├── Control
└── Treatment

Make sample folders in the directories named after each condition

Name sample folders like sample01, sample02, …

This makes batch processing easy.

Use a csv, Google sheet, or whatever else for linking sample IDs to IDs w/ this convention.

Other patterns (e.g., sample???) may be used (commands have a -p option for that).

.
├── Control
│   ├── sample01
│   └── sample02
└── Treatment
    ├── sample03
    └── sample04

Add Images to sample?? Directories#

For example, image.czi, image.h5, folder(s) with tif series, or an .ome.tif.

.
├── Control
│   ├── sample01
│   │   └── <raw/stitched image(s)>
│   └── sample02
│       └── <raw/stitched image(s)>
└── Treatment
    ├── sample03
    │   └── <raw/stitched image(s)>
    └── sample04
        └── <raw/stitched image(s)>

Data can be distributed across multiple drives

Paths to each experiment directory may be passed into scripts using the -d flag for batch processing

This is useful if there is not enough storage on a single drive.

Also, spreading data across ~2-4 external drives allows for faster parallel processing (minimizes i/o botlenecks)

If SSDs are used, distrubuting data may not speed up processing as much.

Log Paths, Commands, etc.#

Make an exp_notes.txt

This helps with keeping track of paths, commands, etc..

cd <path/to/dir/with/sample?? folders>
touch exp_notes.txt  # Make the .txt file

Automatic logging of scripts

Most scripts log the command used to run them, appending to ./.command_log.txt.

This is a hidden file. Use control+h to view it on Linux or command+shift+. to view it on MacOS.

# View command history for the current working directory
cat .command_log.txt  

# View last 10 lines
cat .command_log.txt | tail -10  

Make a sample_key.csv:#

It should have these columns:

dir_name,condition
sample01,control
sample02,treatment
…

Defining Common Variables for Command Arguments#

To simplify running commands, define common variables in a shell script (e.g., env_var.sh).

For example, $BASE is global variable representing the path to the main experiment folder, a place to aggregate analyses and sample?? folders
Example: $BASE represents the main experiment folder where analyses and sample?? folders (or condition folders with sample?? folders) are stored.
Source the script to load variables in each terminal session
Copy /UNRAVEL/unravel/env_var.sh to your experiment directory and update each variable with a code editor (e.g., VS Code or nano)
Using your editor, add this line to your terminal config file (.bashrc or .zshrc) to easily load variables:

alias exp=". /path/to/env_var.sh"  # Update the path. Use a short name for `exp`.

# Reopen the terminal or source your terminal config file to apply changes
. ~/.bashrc  # or . ~/.zshrc

# Run the alias before using commands
exp

# Echo a variable to see its value:
echo $DIRS  # This variable contains a space-separated list of paths to sample?? folders or directories containing them (for batch processing with `-d`)

# Example:
utils_get_samples -d $DIRS  # Finds all sample?? folders in the paths from $DIRS.

Optional: Clean TIFFs#

If raw data is in the form of a tif series, consider running:

utils_clean_tifs -t <path to directories with tifs relative to ./sample?? folders> -m

This will remove spaces from files names and move files other than *.tif to the parent directory

Analysis Overview#

Overview of a typical workflow (voxel-wise analyses followed by cluster validation):

flowchart TD A(((LSFM))) A --> B[Stitched 3D autofluorescence image] B --> C(Resample image to 50 µm resolution) C --> D(Segment tissue with Ilastik to mask external voxels) D --> E(Register image to an average template brain) A --> F[Stitched 3D immunofluorescence image] F --> G(Remove autofluorescence from IF images) G --> H(Warp IF images to atlas space) E --> H H --> I(Preprocess IF images: z-score, smooth, and average left/right sides) I --> J(Perform voxel-wise statistical analyses) J --> K(Apply FDR correction of 1-p value maps) K --> L(Warp significant voxel clusters back to tissue space) F --> N(Segment c-Fos+ cells or other features using Ilastik) L --> M(Validate clusters with cell/label density measurements) N --> M D --> I E --> L

Training Ilastik#

Ilastik has two purposes for our typical workflow:

During registration, Ilastik is used to segment tissue, creating brain masks that refine registration of autofluo images and z-scoring of IF images. After training Ilastik, segmentation of 50 µm autolfluo images is automated with seg_brain_mask.

During cluster validation, Ilastik labels c-Fos+ cells or other features of interest for quantification of cell or label densities within clusters from voxel-wise statistics. Segmentation of full-res IF images is automated with seg_ilastik.

Pixel classification documentation

Setting up Ilastik for batch processing

Install Ilastik

# Add this to your ~/.bashrc or ~/.zshrc terminal config file:
export PATH=/usr/local/ilastik-1.4.0.post1-Linux:$PATH   # Update the path and version

# Optional: add a shortcut command for launching Ilastik via the terminal
alias ilastik=run_ilastik.sh  # run_ilastik.sh could be replaced w/ the full path to the executable file

# Ilastik executable files for each OS:
#     - Linux and WSL: /usr/local/ilastik-1.4.0.post1-Linux/run_ilastik.sh
#     - Mac: /Applications/Ilastik.app/Contents/ilastik-release/run_ilastik.sh
#     - Windows: C:\Program Files\ilastik-1.3.3post3\run_ilastik.bat

# Source your terminal config file for these edits to take effect: 
. ~/.bashrc  # Or close and reopen the terminal

Training Ilastik

Launch Ilastik

Either double click on the application or run: ilastik (if you set up an alias)

Input Data
- Gather training slices to a folder from all samples with seg_copy_tifs
- Drag and drop training slices into the ilastik GUI (e.g., from a dir w/ 3 slices per sample and > 2 samples per condition)
- ctrl+A to select training slices -> right-click -> Edit shared properties -> Storage: Copy into project file -> Ok
Feature Selection
- Select Features… -> select all features (control+a) or an optimized subset (faster but less accurate).
- Optional: to find an optimal subset of features, select all features, train Ilastik, turn off Live Updates, click Suggest Features, select a subset, and refine training.
Training
- Brightness/contrast: select the gradient button and click and drag in the image (faster if zoomed in)
- Zoom: control/command + mouse wheel scroll, control/command + 2 finger scroll, or - and + (i.e., shift + =)
- Pan: shift + left click and drag or click mouse wheel and drag
- With label 1 and the brush tool selected, paint on c-Fos+ cells or another feature of interest
- With label 2 and the brush tool selected, paint on the background (e.g., any pixel that is not a cell)
- Turn on Live Update to preview pixel classification (faster if zoomed in) and refine training (e.g., if some cells are classified as background, paint more cells with label 1).
  - s will toggle the segmentation on and off.
  - p will toggle the prediction on and off.
  - Toggle eyes to show/hide layers and/or adjust transparency of layers.
- Change Current View to see other training slices. Check segmentation for these and refine as needed.
- Save the project in the experiment summary folder and close if using this script to run ilastik in headless mode for segmenting all images.
- Segment 50 µm autofluo images
- Segment immunofluo images to segment c-Fos+ cells or other features

Notes

If you want to go back to steps 1 & 2, turn Live Updates off
It is possible at add extra labels with a (e.g., if you want to segment somata with one label and axons with another label)
If you accidentally press a, turn off Live Updates and press x next to the extra label to delete it.
If the segmentation for label 1 fuses neighboring cells, draw a thin line in between them with label 2.

Registration#

flowchart TD A(((LSFM))) A --> B[Stitch z-stacks if necessary] B --> C(io_metadata: Save raw voxel dimentions in microns to ./sample??/parameters/metadata.txt) C --> D(reg_prep: Resample autofluorescence images to 50 µm resolution for registration) D --> E(seg_copy_tifs: Copy select slices from 50 µm autofl images to create brain masks using Ilastik) E --> F(Train Ilastik using pixel classification to segment autofl brains) F --> G(seg_brain_mask: Generate autofl brain masks with Ilastik and apply them to the 50 µm images) G --> H(Optionally use 3D Slicer to make the masked autofl image and mask better match the average template) H --> I(reg: Register the template brain with the masked autofl images and warp the atlas) I --> J(reg_check: Aggregate padded autofl images and warped atlas images from reg) J --> K(Assess registration quality in FSLeyes and refine if necessary by repeating the 3D Slicer step, etc.)

`io_metadata`#

unravel.image_io.metadata

Extract or specify x and z voxel sizes in microns (saves to ./sample??/parameters/metadata.txt).
You can add these resolutions to env_var.sh as global variables (e.g., $XY and $Z) and load them before running commands with: source path/to/env_var.sh
Guide on defining common command arguments in env_var.sh
For batch processing, run the io_metadata command from a directory containing sample?? folders, within a sample?? folder, or use -d to pass in a list of paths to sample folders or directories containing sample folders.

# To specify x and z voxel sizes in microns, use the -x and -z flags.
io_metadata -i <path to image or directory with TIFFs relative to sample??/> -x $XY -z $Z [-d $DIRS] # Remove square brackets from optional arguments
# If env_var.sh is not used, pass in the voxel sizes like this: -x 3.5232 -z 6.

# If -x and -z are omitted, io_metadata will attempt to extract this information from the image metadata.
io_metadata -i <rel_path/full_res_img> [-d $DIRS]
# 'rel_path' refers to the relative path from within the sample?? folders.
# Glob patterns can be used for -i (e.g., *.czi).

`reg_prep`#

unravel.register.reg_prep

Prepare autofluorescence images for registration (resample them to 50 µm isotropic resolution and save to ./sample??/reg_inputs/)

reg_prep -i <rel_path/image> [-d $DIRS] # -i options: tif_dir, .czi, .h5, .zarr, or .tif (e.g., *.czi)
# If using *.czi, by default the first channel is used (--channel 0), since that is usually the autofluo channel

`seg_copy_tifs`#

unravel.segment.copy_tifs

Copy resampled autofluo .tif files from each sample for making a brain mask with ilastik
In the example below -s 0000 0005 0050 means that the 1st, 6th, and 51st tif will be copied to the target directory
The target directory is the current working directory unless -td is used to specify an output path.

seg_copy_tifs -i reg_inputs/autofl_??um_tifs -s 0000 0005 0050 [-td brain_mask] [-d $DIRS]  

Train Ilastik to segment autofluo tissue to make a brain mask

`seg_brain_mask`#

unravel.segment.brain_mask

Makes reg_inputs/autofl_??um_brain_mask.nii.gz and reg_inputs/autofl_??um_masked.nii.gz for reg

seg_brain_mask -ie <path/ilastik_executable> -ilp <path/trained_ilastik_project.ilp> [-d $DIRS]

Making the autofluorescence image better match the average template image

seg_brain_mask zeros out voxels outside of the brain. This prevents the average template (moving image) from being pulled outward during registration (reg).
If non-zero voxles outside the brain remain and are affecting reg quality, use 3D slicer to zero them out by painting in 3D (segmentation module).
If there is missing tissue, use 3D slicer to fill in gaps.

`reg`#

unravel.register.reg

Register an average template brain/atlas to a resampled autofluo brain.

Determining the 3 letter orientation code

Letter options:
- A/P=Anterior/Posterior
- L/R=Left/Right
- S/I=Superior/Interior
Letter order:
- The side of the brain at the positive direction of the x, y, and z axes determines the 3 letters
- Letter 1: Side of the brain right of z-stack
- Letter 2: Side of the brain facing bottom of z-stack
- Letter 3: Side of the brain facing back of z-stack

reg -m <path/template.nii.gz> -bc -sm 0.4 -ort <3 letter orientation code> -m2 <path/atlas_CCFv3_2020_30um.nii.gz> [-d $DIRS]

# Example if using env_var.sh (assuming an RPS orientation):
reg -m $TEMPLATE -bc -pad -sm 0.4 -ort RPS -a $ATLAS -d $DIRS

# -bc performs N4 bias field correction to make image intensities more uniform
# -sm smooths the autofluo image by the amount specified
# If you want to use an unmasked autofluo image, provide: -mas None -f reg_inputs/autofl_50um.nii.gz. Other commands alos assume that a masked autofluo image was used, so check each help guide for default arguments.

If sample orientations vary

Make a ./sample??/parameters/ort.txt with the 3 letter orientation for each sample and run:

for d in $DIRS ; do cd $d ; for s in sample?? ; do reg -m $TEMPLATE -bc -pad -sm 0.4 -ort $(cat $s/parameters/ort.txt) -a $ATLAS -v -d $PWD/$s ; done ; done 

`reg_check`#

unravel.register.reg_check

Check registration by copying these images from each sample??/reg_ouputs folder to a target directory:
- autofl_??um_masked_fixed_reg_input.nii.gz
- atlas_in_tissue_space.nii.gz

reg_check [-td reg_results] [-d $DIRS]  # Default for -td: copy images to the current dir.
# The default warped atlas from reg is atlas_CCFv3_2020_30um_in_tissue_space.nii.gz (in ./sample??/.reg_outputs). Use -wa <image_name.nii.gz> to set this.

View these images with FSLeyes docs
FSLeyes might not work in Windows without additional set up: Installation on Windows

Checking registration with FSLeyes

_images/FSLeyes_autofl_image_from_reg.jpg — Use FSLeyes to view the autofluo image from `reg` (sample??/reg_ouputs/autofl_50um_masked_fixed_reg_input.nii.gz).#

_images/FSLeyes_atlas_warped_to_tissue_from_reg.jpg — Use FSLeyes to view the atlas warped to the the tissue (sample??/reg_ouputs/atlas_CCFv3_2020_30um_in_tissue_space.nii.gz)#

Setting up Allen brain atlas coloring in FSLeyes

Add UNRAVEL/_other/fsleyes_luts/ccfv3_2020.lut to the following location:
- Linux: /home/<your_username>/.config/fsleyes/luts/ccfv3_2020.lut
- MacOS: /usr/local/fsl/fslpython/envs/fslpython/lib/python3.8/site-packages/fsleyes/assets/luts/ccfv3_2020.lut
- Windows: C:\Users\<your_username>\AppData\Roaming\fsleyes\luts\ccfv3_2020.lut
On MacOS, run the following command to add write permissions to the LUT folder:

sudo chmod a+w /usr/local/fsl/fslpython/envs/fslpython/lib/python3.8/site-packages/fsleyes/assets/luts

On MacOS, edit order.txt to include ccfv3_2020.lut.
To remove other LUTs, move them up a directory level, but keep random.lut in place.
Open atlas_CCFv3_2020_30um.nii.gz in FSLeyes.
Select the atlas and change “3D/4D volume” to “Label image.”
Switch the lookup table from random to ccfv3_2020.
Click the circle icon at the top to convert the atlas to a wireframe view.
If you don’t want to select the LUT every time, make a copy of random.lut and replace its contents with those of ccfv3_2020.lut.

Voxel-wise Statistics#

flowchart TD A(((LSFM))) A --> B[Stitch z-stacks if required] B --> C(vstats_prep: Optionally remove autofluorescence and warp immunofluo images to atlas space) C --> D(vstats_z_score: Optionally z-score IF images from each brain individually) D --> E(utils_agg_files: Aggregate atlas-space IF images from all sample folders into the current directory) E --> F(vstats_whole_to_avg: Optionally smooth image and average left/right sides) F --> G(Check inputs for vstats: Open IF images and the atlas in FSLeyes to confirm aligment with the atlas and other IF images) G --> H(Prep inputs for vstats: Add IF images to the current directory, without other .nii.gz files) H --> I(utils_prepend: Prepend a one-word condition to each IF image) I --> J(For ANOVA designs, run fsl and set up the design. If two groups are present, proceed to the next step) J --> K(vstats: Run voxel-wise analyses using FSL's randomise tool) K --> L(View 1-p value maps in FSLeyes and consult 'Cluster-wise stats' for multiple comparisons correction and cluster validation)

`vstats_prep`#

unravel.voxel_stats.vstats_prep

Preprocess immunofluo images and warp them to atlas space for voxel-wise statistics.

# Example with rolling ball background subtration (pixel radius of 4):
vstats_prep -i cfos -o cFos_rb4_atlas_space.nii.gz -rb 4 -a $ATLAS [-d $DIRS]

# Usage:
vstats_prep -i <rel_path/image> -o cFos_rb4_atlas_space.nii.gz -a $ATLAS [-sa 3] [-rb 4] [--channel 1] [-d $DIRS]
# --channel is for .czi images (1 is the second channel)

Info on background subtraction

Removing autofluorescence from immunolabeling improves the sensitivity of voxel-wise comparisons.
Use -sa 3 for 3x3x3 spatial averaging if there is notable noise from voxel to voxel.
Use -rb for rolling ball background subtraction.
Use a smaller rolling ball radius if you want to preserve punctate signal like c-Fos+ nuclei (e.g., 4)
Use a larger rolling ball radius if you want to preserve more diffuse signal (e.g., 20).
The radius should be similar to the largest feature that you want to preserve.

You can test parameters for background subtraction with:

unravel.image_tools.spatial_averaging
unravel.image_tools.rb
- Copy a tif to a test dir for this (e.g., with seg_copy_tifs)
- Use unravel.image_io.io_img to create a tif series

_images/FSLeyes_Ai14_image_in_CCFv3_30um_space.jpg — Use FSLeyes to view the fluorescently labeled image in atlas space.#

`vstats_z_score`#

unravel.voxel_stats.z_score

Z-score atlas space images using tissue masks (from brain_mask) and/or an atlas mask.

# Z-scoring using the brain mask (-i and -tmas paths are relative to the sample?? folder)
vstats_z_score -i atlas_space/sample??_cFos_rb4_atlas_space.nii.gz [-d $DIRS] 

# Z-scoring using an atlas mask
vstats_z_score -i 'atlas_space/*.nii.gz' -amas $MASK [-d $DIRS] 

# Using a tissue mask and an atlas mask
vstats_z_score -i 'atlas_space/*.nii.gz' -tmas reg_inputs/autofl_50um_brain_mask.nii.gz -amas $MASK [-d $DIRS] 

`utils_agg_files`#

unravel.utilities.aggregate_files_from_sample_dirs

Aggregate pre-processed immunofluorescence (IF) images for voxel-wise stats

utils_agg_files -i atlas_space/*_cFos_rb4_atlas_space_z.nii.gz [-td path/target_dir] [-d $DIRS]

`vstats_whole_to_avg`#

unravel.voxel_stats.whole_to_LR_avg

Smooth and average left and right hemispheres together

# Run this in the folder with the .nii.gz images to process
vstats_whole_to_avg -k 0.1 --parallel -v  # A 0.05 mm - 0.1 mm kernel radius is recommended for smoothing

# Process all *.nii.gz files in the current directory with no smoothing:
vstats_whole_to_avg [-tp] [-amas $MASK] # Provide -tp for parallel processing.

# Process files matching the pattern with smoothing:
vstats_whole_to_avg -i '*_cFos_rb4_atlas_space_z.nii.gz.nii.gz' -k 0.1 [-tp] [-amas $MASK]
# A 0.1 mm kernel radius is recommended for smoothing (-k)

`vstats`#

unravel.voxel_stats.vstats

Run voxel-wise stats using FSL’s randomise_parallel command
Outputs saved in stats/

# Run this in the folder with the input IF images (the current directory name will be prepended to outputs)
vstats -mas $MASK -a $ATLAS [-p 18000] [-k 0.1]
# -p sets the number of permutations.
# -k is for smoothing if inputs have not yet been smoothed (-k is the kernel size in mm)
# See the vstats help for providing additional arguments to FSL's randomise_parallel

Outputs from voxel-wise stats

Outputs in ./stats/
- vox_p maps are uncorrected 1-p value maps
- tstat1: group1 > group2
- tstat2: group2 > group1
- fstat*: f contrast w/ ANOVA design (non-directional p value maps)
- fstat1 corresponds to the first Contrast defined in step 5 above.

Cluster-wise Statistics#

flowchart TD A(vstats: Generates 1-p value maps with vox_p in the filename) A --> B(cstats_fdr_range: Input a 1-p map to find FDR q values that yield clusters) B --> C(cstats_fdr: Apply FDR correction on the 1-p map to identify clusters of significant voxels) C --> D(cstats_mirror_indices: For whole-brains, recursively mirror cluster maps in ./stats/) D --> E(seg_copy_tifs: Copy select full resolution TIFFs for Ilastik training) E --> F(seg_ilastik: Perform pixel classification with a trained Ilastik project) F --> G(cstats_validation: Warp clusters from atlas to full resolution tissue space, count cells, and calculate cluster volumes) G --> H(cstats_summary: Aggregate and analyze cluster validation data)

False Discovery Rate (FDR) Correction#

`cstats_fdr_range`#

unravel.cluster_stats.fdr_range

Outputs a list of FDR q values that yield clusters.

cstats_fdr_range -i <path/vox_p_tstat1.nii.gz> -mas $MASK

`cstats_fdr`#

unravel.cluster_stats.fdr

Perform FDR correction on a 1-p value map to define clusters
Outputs are saved to a new folder in stats/ named after the input image and q value

cstats_fdr -i <path/vox_p_tstat1.nii.gz> -mas $MASK -q 0.05 0.01 0.001 [-ms 100] 
# For -q, copy the output from cstats_fdr_range
# For -ms, set the number of voxels for a cluster (e.g., 100 for c-Fos or 400 if labeling is sparse)

# Perform FDR correction on multiple directional 1-p value maps
for j in *_vox_p_*.nii.gz ; do q_values=$(cstats_fdr_range -mas $MASK -i $j) ; cstats_fdr -mas $MASK -i $j -q $q_values ; done

# ANOVAs output non-directioanl 1-p value maps. Optionally convert them into a directional cluster index:
q_values=$(cstats_fdr_range -i vox_p_fstat1.nii.gz -mas $MASK) ; cstats_fdr -i vox_p_fstat1.nii.gz -mas $MASK -o fstat1 -v -a1 Control_avg.nii.gz -a2 Deep_avg.nii.gz -q $q_values
# Make averaged images for each group (-a1 and -a2 args) with the img_avg command

`cstats_mirror_indices`#

unravel.cluster_stats.recursively_mirror_rev_cluster_indices

Recursively flip the content of rev_cluster_index.nii.gz images (to validate w/ clusters in left and right hemispheres)
Run this in the ./stats/ folder to process all subdirs with reverse cluster maps (cluster IDs go from large to small)

# Use -m RH if a right hemisphere mask was used (otherwise use -m LH)
cstats_mirror_indices -m RH [-i glob_pattern]

Segmentation of Full-Resolution Fluorescence Images#

1) `seg_copy_tifs`#

unravel.segment.copy_tifs

Copy or extract full res tif files to a target dir for training Ilastik to segment labels of interest

Tip

Copy/extract 3 slices from each sample, with at least 3 samples per condition.

seg_copy_tifs -i <rel_path/raw_image> -s 0100 0500 1000 [-td ilastik_segmentation] [-d $DIRS]
# Default for target dir (-td) is current dir
# The name of the dir with full-res tifs can be passed in for -i

2) Train Ilastik to segment c-Fos+ cells or other features#

3) `seg_ilastik`#

unravel.segment.ilastik_pixel_classification

Segment features of interest (e.g., c-Fos+ cells) in full-resulution images using a trained Ilastik project (pixel classification)

seg_ilastik -ie <path/ilastik_executable> -ilp <path/ilastik_project.ilp> -i <rel_path to tif_dir or image> -o seg_dir --labels 1 [--rm_out_tifs] [--channel 1] [-d $DIRS]
# --labels is a space-separated list of labels to convert into binary .nii.gz images.
# --channel is for .czi images (1 is the second channel)

Cluster Validation and Statistics#

flowchart TD A(cstats_org_data: Organize cell/label density data from cluster validation) A --> B(cstats_group_data: Pool left and right hemisphere data) B --> C(utils_prepend: Prepend CSV names with conditions) C --> D(cstats: Run statistics to determine cluster validity) D --> E(cstats_index: Create valid cluster maps and CSVs for sunburst plots) E --> F(cstats_brain_model: Create files for 3D brain models) F --> G(cstats_table: Create tables summarizing top regions and volumes) G --> H(cstats_prism: Generate CSVs for plotting bar graphs with Prism) H --> I(cstats_legend: Create legend files defining region abbreviations) I --> J(Summarize cluster validation rates, aggregate files for 3D brains, compile SI tables, etc.)

`cstats_validation`#

unravel.cluster_stats.validation

Warps cluster index from atlas space to tissue space, crops clusters, applies segmentation mask, and quantifies cell or label densities

# Cell density measurements in all clusters
cstats_validation -m <path/rev_cluster_index_to_warp_from_atlas_space.nii.gz> -s  <rel_path/seg_img.nii.gz> [-d $DIRS]

# Label density measurements in select clusters
cstats_validation -m <path/rev_cluster_index_to_warp_from_atlas_space.nii.gz> -s  <rel_path/seg_img.nii.gz> -de label_density -c 1 3 4 [-d $DIRS]

# Use -n to save the cluster map in tissue space (-n rel_path/native_cluster_index.zarr). Slower than w/o saving.
# -inp nearestNeighbor is default, but for smoother clusters use multiLabel (slower)

# Processing multiple FDR q value thresholds and both hemispheres (assumes that cstats_mirror_indices has been run):
for q in 0.005 0.01 0.05 0.1 ; do for side in LH RH ; do cstats_validation  -m path/vstats/contrast/stats/contrast_vox_p_tstat1_q${q}/contrast_vox_p_tstat1_q${q}_rev_cluster_index_${side}.nii.gz -s rel_path/seg_img.nii.gz [-d $DIRS] ; done ; done

`cstats_summary`#

unravel.cluster_stats.summary

Aggregates and analyzes cluster validation data from cstats_validation
Update parameters in /UNRAVEL/unravel/cstats/cluster_summary.ini and save it with the experiment
group1 and group2 must match conditions in the sample_key.csv

# Usage if running directly after ``cstats_validation``
cstats_summary -c <path/cluster_summary.ini> -cvd 'psilocybin_v_saline_tstat1_q*' -vd <path/vstats_dir> -sk $SAMPLE_KEY --groups <group1> <group2> -hg <higher_group> [-d $DIRS]

# Usage if running after ``cstats_validation`` and ``cstats_org_data``:
cstats_summary -c <path/cluster_summary.ini> -sk $SAMPLE_KEY  --groups <group1> <group2> -hg <higher_group> [-d $DIRS]

# See the help guide for cstats_validation if you want to pool data using condition prefixes

Sunburst Plots#

Sunburst plots of regional volumes

CSVs for making sunburst plots are output by cstats_validation for each valid cluster map.
Output location: main_experiment_dir/cstats/<vstats_contrast>_q/_valid_clusters/valid_clusters_sunburst.csv
Or sunburst CSVs can be generated using cstats_sunburst
Sunburst CSVs can be copy and pasted into the sunburst plot tool of the Flourish web app

Make a free account w/ Flourish.
Login.
Select “New visualization”.

_images/sunburst_2_plot_types.jpg — Under “Hierarchy” select “Sunburst”#

Select the “Data” tab.
Set it up like this with columns A-K. Categories/nesting: B-J, Size by: K.
Always delete the entirety of the columns before pasting in values.
Paste in values for columns A-K from a *_sunburst.csv.
Switch to the “Preview” tab to view the plot.

Paste the contents of sunburst_RGBs.csv into Colors –> Custom overrides.
Location: UNRAVEL/unravel/core/csvs/sunburst_RGBs.
cstats_sunburst can output it w/ -rgb.
Set Min font size under “Labels” to 0.5 to show labels.
Increase Min font size above the Max size to hide them.
Take a screenshot of the plot to save it.

3D Brain Models#

Visualize cluster maps, etc. in a 3D brain

Files for making 3D brains in DSI Studio are output by cstats_validation for each valid cluster map.
Output location: main_experiment_dir/cstats/3D_brains/
Alternate location: main_experiment_dir/cstats/_q/_valid_clusters/
Output image: _q_rev_cluster_index__valid_clusters_ABA_WB.nii.gz
Output look up table: _q_rev_cluster_index__valid_clusters_rgba.txt
These files can be generated with cstats_brain_model (use -m for a bilateral representation of a unilateral map)

Visualization in DSI Studio

Open DSI studio
Click on “Step T3”
Change the drop down to select all files and select the binary atlas (mask_CCFv3_2020_30um.nii.gz)
If you open the binary atlas in the 3D_brains folder, it is easier to open other files from there later
Make it visible: Slices –> Add Isosurface –> Full –> OK –> Zoom out (mouse wheel or 2 finger scroll)

Display settings

Layout

The zoom and viewer dimensions determine the output video size.
This layout allows for 1080p videos with my MacBook (adjust as needed).
Move panels on the right down and make them as small as possible.
Line up the left edge of the viewer with View.
Zoom to the view 0.28 (lower right of viewer)
Zoom out to 0.18 for axial.

Adding Regions and Color

Regions –> Open Region… –> load _q_rev_cluster_index__valid_clusters_ABA_WB.nii.gz
Regions Misc –> Load Region Color… –> load _q_rev_cluster_index__valid_clusters_rgba.txt

Making Figures

Use z, x, and c to snap to specific orientations.
Use View –> Save Camera View… to save an oblique orientation
Load it with: View –> Open Camera View…
Capture screenshots

Recording Videos

For a video: press z for a sagittal orientation (e.g., facing to the right), View –> Save Rotation Video/Images…
The output .avi is large, but can be exported in another format
For example, in Adobe Premiere Pro –> add the video to the timeline –> right click to double video speed –> export with SD resolution (480p) –> scale to fit –> export.
Composite videos with multiple cluster maps can be made with Adobe Premiere Pro using the exported videos.

Valid Cluster Info Tables#

Cluster info .xlsx tables are output by cstats_validation for each valid cluster map.
Output location: valid_clusters_tables_and_legend
Alternatively, they can be made with cstats_table and cstats_legend
Compile each _q_valid_clusters_table.xlsx and legend.xlsx into one .xlsx SI table
Example SI Table from our initial UNRAVEL paper

Region-wise Statistics#

flowchart TD A(((LSFM))) A --> B[3D autofluorescence image] B --> C(Registration to an average template brain) C --> D(Optional: warp_to_native: Warp the atlas to full-resolution tissue space and save as .zarr) A --> E[3D immunofluorescence image] E --> F(seg_copy_tifs & seg_ilastik: Segment c-Fos+ cells using Ilastik) F --> G(rstats: Perform regional cell counting and volume measurements) D --> G G --> H(utils_agg_files: Aggregate CSV outputs from rstats across all sample folders) H --> I(rstats_summary: Plot cell densities by region and summarize results)

`rstats`#

unravel.region_stats.regional_cell_densities

Quantify regional cell densities or label densities

# Usage if the atlas is already in native space from ``warp_to_native``:
rstats -s <rel_path/segmentation_image.nii.gz> -a <rel_path/native_atlas_split.nii.gz> -c Saline -t cell_densities [-d <paths to sample folders in the Saline group or the folders containing them>]

# Usage if the native atlas is not available; it is not saved (faster):
rstats -s rel_path/segmentation_image.nii.gz -m $SPLIT -c Saline -t cell_densities [-d <paths to sample folders in the Saline group or the folders containing them>]

Aggregate outputs from rstats with utils_agg_files.

`rstats_summary`#

unravel.region_stats.regional_cell_densities_summary

Plot cell densities for each region and summarize results.
CSVs from rstats should be in the current directory
CSV columns:
- Region_ID,Side,Name,Abbr,Saline_sample06,Saline_sample07,…,MDMA_sample01,…,Meth_sample23,…

# Usage for Tukey tests:
rstats_summary --groups Saline MDMA Meth -hemi <both l or r> [-y cell_density] [-div 10000]
# -div 10000 divides cells per cubic mm by 10000 (for plotting)
# -y is the y-axis label

# Usage for t-tests:
rstats_summary --groups Saline MDMA -hemi <both l or r>  -t t-test -c Saline [-alt two-sided] [-y cell_density] [-div 10000] 

Example sample?? Folder Structure#

.
├── atlas_space  # Dir with images warped to atlas space
├── cfos_seg_ilastik_1  # Example dir with segmentations from ilastik
├── clusters  # Dir with cell/label density CSVs from cstats_validation
│   ├── Control_v_Treatment_vox_p_tstat1_q0.005
│   │   └── cell_density_data.csv
│   └── Control_v_Treatment_vox_p_tstat2_q0.05
│       └── cell_density_data.csv
├── parameters  # Optional dir for things like metadata.txt
├── reg_inputs  # From reg_prep (autofl image resampled for reg) and seg_brain_mask (mask, masked autofl)
├── regional_cell_densities  # CSVs with regional cell densities data
├── reg_outputs  # Outputs from reg. These images are typically padded w/ empty voxels. 
└── image.czi # Or other raw/stitched image type

Example Experiment Folder Structure#

.
├── exp_notes.txt
├── env_var.sh
├── sample_key.csv
├── Control
│   ├── sample01
│   └── sample02
├── Treatment
│   ├── sample03
│   └── sample04
├── atlas
│   ├── atlas_CCFv3_2020_30um.nii.gz  # Each atlas region has a unique intensity/ID
│   ├── atlas_CCFv3_2020_30um_split.nii.gz  # Intensities in the left hemisphere are increased by 20,000
│   ├── average_template_CCFv3_30um.nii.gz  # Average template brain that is aligned with the atlas
│   └── mask_CCFv3_2020_30um_RH_wo_root_ventricles_fibers_OB.nii.gz  # Right hemisphere mask that excludes undefined regions (root), ventricles, fiber tracts, and the olfactory bulb
├── check_reg  # run check_reg from here
├── brain_mask
│   ├── brain_mask.ilp  # Ilastik project trained with the pixel classification workflow to segment the brain in resampled autofluo images
│   ├── sample01_slice_0000.tif
│   ├── sample01_slice_0005.tif
│   ├── sample01_slice_0050.tif
│   ├── ...
│   └── sample04_slice_0050.tif
├── vstats  # Voxel-wise stats
│   └── Control_v_Treatment 
│       ├── Control_sample01_rb4_atlas_space_z.tif
│       ├── Control_sample02_rb4_atlas_space_z.tif
│       ├── Treatment_sample03_rb4_atlas_space_z.tif
│       ├── Treatment_sample04_rb4_atlas_space_z.tif
│       └── stats 
│           ├── Control_v_Treatment_vox_p_tstat1.nii.gz  # 1 minus p value map showing where Control (group 1) > Treatment (group2)
│           ├── Control_v_Treatment_vox_p_tstat2.nii.gz  # 1-p value map showing where Treatment (group 2) > Control (group 1)
│           ├── Control_v_Treatment_vox_p_tstat1_q0.005  # cluster correction folder
│           │   ├── 1-p_value_threshold.txt  # FDR adjusted 1-p value threshold for the uncorrected 1-p value map
│           │   ├── p_value_threshold.txt  # FDR adjusted p value threshold
│           │   ├── min_cluster_size_in_voxels.txt  # Often 100 voxels for c-Fos. For sparser signals like amyloid beta plaques, consider 400 or more
│           │   └── ..._rev_cluster_index.nii.gz # cluster map (index) with cluster IDs going from large to small (input for cstats_validation)
│           ├── ...
│           └── Control_v_Treatment_vox_p_tstat2_q0.05
├── cstats  # cluster stats (validation)
│   ├── Control_v_Treatment_vox_p_tstat1_q0.005
│   │   ├── Control_sample01_cell_density_data.csv
│   │   ├── Control_sample02_cell_density_data.csv
│   │   ├── Treatment_sample03_cell_density_data.csv
│   │   ├── Treatment_sample04_cell_density_data.csv
│   │   ├── _valid_clusters  # Contains data for sunburst plots, 3D brains, the xlsx table, etc.
│   │   ├── _valid_cluster_stats  # Contains t-test_results.csv for adding asterisks to the xlsx table, etc.
│   │   ├── _valid_clusters_prism  # For making valid cluster bar graphs with GraphPad Prism. Clusters are sorted by anatomy (refer to the xlsx table for annotating it)
│   │   └── p_value_threshold.txt
│   ├── 3D_brains  # Use these for vizualization in DSI Studio
│   ├── valid_clusters_tables_and_legend  # Use these .xlsx tables to make an SI table
│   ├── cluster_validation_summary_t-test.csv  # Use this to summarize cluster validation rates, etc.
│   ├── ...
│   └── Control_v_Treatment_vox_p_tstat2_q0.05
└── rstats # regional stats
    ├── Control_sample01_regional_cell_densities.csv
    ├── ...
    ├── Treatment_sample04_regional_cell_densities.csv
    ├── regional_cell_densities_all.csv
    └── Folders with plots for each region

Guide#

Command Help Guides#

Listing Commands#

Common Commands#

Set Up#

Back Up Raw Data#

Stitch Z-stacks#

Make Sample Folders#

Add Images to sample?? Directories#

Log Paths, Commands, etc.#

Make a sample_key.csv:#

Defining Common Variables for Command Arguments#

Optional: Clean TIFFs#

Analysis Overview#

Training Ilastik#

Registration#

io_metadata#

reg_prep#

seg_copy_tifs#

seg_brain_mask#

reg#

reg_check#

Voxel-wise Statistics#

vstats_prep#

vstats_z_score#

utils_agg_files#

vstats_whole_to_avg#

vstats#