Setup¶

Intro¶

Welcome to Spyglass, a DataJoint pipeline maintained by the Frank Lab at UCSF.

Spyglass will help you take an NWB file from raw data to analysis-ready preprocessed formats using DataJoint to (a) connect to a relational database (here, MySQL), and (b) automate processing steps. To use Spyglass, you'll need to ...

1. Set up your local environment
2. Connect to a database

Local environment¶

JupyterHub users can skip this step. Frank Lab members should first follow 'rec to nwb overview' steps on Google Drive to set up an ssh connection.

For local use, download and install ...

1. Python 3.9.
2. mamba as a replacement for conda. Spyglass installation is significantly faster with mamba.

   wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
   bash Miniforge3-$(uname)-$(uname -m).sh
   

3. VS Code with relevant python extensions, including Jupyter. Hold off on selecting your interpreter until after you make the environment with mamba.
4. git for downloading the repository, including notebooks.

In a terminal, ...

1. navigate to your project directory.
2. use git to download the Spyglass repository.
3. navigate to the newly downloaded directory.
4. create a mamba environment with either the standard environment.yml or the environment_position.yml, if you intend to use the full position pipeline. The latter will take longer to install.
5. open this notebook with VSCode

Commands for the steps above ...

cd /your/project/directory/ # 1
git clone https://github.com/LorenFrankLab/spyglass/ # 2
cd spyglass # 3
mamba env create -f environment.yml # 4
code notebooks/00_Setup.ipynb # 5

Note: Spyglass is also installable via pip and pypi with pip install spyglass-neuro, but downloading from GitHub will also download other files.

Next, within VSCode, select the kernel that matches your spyglass environment created with mamba. To use other Python interfaces, be sure to activate the environment: conda activate spyglass

See this guide for additional details on each of these programs and the role they play in using the pipeline.

Database¶

You have a few options for databases.

1. Connect to an existing database.
2. Run your own database with Docker
3. JupyterHub (coming soon...)

Your choice above should result in a set of credentials, including host name, host port, user name, and password. Note these for the next step.

Note for MySQL 8 users, including Frank Lab members

Using a MySQL 8 server, like the server hosted by the Frank Lab, will require the pre-release version of DataJoint to change one's password.

cd /location/for/datajoint/source/files/
git clone https://github.com/datajoint/datajoint-python
pip install ./datajoint-python

Existing Database¶

Connecting to an existing database will require a user name and password. Please contact your database administrator for this information.

Frank Lab members should contact Chris.

Running your own database with Docker¶

• First, install Docker.

• Add yourself to the docker group so that you don't have to be sudo to run docker.

• Download the docker image for datajoint/mysql:8.0.

  docker pull datajoint/mysql:8.0
  

• When run, this is referred to as a 'Docker container'

• Next start the container with a couple additional pieces of info...

  • Root password. We use tutorial.
  • Database name. Here, we use spyglass-db.
  • Port mapping. Here, we map 3306 across the local machine and container.

  docker run --name spyglass-db -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql:8.0
  

• For data to persist after terminating the container, attach a volume when running:

  docker volume create dj-vol
  docker run --name spyglass-db -v dj-vol:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql
  

Docker credentials are as follows:

• Host: localhost
• Password: tutorial
• User: root
• Port: 3306

Config and Connecting to the database¶

Spyglass can load settings from either a DataJoint config file (recommended) or environmental variables. The code below will generate a config file, but we first need to decide a 'base path'. This is generally the parent directory where the data will be stored, with subdirectories for raw, analysis, and other data folders. If they don't exist already, they will be created.

The function below will create a config file (~/.datajoint.config if global, ./dj_local_conf.json if local). Local is recommended for the notebooks, as each will start by loading this file. Custom json configs can be saved elsewhere, but will need to be loaded in startup with dj.config.load('your-path').

To point spyglass to a folder elsewhere (e.g., an external drive for waveform data), simply edit the json file. Note that the raw and analysis paths appear under both stores and custom.

In [ ]:

import os
from spyglass.settings import SpyglassConfig

# change to the root directory of the project
if os.path.basename(os.getcwd()) == "notebooks":
    os.chdir("..")

SpyglassConfig().save_dj_config(
    save_method="local",  # global or local
    base_dir="/path/like/stelmo/nwb/",
    database_user="your username",
    database_password="your password",  # remove this line for shared machines
    database_host="localhost or lmf-db.cin.ucsf.edu",
    database_port=3306,
    set_password=False,
)

import os from spyglass.settings import SpyglassConfig # change to the root directory of the project if os.path.basename(os.getcwd()) == "notebooks": os.chdir("..") SpyglassConfig().save_dj_config( save_method="local", # global or local base_dir="/path/like/stelmo/nwb/", database_user="your username", database_password="your password", # remove this line for shared machines database_host="localhost or lmf-db.cin.ucsf.edu", database_port=3306, set_password=False, )

If you used either a local or global save method, we can check the connection to the database with ...

In [ ]:

import datajoint as dj

dj.conn()  # test connection
dj.config  # check config

from spyglass.common import Nwbfile

Nwbfile()

import datajoint as dj dj.conn() # test connection dj.config # check config from spyglass.common import Nwbfile Nwbfile()

If you see an error saying Could not find SPYGLASS_BASE_DIR, try loading your config before importing Spyglass, try setting this as an environmental variable before importing Spyglass.

os.environ['SPYGLASS_BASE_DIR'] = '/your/base/path'

import spyglass
from spyglass.settings import SpyglassConfig
import datajoint as dj
print(SpyglassConfig().config)
dj.config.save_local() # or global

Up Next¶

Next, we'll try inserting data