Spike Sorting Analysis¶

Sorted spike times are a starting point of many analysis pipelines. Spyglass provides several tools to aid in organizing spikesorting results and tracking annotations across multiple analyses depending on this data.

For practical examples see Sorted Spikes Decoding

SortedSpikesGroup¶

In practice, downstream analyses of spikesorting will often need to combine results from multiple sorts (e.g. across tetrodes groups in a single interval). To make this simple with spyglass's relational database, we use the SortedSpikesGroup table.

SortedSpikesGroup is a child table of SpikeSortingOutput in the spikesorting pipeline. It allows us to group the spikesorting results from multiple sources into a single entry for downstream reference, and provides tools for easily accessing the compiled data. Here we will group together the spiking of multiple tetrode groups.

This table allows us filter units by their annotation labels from curation (e.g only include units labeled "good", exclude units labeled "noise") by defining parameters from UnitSelectionParams. When accessing data through SortedSpikesGroup the table will include only units with at least one label in include_labels and no labels in exclude_labels. We can look at those here:

In [2]:

Copied!





from spyglass.spikesorting.analysis.v1.group import UnitSelectionParams

UnitSelectionParams().insert_default()

# look at the filter set we'll use here
unit_filter_params_name = "default_exclusion"
print(
    (
        UnitSelectionParams()
        & {"unit_filter_params_name": unit_filter_params_name}
    ).fetch1()
)
# look at full table
UnitSelectionParams()
from spyglass.spikesorting.analysis.v1.group import UnitSelectionParams

UnitSelectionParams().insert_default()

# look at the filter set we'll use here
unit_filter_params_name = "default_exclusion"
print(
    (
        UnitSelectionParams()
        & {"unit_filter_params_name": unit_filter_params_name}
    ).fetch1()
)
# look at full table
UnitSelectionParams()

{'unit_filter_params_name': 'default_exclusion', 'include_labels': [], 'exclude_labels': ['noise', 'mua']}

Out[2]:

unit_filter_params_name	include_labels	exclude_labels
all_units	=BLOB=	=BLOB=
default_exclusion	=BLOB=	=BLOB=
exclude_noise	=BLOB=	=BLOB=
MS2220180629	=BLOB=	=BLOB=

Total: 4

We then define the set of curated sorting results to include in the group

Finding the merge id's corresponding to an interpretable restriction such as merge_id or interval_list can require several join steps with upstream tables. To simplify this process we can use the included helper function SpikeSortingOutput().get_restricted_merge_ids() to perform the necessary joins and return the matching merge id's

In [3]:

Copied!





from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput

nwb_file_name = "mediumnwb20230802_.nwb"

sorter_keys = {
    "nwb_file_name": nwb_file_name,
    "sorter": "mountainsort4",
    "curation_id": 1,
}

# get the merge_ids for the selected sorting
spikesorting_merge_ids = SpikeSortingOutput().get_restricted_merge_ids(
    sorter_keys, restrict_by_artifact=True
)

keys = [{"merge_id": merge_id} for merge_id in spikesorting_merge_ids]
(SpikeSortingOutput.CurationV1 & keys)
from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput

nwb_file_name = "mediumnwb20230802_.nwb"

sorter_keys = {
    "nwb_file_name": nwb_file_name,
    "sorter": "mountainsort4",
    "curation_id": 1,
}

# get the merge_ids for the selected sorting
spikesorting_merge_ids = SpikeSortingOutput().get_restricted_merge_ids(
    sorter_keys, restrict_by_artifact=True
)

keys = [{"merge_id": merge_id} for merge_id in spikesorting_merge_ids]
(SpikeSortingOutput.CurationV1 & keys)

[12:11:56][WARNING] Spyglass: V0 requires artifact restrict. Ignoring "restrict_by_artifact" flag.

Out[3]:

merge_id	sorting_id	curation_id
143dff79-3779-c0d2-46fe-7c5040404219	a4b5a94d-ba41-4634-92d0-1d31c9daa913	1
2249c566-cc17-bdda-4074-d772ee40b772	874775be-df0f-4850-8f88-59ba1bbead89	1
4a191cc4-945b-3ad8-592a-a95e874b2507	a4b5a94d-ba41-4634-92d0-1d31c9daa913	1
75286bf3-f876-4550-f235-321f2a7badef	642242ff-5f0e-45a2-bcc1-ca681f37b4a3	1
76ec4894-300d-4ed3-ce26-0327e7ed3345	642242ff-5f0e-45a2-bcc1-ca681f37b4a3	1
a900c1c8-909d-e583-c377-e98c4f0deebf	874775be-df0f-4850-8f88-59ba1bbead89	1

Total: 6

We can now combine this information to make a spike sorting group

In [9]:

Copied!





from spyglass.spikesorting.analysis.v1.group import SortedSpikesGroup

# create a new sorted spikes group
unit_filter_params_name = "default_exclusion"
SortedSpikesGroup().create_group(
    group_name="demo_group",
    nwb_file_name=nwb_file_name,
    keys=[
        {"spikesorting_merge_id": merge_id}
        for merge_id in spikesorting_merge_ids
    ],
    unit_filter_params_name=unit_filter_params_name,
)
# check the new group
group_key = {
    "nwb_file_name": nwb_file_name,
    "sorted_spikes_group_name": "demo_group",
}
SortedSpikesGroup & group_key
from spyglass.spikesorting.analysis.v1.group import SortedSpikesGroup

# create a new sorted spikes group
unit_filter_params_name = "default_exclusion"
SortedSpikesGroup().create_group(
    group_name="demo_group",
    nwb_file_name=nwb_file_name,
    keys=[
        {"spikesorting_merge_id": merge_id}
        for merge_id in spikesorting_merge_ids
    ],
    unit_filter_params_name=unit_filter_params_name,
)
# check the new group
group_key = {
    "nwb_file_name": nwb_file_name,
    "sorted_spikes_group_name": "demo_group",
}
SortedSpikesGroup & group_key

Out[9]:

nwb_file_name name of the NWB file	unit_filter_params_name	sorted_spikes_group_name
mediumnwb20230802_.nwb	default_exclusion	demo_group

Total: 1

In [10]:

Copied!

SortedSpikesGroup.Units & group_key
SortedSpikesGroup.Units & group_key

Out[10]:

nwb_file_name name of the NWB file	unit_filter_params_name	sorted_spikes_group_name	spikesorting_merge_id
mediumnwb20230802_.nwb	default_exclusion	demo_group	143dff79-3779-c0d2-46fe-7c5040404219
mediumnwb20230802_.nwb	default_exclusion	demo_group	2249c566-cc17-bdda-4074-d772ee40b772
mediumnwb20230802_.nwb	default_exclusion	demo_group	4a191cc4-945b-3ad8-592a-a95e874b2507
mediumnwb20230802_.nwb	default_exclusion	demo_group	75286bf3-f876-4550-f235-321f2a7badef
mediumnwb20230802_.nwb	default_exclusion	demo_group	76ec4894-300d-4ed3-ce26-0327e7ed3345
mediumnwb20230802_.nwb	default_exclusion	demo_group	a900c1c8-909d-e583-c377-e98c4f0deebf

Total: 6

We can access the spikesorting results for this data using SortedSpikesGroup.fetch_spike_data()

In [11]:

Copied!





# get the complete key
group_key = (SortedSpikesGroup & group_key).fetch1("KEY")
# get the spike data, returns a list of unit spike times
SortedSpikesGroup().fetch_spike_data(group_key)
# get the complete key
group_key = (SortedSpikesGroup & group_key).fetch1("KEY")
# get the spike data, returns a list of unit spike times
SortedSpikesGroup().fetch_spike_data(group_key)

Out[11]:

[array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593717e+09, 1.62593717e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593572e+09, ...,
        1.62593717e+09, 1.62593717e+09, 1.62593717e+09]),
 array([1.62593571e+09, 1.62593571e+09, 1.62593571e+09, ...,
        1.62593717e+09, 1.62593717e+09, 1.62593717e+09]),
 array([1.62593571e+09, 1.62593572e+09, 1.62593574e+09, ...,
        1.62593714e+09, 1.62593714e+09, 1.62593715e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593577e+09, 1.62593577e+09, 1.62593577e+09, 1.62593577e+09,
        1.62593577e+09, 1.62593577e+09, 1.62593577e+09, 1.62593577e+09,
        1.62593577e+09, 1.62593577e+09, 1.62593579e+09, 1.62593579e+09,
        1.62593579e+09, 1.62593579e+09, 1.62593579e+09, 1.62593583e+09,
        1.62593583e+09, 1.62593583e+09, 1.62593583e+09, 1.62593583e+09,
        1.62593583e+09, 1.62593583e+09, 1.62593583e+09, 1.62593583e+09,
        1.62593583e+09, 1.62593583e+09, 1.62593583e+09, 1.62593594e+09,
        1.62593594e+09, 1.62593594e+09, 1.62593594e+09, 1.62593594e+09,
        1.62593594e+09, 1.62593594e+09, 1.62593594e+09, 1.62593594e+09,
        1.62593595e+09, 1.62593595e+09, 1.62593595e+09, 1.62593595e+09,
        1.62593595e+09, 1.62593595e+09, 1.62593595e+09, 1.62593595e+09,
        1.62593595e+09, 1.62593595e+09, 1.62593599e+09, 1.62593599e+09,
        1.62593599e+09, 1.62593599e+09, 1.62593599e+09, 1.62593599e+09,
        1.62593599e+09, 1.62593599e+09, 1.62593599e+09, 1.62593599e+09,
        1.62593625e+09, 1.62593625e+09, 1.62593625e+09, 1.62593626e+09,
        1.62593643e+09, 1.62593643e+09, 1.62593643e+09, 1.62593643e+09,
        1.62593643e+09, 1.62593643e+09, 1.62593643e+09, 1.62593643e+09,
        1.62593643e+09, 1.62593643e+09, 1.62593643e+09, 1.62593647e+09,
        1.62593647e+09, 1.62593651e+09, 1.62593651e+09, 1.62593651e+09,
        1.62593653e+09, 1.62593653e+09, 1.62593653e+09, 1.62593653e+09,
        1.62593653e+09, 1.62593654e+09, 1.62593654e+09, 1.62593654e+09,
        1.62593654e+09, 1.62593654e+09, 1.62593654e+09, 1.62593654e+09,
        1.62593654e+09, 1.62593654e+09, 1.62593654e+09, 1.62593654e+09,
        1.62593654e+09, 1.62593654e+09, 1.62593654e+09, 1.62593657e+09,
        1.62593657e+09, 1.62593657e+09, 1.62593657e+09, 1.62593657e+09,
        1.62593657e+09, 1.62593657e+09, 1.62593657e+09, 1.62593657e+09,
        1.62593657e+09, 1.62593657e+09, 1.62593657e+09, 1.62593657e+09,
        1.62593658e+09, 1.62593658e+09, 1.62593658e+09, 1.62593658e+09,
        1.62593658e+09, 1.62593658e+09, 1.62593658e+09, 1.62593658e+09,
        1.62593658e+09, 1.62593658e+09, 1.62593658e+09, 1.62593658e+09,
        1.62593659e+09, 1.62593659e+09, 1.62593659e+09, 1.62593659e+09,
        1.62593659e+09, 1.62593659e+09, 1.62593659e+09, 1.62593659e+09,
        1.62593659e+09, 1.62593659e+09, 1.62593660e+09, 1.62593660e+09,
        1.62593661e+09, 1.62593661e+09, 1.62593661e+09, 1.62593661e+09,
        1.62593661e+09, 1.62593661e+09, 1.62593661e+09, 1.62593661e+09,
        1.62593671e+09, 1.62593674e+09, 1.62593678e+09, 1.62593695e+09,
        1.62593712e+09, 1.62593712e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593715e+09, 1.62593715e+09, 1.62593715e+09]),
 array([1.62593570e+09, 1.62593572e+09, 1.62593572e+09, ...,
        1.62593717e+09, 1.62593717e+09, 1.62593717e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593715e+09, 1.62593715e+09, 1.62593716e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593715e+09, 1.62593715e+09, 1.62593715e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09]),
 array([1.62593570e+09, 1.62593570e+09, 1.62593570e+09, ...,
        1.62593718e+09, 1.62593718e+09, 1.62593718e+09])]

Unit Annotation¶

Many neuroscience applications are interested in the properties of individual neurons or units. For example, one set of custom analysis may classify each unit as a cell type based on firing properties, and a second analysis step may want to compare additional features based on this classification.

Doing so requires a consistent manner of identifying a unit, and a location to track annotations

Spyglass uses the unit identification system: {"spikesorting_merge_id" : merge_id, "unit_id" : unit_id}", where unit_id is the index of a units in the saved nwb file. fetch_spike_data can return these identifications by setting return_unit_ids = True

In [12]:

Copied!





spike_times, unit_ids = SortedSpikesGroup().fetch_spike_data(
    group_key, return_unit_ids=True
)
print(unit_ids[0])
print(spike_times[0])
spike_times, unit_ids = SortedSpikesGroup().fetch_spike_data(
    group_key, return_unit_ids=True
)
print(unit_ids[0])
print(spike_times[0])

{'spikesorting_merge_id': UUID('143dff79-3779-c0d2-46fe-7c5040404219'), 'unit_id': 0}
[1.62593570e+09 1.62593570e+09 1.62593570e+09 ... 1.62593718e+09
 1.62593718e+09 1.62593718e+09]

Further analysis may assign annotations to individual units. These can either be a string label (e.g. "pyridimal_cell", "thirst_sensitive"), or a float quantification (e.g. firing_rate, signal_correlation).

The UnitAnnotation table can be used to track and cross reference these annotations between analysis pipelines. Each unit has a single entry in UnitAnnotation, which can be connected to multiple entries in the UnitAnnotation.Annotation part table.

An Annotation entry should include an annotation describing the originating analysis, along with a label and/or quantification with the analysis result.

Here, we demonstrate adding quantification and label annotations to the units in the spike group we created using the add_annotation function.

In [18]:

Copied!





from spyglass.spikesorting.analysis.v1.unit_annotation import UnitAnnotation

for spikes, unit_key in zip(spike_times, unit_ids):
    # add a quantification annotation for the number of spikes
    annotation_key = {
        **unit_key,
        "annotation": "spike_count",
        "quantification": len(spikes),
    }
    UnitAnnotation().add_annotation(annotation_key, skip_duplicates=True)
    # add a label annotation for the unit id
    annotation_key = {
        **unit_key,
        "annotation": "cell_type",
        "label": "pyridimal" if len(spikes) < 1000 else "interneuron",
    }
    UnitAnnotation().add_annotation(annotation_key, skip_duplicates=True)

annotations = UnitAnnotation().Annotation() & unit_ids
annotations
from spyglass.spikesorting.analysis.v1.unit_annotation import UnitAnnotation

for spikes, unit_key in zip(spike_times, unit_ids):
    # add a quantification annotation for the number of spikes
    annotation_key = {
        **unit_key,
        "annotation": "spike_count",
        "quantification": len(spikes),
    }
    UnitAnnotation().add_annotation(annotation_key, skip_duplicates=True)
    # add a label annotation for the unit id
    annotation_key = {
        **unit_key,
        "annotation": "cell_type",
        "label": "pyridimal" if len(spikes) < 1000 else "interneuron",
    }
    UnitAnnotation().add_annotation(annotation_key, skip_duplicates=True)

annotations = UnitAnnotation().Annotation() & unit_ids
annotations

Out[18]:

spikesorting_merge_id	unit_id	annotation the kind of annotation (e.g. a table name, "cell_type", "firing_rate", etc.)	label text labels from analysis	quantification quantification label from analysis
143dff79-3779-c0d2-46fe-7c5040404219	0	cell_type	interneuron	nan
143dff79-3779-c0d2-46fe-7c5040404219	0	spike_count	None	40509.0
143dff79-3779-c0d2-46fe-7c5040404219	1	cell_type	interneuron	nan
143dff79-3779-c0d2-46fe-7c5040404219	1	spike_count	None	40181.0
143dff79-3779-c0d2-46fe-7c5040404219	2	cell_type	interneuron	nan
143dff79-3779-c0d2-46fe-7c5040404219	2	spike_count	None	18233.0
143dff79-3779-c0d2-46fe-7c5040404219	3	cell_type	interneuron	nan
143dff79-3779-c0d2-46fe-7c5040404219	3	spike_count	None	36711.0
2249c566-cc17-bdda-4074-d772ee40b772	0	cell_type	interneuron	nan
2249c566-cc17-bdda-4074-d772ee40b772	0	spike_count	None	48076.0
2249c566-cc17-bdda-4074-d772ee40b772	1	cell_type	interneuron	nan
2249c566-cc17-bdda-4074-d772ee40b772	1	spike_count	None	97667.0

...

Total: 54

Subsets of the the spikesorting data can then be accessed by calling fetch_unit_spikes on a restricted instance of the table. This allows the user to perform further analysis based on these labels.

Note: This function will return the spike times for all units in the restricted table

In [19]:

Copied!





# restrict to units from our sorted spikes group
annotations = UnitAnnotation.Annotation & (SortedSpikesGroup.Units & group_key)
# restrict to units with more than 3000 spikes
annotations = annotations & {"annotation": "spike_count"}
annotations = annotations & "quantification > 3000"

selected_spike_times, selected_unit_ids = annotations.fetch_unit_spikes(
    return_unit_ids=True
)
print(selected_unit_ids[0])
print(selected_spike_times[0])
# restrict to units from our sorted spikes group
annotations = UnitAnnotation.Annotation & (SortedSpikesGroup.Units & group_key)
# restrict to units with more than 3000 spikes
annotations = annotations & {"annotation": "spike_count"}
annotations = annotations & "quantification > 3000"

selected_spike_times, selected_unit_ids = annotations.fetch_unit_spikes(
    return_unit_ids=True
)
print(selected_unit_ids[0])
print(selected_spike_times[0])

{'spikesorting_merge_id': UUID('143dff79-3779-c0d2-46fe-7c5040404219'), 'unit_id': 0}
[1.62593570e+09 1.62593570e+09 1.62593570e+09 ... 1.62593718e+09
 1.62593718e+09 1.62593718e+09]