Spyglass Mixin¶

The Spyglass Mixin provides a way to centralize all Spyglass-specific functionalities that have been added to DataJoint tables. This includes...

Fetching NWB files
Long-distance restrictions.
Permission checks on delete
Export logging. See export doc for more information.
Miscellaneous helper functions

To add this functionality to your own tables, simply inherit from the mixin:

import datajoint as dj

from spyglass.utils import SpyglassMixin

schema = dj.schema("my_schema")


@schema
class MyOldTable(dj.Manual):
    pass


@schema
class MyNewTable(SpyglassMixin, dj.Manual):
    pass

NOTE: The mixin must be the first class inherited from in order to override default DataJoint functions.

Fetching NWB Files¶

Many tables in Spyglass inheris from central tables that store records of NWB files. Rather than adding a helper function to each table, the mixin provides a single function to access these files from any table.

from spyglass.example import AnyTable

(AnyTable & my_restriction).fetch_nwb()

This function will look at the table definition to determine if the raw file should be fetched from Nwbfile or an analysis file should be fetched from AnalysisNwbfile. If neither is foreign-key-referenced, the function will refer to a _nwb_table attribute.

Custom Analysis File Tables: Spyglass supports individualized AnalysisNwbfile tables to address transaction lock contention in multi-team environments. The fetch_nwb() method automatically detects whether your table references the common AnalysisNwbfile table or a custom team-specific table and fetches from the appropriate location. See Custom Analysis Tables for details on using custom analysis file tables.

Migration from standalone `fetch_nwb` / `get_nwb_table` helpers¶

Older Spyglass code used two standalone functions from spyglass.utils.dj_helper_fn that required passing the NWB table and attribute explicitly. Both were removed in 0.6.0. get_nwb_table is gone entirely — importing it raises an ImportError. fetch_nwb is kept as a stub that raises NotImplementedError when called, since it is a commonly imported name and a runtime message is clearer than an import failure. Use the fetch_nwb() method below instead.

`fetch_nwb` (helper function)¶

# Before
from spyglass.utils.dj_helper_fn import fetch_nwb
from spyglass.common import Nwbfile

results = fetch_nwb(
    MyTable & my_key,
    (Nwbfile, "nwb_file_abs_path"),
)

# After
results = (MyTable & my_key).fetch_nwb()

`get_nwb_table`¶

# Before
from spyglass.utils.dj_helper_fn import get_nwb_table
from spyglass.common import Nwbfile

nwb_files, path_fn = get_nwb_table(
    MyTable & my_key,
    Nwbfile,
    "nwb_file_abs_path",
)
file_path = path_fn(nwb_files[0])

# After — fetch_nwb handles file resolution internally
results = (MyTable & my_key).fetch_nwb()

The method inspects the table's foreign-key references to determine whether to resolve paths from Nwbfile or AnalysisNwbfile automatically, so the nwb_master tuple is no longer needed. The return value is a list of dicts, each containing the fetched row fields plus the loaded NWB object.

NOTE: Both functions are always called together in practice — get_nwb_table provides the file list and path resolver that fetch_nwb then uses. The fetch_nwb() method replaces both in a single call.

Long-Distance Restrictions¶

In complicated pipelines like Spyglass, there are often tables that 'bury' their foreign keys as secondary keys. This is done to avoid having to pass a long list of foreign keys through the pipeline, potentially hitting SQL limits (see also Merge Tables). This burrying makes it difficult to restrict a given table by familiar attributes.

Spyglass provides a function, restrict_by, to handle this. The function takes your restriction and checks parents/children until the restriction can be applied. Spyglass introduces << as a shorthand for restrict_by an upstream key and >> as a shorthand for restrict_by a downstream key.

from spyglass.example import AnyTable

AnyTable() << 'upstream_attribute="value"'
AnyTable() >> 'downstream_attribute="value"'

# Equivalent to
AnyTable().restrict_by('downstream_attribute="value"', direction="down")
AnyTable().restrict_by('upstream_attribute="value"', direction="up")

Some caveats to this function:

'Peripheral' tables, like IntervalList and AnalysisNwbfile make it hard to determine the correct parent/child relationship and have been removed from this search by default.
This function will raise an error if it attempts to check a table that has not been imported into the current namespace. It is best used for exploring and debugging, not for production code.
It's hard to determine the attributes in a mixed dictionary/string restriction. If you are having trouble, try using a pure string restriction.
The most direct path to your restriction may not be the path your data took, especially when using Merge Tables. When the result is empty see the warning about the path used. Then, ban tables from the search to force a different path.

my_table = MyTable()  # must be instanced
my_table.ban_search_table(UnwantedTable1)
my_table.ban_search_table([UnwantedTable2, UnwantedTable3])
my_table.unban_search_table(UnwantedTable3)
my_table.see_banned_tables()

my_table << my_restriction
my_table << upstream_restriction >> downstream_restriction

When providing a restriction of the parent, use 'up' direction. When providing a restriction of the child, use 'down' direction.

For a more concrete example, imagine that we want to know which entries in the BurstPair table have a 1 in the session_id. The DataJoint-native way to approach this is to manually figure out the path from Session -> Raw ... -> BurstPair by looking at the diagram, or using various graph-structure methods: parents, children, ancestors, and descendants. We then use the * join operator for each link in the chain. Because there are some clashing secondary keys, we would need to drop them with proj().

from spyglass.common import Raw, Session
from spyglass.spikesorting.v1 import (
    BurstPair,
    BurstPairSelection,
    CurationV1,
    MetricCuration,
    MetricCurationSelection,
    SpikeSorting,
    SpikeSortingRecording,
    SpikeSortingRecordingSelection,
    SpikeSortingSelection,
)

join = (
    (
        (
            (Session * Raw).proj()
            * SpikeSortingRecordingSelection
            * SpikeSortingRecording
        ).proj()
        * SpikeSortingSelection
        * SpikeSorting
    ).proj()
    * CurationV1
    * MetricCurationSelection
).proj() * (
    MetricCuration * BurstPairSelection * BurstPair
) * Session & "session_id LIKE '%1%'"  # Last join adds back secondary keys

Pros:
- Reliable - will not break if new tables are added elsewhere
- Fast - deterministic computational process, no guess-and-check
- All primary keys present, so it's easier to fetch other attributes
Cons:
- Effortful - for long pipelines, takes time to find the path and then add relevant parentheses and proj
- Inconvenient for one-off queries

Instead, we can use the long-distance operator, or the equivalent restrict_by method to restrict the current table. By default, verbose and return_graph are false. When turned on, these display the search process and return the RestrGraph object, respectively. The graph object can be used to look at the path or check the same restriction at midpoints.

from spyglass.spikesorting.v1 import BurstPair

burst_tbl = BurstPair()
by_operator = burst_tbl << "session_id LIKE '%1%'"
restr_graph = burst_tbl.restrict_by(
    "session_id LIKE '%1%'", direction="up", verbose=True, return_graph=True
)
restr_graph.path  # see the list of tables in the path
restr_graph.all_ft  # see each table as restricted by session_restr

Pros:
- Easy to write
- Easier to read - the result is a restriction on the current table, not including any additional primary keys.
Cons:
- Slow - ~10x processing time to run restriction compatibility checks
- Unreliable - may not traverse multi-directional paths (i.e., parent -> child -> other-parent)
- False negatives - may show no results if the discovered path does not follow the data provenance

Delete Permission Checks¶

By default, DataJoint is unable to set delete permissions on a per-table basis. If a user is able to delete entries in a given table, she can delete entries in any table in the schema.

The mixin relies on the Session.Experimenter and LabTeams tables to ...

Check the session and experimenter associated with the attempted deletion.
Check the lab teams associated with the session experimenter and the user.

If the user shares a lab team with the session experimenter, the deletion is permitted.

This is not secure system and is not a replacement for database backups (see database management). A user could readily curcumvent the default permission checks by adding themselves to the relevant team or removing the mixin from the class declaration. However, it provides a reasonable level of security for the average user.

Because parts of this process rely on caching, this process will be faster if you assign the instanced table to a variable.

# Slower
YourTable().delete()
YourTable().delete()

# Faster
nwbfile = YourTable()
nwbfile.delete()
nwbfile.delete()

Deprecated delete feature

Previous versions of Spyglass also deleted masters of parts with foreign key references. This functionality has been migrated to DataJoint in version 0.14.2 via the `force_masters` delete argument. This argument is `True` by default in Spyglass tables.

Populate Calls¶

The mixin also overrides the default populate function to provide additional functionality for non-daemon process pools and disabling transaction protection.

Non-Daemon Process Pools¶

To allow the make function to spawn a new process pool, the mixin overrides the default populate function for tables with _parallel_make set to True. See issue #1000 and PR #1001 for more information.

Miscellaneous Helper functions¶

file_like allows you to restrict a table using a substring of a file name. This is equivalent to the following:

MyTable().file_like("eg")
MyTable() & ('nwb_file_name LIKE "%eg%" OR analysis_file_name LIKE "%eg%"')

find_insert_fail is a helper function to find the cause of an IntegrityError when inserting into a table. This checks parent tables for required keys.

my_key = {"key": "value"}
MyTable().insert1(my_key)  # Raises IntegrityError
MyTable().find_insert_fail(my_key)  # Shows the parent(s) missing the key

Populate Calls¶

The mixin also overrides the default populate function to provide additional functionality for non-daemon process pools and disabling transaction protection.

Non-Daemon Process Pools¶

To allow the make function to spawn a new process pool, the mixin overrides the default populate function for tables with _parallel_make set to True. See issue #1000 and PR #1001 for more information.

Disable Transaction Protection¶

By default, DataJoint wraps the populate function in a transaction to ensure data integrity (see Transactions).

This can cause issues when populating large tables if another user attempts to declare/modify a table while the transaction is open (see issue #1030 and DataJoint issue #1170).

NOTE: Setting _use_transaction = False on a table class is deprecated. The recommended replacement is the tri-part make pattern (make_fetch, make_compute, make_insert), which keeps the database insert inside a transaction while allowing long computations to run without holding a lock. See Populate and Long-Running Computations for a migration guide.

Spyglass Mixin¶

Fetching NWB Files¶

Migration from standalone fetch_nwb / get_nwb_table helpers¶

fetch_nwb (helper function)¶

get_nwb_table¶

Long-Distance Restrictions¶

Delete Permission Checks¶

Populate Calls¶

Non-Daemon Process Pools¶

Miscellaneous Helper functions¶

Populate Calls¶

Non-Daemon Process Pools¶

Disable Transaction Protection¶

Migration from standalone `fetch_nwb` / `get_nwb_table` helpers¶

`fetch_nwb` (helper function)¶

`get_nwb_table`¶