SpikeInterface, a unified framework for spike sorting (DANDI:000034)

Claims

From the paper SpikeInterface, a unified framework for spike sorting

Reproducing Buccino et al. (2020) from dandiset 000034 2

Notebook reproduce_buccino_2020.ipynb

Partially reproducedResults, Consensus units highly overlap with manually curated ones, paragraph 1; Figure 4Averifier disagreed at first
The two experts' manual curations of the Kilosort2 output agree on most curated units (226 of 351, 64.2%), and 174 Kilosort2 units are discarded by both.
most of the curated units (226 out of 351–64.2%) are agreed upon by both experts. Notably, 174 units found by Kilosort2 are discarded by both experts
Results, Consensus units highly overlap with manually curated ones, paragraph 1; Figure 4A
Computed from the two curated files under the criterion's second reading (compare_two_sorters of spikecomparison 0.2.6, defaults). Curated 1 has 258 units and curated 2 has 319; 226 are matched and 351 belong to either, 64.39%, so 226 and 351 equal the paper's and 64.39% is within 6.42 of its 64.2%. With a minimum score from 0.3 to 0.8 the matched count goes from 231 to 209, inside its tolerance; the ratio stays inside its tolerance up to 0.7 (59.83%) and falls below it at 0.8 (56.79%, lower bound 57.78%). The criterion fixes the default score of 0.5, so this does not decide the outcome. The count of 174 Kilosort2 units discarded by both curators needs the paper's Kilosort2 run (fig1b-unit-counts, blocked); the curated files hold unit ids and spike times only, with no link to the Kilosort2 clusters (HDF5 tree printed), so it is untested. The criterion gives reproduced when all four values hold under one reading and partially reproduced when 226, 351 and 64.2% hold and 174 does not; with 174 untested the claim is partially reproduced at least, and reproduced only if a Kilosort2 run confirmed the 174. The first reading (compare_multiple_sorters with the Kilosort2 run) could not be run. measured is not given, as the fourth expected value has no measurement.
How the test differed from the paper’s method
- spikecomparison 0.2.6 and spikeextractors 0.8.4 were run on NumPy 2 with the removed aliases np.float, np.int, np.Inf, np.product and np.in1d restored as the builtins and functions they stood for.
- The curated files carry no sampling rate; their spike times were turned into frames at the mouse recording's 30 kHz, as the paper's loader (NwbSortingExtractor) does with the rate it is given.
ReproducedResults, opening paragraph; Data availability; the four notebooks at ee08fb6checks the link to the paper’s data
The dandiset holds the recordings the paper analyses and the files its notebooks name, with the properties the paper and the notebooks give.
The datasets are publicly available in NWB format on the DANDI archive
Results, opening paragraph; Data availability; the four notebooks at ee08fb6
All 15 checks hold. The five files the claims file names are present. The mouse 412804 recording is sub-mouse412804_ecephys.nwb, 27,000,000 frames at 30 kHz (900 s), 248 channels (the notebook's count; the paper says 246 active channels). The simulation lasts 600.00 s at 32 kHz and holds 250 ground-truth units. The rat recording has 384 channels at 30 kHz (270.01 s) and the retina recording 1024 channels at 23199.09 Hz (135.73 s). The curated and ground-truth spike times share the clocks of their recordings on the first seconds read (deepest mean trough at lag 0 in median; within 3 frames for 0.88 and 0.86 of the curated units and 0.63 of the ground-truth units).
How the test differed from the paper’s method
- The paper's notebook loads a local file, mouse412804_probeC_15min.nwb; the dandiset's mouse recording is matched to it by subject and properties, not by name.

Not tested in a notebook 22

22 method cannot be run here

BlockedResults, Spike sorters show low agreement for the same high-density dataset, paragraph 3; Figure 1B· Method cannot be run here
On the real Neuropixels recording the six sorters return very different numbers of units: Tridesclous the fewest (187) and SpyKING Circus the most (628); HerdingSpikes2 210, Kilosort2 446, IronClust 233 and HDSort 317.
with Tridesclous (TDC) finding the least units (187) and SpyKING Circus (SC) finding the most units (628). HerdingSpikes2 finds 210 units; Kilosort2 finds 446 units; IronClust finds 233 units; and HDSort finds 317 units
Results, Spike sorters show low agreement for the same high-density dataset, paragraph 3; Figure 1B
Needs the six sorters' output on the mouse Neuropixels recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Spike sorters show low agreement for the same high-density dataset, paragraph 4; Figure 1C· Method cannot be run here
Of the 2031 units detected on the real recording, all six sorters agree on only 33, and two or more sorters agree on only 263.
Of the 2031 total detected units, all six sorters agree on just 33 of the units ... We also find that two or more sorters agree on just 263 of the total units
Results, Spike sorters show low agreement for the same high-density dataset, paragraph 4; Figure 1C
Needs the six sorters' output on the mouse Neuropixels recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Spike sorters show low agreement for the same high-density dataset, paragraph 4; Figure 1D· Method cannot be run here
For most sorters more than half of their units match no other sorter; IronClust and Tridesclous are the exceptions.
For most sorters, over 50% of the units that they find do not match with any other sorter (with the exceptions of Ironclust and Tridesclous)
Results, Spike sorters show low agreement for the same high-density dataset, paragraph 4; Figure 1D
Needs the six sorters' output on the mouse Neuropixels recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Spike sorters show low agreement for the same high-density dataset, paragraph 4; Figure 1, figure supplement 2· Method cannot be run here
Matched units mostly agree closely: around 80% of the agreement scores of agreed-upon units are 0.8 or higher.
For agreed-upon units, around 80% of the agreement scores are 0.8 or higher, indicating that matched units typically have high spike train agreement
Results, Spike sorters show low agreement for the same high-density dataset, paragraph 4; Figure 1, figure supplement 2
Needs the six sorters' output on the mouse recording or the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Spike sorters show low agreement for the same high-density dataset, paragraph 5; Figure 1, figure supplements 3 and 4· Method cannot be run here
On a rat cortex Neuropixels recording (six sorters) and a mouse retina Biocam recording (the four sorters able to process it, HerdingSpikes2, Kilosort2, IronClust and HDSort) the sorters disagree as much as on the first recording.
In both cases, we find similar disagreement among the sorters
Results, Spike sorters show low agreement for the same high-density dataset, paragraph 5; Figure 1, figure supplements 3 and 4
Needs the sorters' output on the rat and retina recordings. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Evaluating spike sorters on a simulated dataset, paragraph 2; Figure 2C· Method cannot be run here
On the simulated recording with 250 ground-truth units the sorters again return very different numbers of units: Tridesclous 189, HDSort 458, HerdingSpikes2 233, Kilosort2 415, IronClust 283 and SpyKING Circus 343.
The number of detected units range from the 189 units found by Tridesclous to the 458 units found by HDSort. HerdingSpikes2 finds 233 units; Kilosort2 finds 415 units; IronClust finds 283 units; and SpyKING Circus finds 343 units
Results, Evaluating spike sorters on a simulated dataset, paragraph 2; Figure 2C
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedFigure 2, figure supplement 1A legend; Results, Evaluating spike sorters on a simulated dataset, paragraph 2· Method cannot be run here
Sorters trade precision against recall differently: HerdingSpikes2, SpyKING Circus and HDSort favour precision, IronClust recall, Kilosort2 scores well on both, and Tridesclous shows no bias.
Some sorters seem to favor precision (HerdingSpikes, SpyKING Circus, HDSort), others instead have higher recall (Ironclust) or score well on both measures (Kilosort2). Tridesclous does not show a bias towards precision or recall
Figure 2, figure supplement 1A legend; Results, Evaluating spike sorters on a simulated dataset, paragraph 2
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Evaluating spike sorters on a simulated dataset, paragraph 2; Figure 2, figure supplement 1B· Method cannot be run here
Accuracy depends on the SNR of the ground-truth units for every sorter except Kilosort2, which reaches almost perfect accuracy on low-SNR units.
Moreover, the accuracy is modulated by the SNR of the ground-truth units for all spike sorters except Kilosort2 which achieves an almost perfect performance on the low-SNR units
Results, Evaluating spike sorters on a simulated dataset, paragraph 2; Figure 2, figure supplement 1B
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook). The ground-truth SNR alone could be computed from the simulated traces, but the claim relates the sorters' accuracy to it.
BlockedResults, Evaluating spike sorters on a simulated dataset, paragraph 2; Figure 2D· Method cannot be run here
Kilosort2 scores higher than the other sorters for most ground-truth units.
Kilosort2 attains significantly higher scores than the rest of the spike sorters for most ground-truth units
Results, Evaluating spike sorters on a simulated dataset, paragraph 2; Figure 2D
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Evaluating spike sorters on a simulated dataset, paragraph 3; Figure 2E· Method cannot be run here
Kilosort2 has the most well-detected units (245) but also many false positive (147) and redundant (21) units, and two overmerged units.
While Kilosort2 has the most well-detected units (245), this comes at the cost of a high percentage of false positive (147) and redundant (21) units ... In both cases two overmerged units are found
Results, Evaluating spike sorters on a simulated dataset, paragraph 3; Figure 2E
Needs Kilosort2's output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Evaluating spike sorters on a simulated dataset, paragraph 3 (parenthesis)· Method cannot be run here
With Kilosort2's own curation, which removes units with an estimated contamination above 20%, false positive and redundant units are fewer but persist: 241 well detected, 93 false positive, 18 redundant and 2 overmerged.
In that case the number of well-detected units is 241, false positives are 93, and redundant units are 18. In both cases two overmerged units are found
Results, Evaluating spike sorters on a simulated dataset, paragraph 3 (parenthesis)
Needs Kilosort2's output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Evaluating spike sorters on a simulated dataset, paragraph 3; Figure 2E· Method cannot be run here
Tridesclous detects very few false positive or redundant units while finding many well-detected units, and HDSort finds many more false positive units than any other sorter.
Notably, Tridesclous detects very few false positive/redundant units while still finding many well-detected units. HDSort, on the flip side, finds many more false positive units than any other spike sorter
Results, Evaluating spike sorters on a simulated dataset, paragraph 3; Figure 2E
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Low-agreement units are mainly false positives, paragraph 1; Figure 3A· Method cannot be run here
On the simulated recording only 139 of the 1921 detected units (7.24%) are agreed upon by all six sorters.
Again, we observe a large disagreement among the spike sorting outputs with only 139 units of the 1921 total units (7.24%) being in agreement among all sorters
Results, Low-agreement units are mainly false positives, paragraph 1; Figure 3A
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Low-agreement units are mainly false positives, paragraph 1; Figure 3B· Method cannot be run here
HDSort, SpyKING Circus and Kilosort2 are more prone to finding low-agreement units than HerdingSpikes2, IronClust and Tridesclous.
highlighting that some sorters are more prone to finding low agreement units (HDSort, SpyKING Circus, Kilosort2) than other sorters (HerdingSpikes2, Ironclust, Tridesclous)
Results, Low-agreement units are mainly false positives, paragraph 1; Figure 3B
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Low-agreement units are mainly false positives, paragraph 2; Figure 3C legend· Method cannot be run here
Almost all false positive units are found by a single sorter, while most real units are matched by more than one sorter; false positives found by two or more sorters number 4 for HerdingSpikes2, Kilosort2 and IronClust, 2 for SpyKING Circus and HDSort and 1 for Tridesclous.
The plots show that (almost) all false positive units are ones that are found by only a single sorter (not matched with any other sorters), while most real units are matched by more than one sorter
Results, Low-agreement units are mainly false positives, paragraph 2; Figure 3C legend
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Low-agreement units are mainly false positives, paragraph 2; Figure 3, figure supplement 1 legend· Method cannot be run here
With two sorters, combining Kilosort2 and IronClust identified true positives most reliably, with false positive and true positive detection rates of 96% and 95%.
For two sorters, the most reliable identification of true positives for this dataset was achieved by combining Kilosort2 and Ironclust (96% and 95% false positive and true positive detection rate, respectively)
Results, Low-agreement units are mainly false positives, paragraph 2; Figure 3, figure supplement 1 legend
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Low-agreement units are mainly false positives, paragraph 2; Figure 3, figure supplement 1 legend· Method cannot be run here
A pair of sorters suffices to isolate almost all false positive units, but with fewer than four sorters a significant fraction of true positive units, found by only one sorter, are wrongly classed as false positives; four or more sorters reliably identify most true positives.
This analysis showed that using a pair of sorters is sufficient to isolate almost all false positive units in each sorter, yet when fewer than four sorter outputs are compared, a significant fraction of true positive units found by only one sorter can be wrongly classified as false positives with this approach
Results, Low-agreement units are mainly false positives, paragraph 2; Figure 3, figure supplement 1 legend
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Low-agreement units are mainly false positives, paragraph 3; Figure 3D· Method cannot be run here
Ground-truth units with higher SNR are found by more sorters.
This shows, as expected, that higher SNR units have higher agreement among sorters
Results, Low-agreement units are mainly false positives, paragraph 3; Figure 3D
Needs the six sorters' output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Low-agreement units are mainly false positives, paragraph 3; Figure 3, figure supplement 2 legend· Method cannot be run here
For Kilosort2, SNR is not a good predictor of false positives: many of its false positive units have an SNR above the mode of the ground-truth SNR.
Many detected false positive units have an SNR above the mode of the ground-truth SNR, indicating that SNR is not a good measure to separate false and true positives in this case
Results, Low-agreement units are mainly false positives, paragraph 3; Figure 3, figure supplement 2 legend
Needs Kilosort2's output on the simulated recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook).
BlockedResults, Consensus units highly overlap with manually curated ones, paragraph 2; Figure 4B· Method cannot be run here
Only a small percentage of each sorter's units match the curated units; IronClust matches best (59.83% and 61.1%), above Kilosort2 (50.67% and 56.25%), from whose output the curations were made.
The highest percentage match is actually IronClust which is surprising given that the initial sorting output was curated from Kilosort2’s output (IC ∩ C1 = 59.83%, IC ∩ C2 = 61.1%, KS ∩ C1 = 50.67%, KS ∩ C2 = 56.25%)
Results, Consensus units highly overlap with manually curated ones, paragraph 2; Figure 4B
Needs the six sorters' output on the mouse Neuropixels recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook). The curated sortings are in the dandiset (see fig4a-curators-agree).
BlockedResults, Consensus units highly overlap with manually curated ones, paragraph 2; Figure 4C· Method cannot be run here
Consensus units (found by at least two sorters) match the curated units at above 70% for every sorter; Kilosort2 is highest (84.55% and 89.55%), slightly above IronClust (82.63% and 83.83%).
The average match percentage is above 70% for all sorters showing that there is a large agreement between the manually curated outputs and the consensus-based output. Kilosort2 has the highest match (KS c ∩ C1 = 84.55%, KS c ∩ C2 = 89.55%), slightly higher than Ironclust (IC c ∩ C1 = 82.63%, IC c ∩ C2 = 83.83%)
Results, Consensus units highly overlap with manually curated ones, paragraph 2; Figure 4C
Needs the six sorters' output on the mouse Neuropixels recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook). The curated sortings are in the dandiset.
BlockedResults, Consensus units highly overlap with manually curated ones, paragraph 2; Figure 4D legend· Method cannot be run here
Units found by only one sorter rarely match the curated units; the only notable overlap is Kilosort2's (18.58% with C1 and 24.34% with C2).
The only significant overlap is between Curator one and Kilosort2, with a percent around 18% (KS nc ∩ C1 = 18.58%, KS nc ∩ C2 = 24.34%)
Results, Consensus units highly overlap with manually curated ones, paragraph 2; Figure 4D legend
Needs the six sorters' output on the mouse Neuropixels recording. The dandiset holds no sorter output. Kilosort2, IronClust and HDSort are MATLAB programs (Kilosort2 also needs a CUDA GPU); the runner has no MATLAB, GPU or MPI (printed in the notebook) and they cannot be installed from a lock file. Reading time does not add to the block: read chunk by chunk through h5py a whole recording would take hours (notebook table), but one bulk ranged read of the mouse file ran at 72 MB/s in this run, so each recording could be read whole in 1.4 to 6.5 min (notebook). The curated sortings are in the dandiset.

Findings about the data

What attempts found about the dataset itself, for its owner and the archive.

Data missingThe dandiset holds no output of the six spike sorters the paper compares, including the Kilosort2 run that the two experts curated. Every comparison of sorters in the paper, and the comparison of the curations with Kilosort2, can be repeated only by running the sorters again, three of them in MATLAB, and a new Kilosort2 run will not give the units the curators worked on.
Description missingThe two curated files hold a Units table of ids and spike times only (description 'Autogenerated by NWBFile'), with no sampling rate, no unit labels, no curator name and no link to the Kilosort2 clusters they were curated from; which file is curator 1 and which curator 2 is not stated (the files differ only by session time, 15:55:42 and 15:55:43). Their spike times fall on whole frames at 30 kHz and match the mouse recording's clock.
Metadata errorThe retina recording is stored as unsigned 16-bit values centred near 2048 and reaching 4096, consistent with raw counts of a 12-bit converter, while the file states a conversion of 1e-6 V per unit and no offset, which would put every channel near +2 mV. The converter's gain and offset are not in the file.
Metadata errorThe mouse 412804 recording stores int16 values with a conversion of 1e-6 V and no gain or channel_conversion per channel. Read that way its band-passed noise SD is about 50 uV and the median curated trough about 455 uV, several times usual Neuropixels levels; at the Neuropixels 1.0 gain of about 0.195 uV per bit the noise would be about 10 uV. The values are consistent with raw counts given a conversion that treats them as microvolts; the file holds nothing to show which.
Description missingThe electrode tables give location 'unknown' and filtering 'none' for every channel and placeholder group descriptions; the mouse recording has 248 channels where the paper says 246 are active, without marking which; the simulated file does not say which ground-truth cells are excitatory or inhibitory or what noise was added; the archive's species summary lists only house mouse though one recording is from a rat (its file gives 'rattus rattus').
OtherEvery raw series is stored in chunks of one channel by 7911 to 26368 frames, so reading a stretch of all channels through h5py fetches one chunk per channel. Streamed from the archive at 0.087 to 0.155 s per chunk in this run, reading a whole recording that way would take 6.2 to 16.9 h; one bulk ranged read of a file ran at 72 MB/s, 1.4 to 6.5 min per recording. These times vary between runs.
Paper inconsistentThe paper gives 226 of 351 curated units as 64.2%; 226/351 is 64.39%.

Notebooks

Every registered entry that uses this dataset. An entry can hold several notebooks, and can use more than one dataset.

Reproduction SpikeInterface spike sorter comparison (Buccino et al. 2020)

Tests the findings of Buccino, Hurwitz, Garcia, Magland, Siegle, Hurwitz and Hennig (2020) on dandiset 000034, which holds the mouse, simulated, rat and retina recordings the paper sorts and two manual curations of its Kilosort2 output. It checks that the recordings are as described, that the curated and ground-truth spike times share the recordings' clocks, and compares the two curations. The findings about sorter outputs are blocked, as three of the six sorters need MATLAB and a GPU, which the runner lacks. All files

Reproducing Buccino et al. (2020) from dandiset 000034
reproduce_buccino_2020.ipynbExecuted, with outputs · Source

Verification

Round 1
2026-10-22
Agreed with 23 of 24 outcomes. Independent agent run by CatalystNeuro (same harness, separate context, did not write the notebook). Reviewed: The executed notebook of the first run with the conclusion cells and the summary removed, the helper module and the claims file, with the paper's full text; not the attempt record. The verifier checked the asset list and metadata with the archive API and tested to_frames and lag_scan on synthetic input. Report in transcripts/000034/verifier-1-report.md. The cell numbers in the verifier's round-1 reasons refer to that notebook; in the revised notebook the link check is cell 18, the software check cell 21 and fig4a cell 24.
- fig4a-curators-agree: the verifier’s outcome was “Partially reproduced”. Cell 21: pairwise compare_two_sorters(C1, C2) with defaults gives 226 (expected 226 ± 22.6), 351 (351 ± 35.1) and 64.39% (64.2 ± 6.42). The 174 needs the Kilosort2 run and is untested. The criterion gives reproduced if 174 holds and partially reproduced if not, so the claim is at least partially reproduced either way. The notebook calls the whole claim blocked, as it depends on fig1b, yet the dependency only touches the 174 (point B1). The verifier could not confirm that the curated Units tables carry no Kilosort2 labels or links, and asked for the HDF5 tree of both curated files and of the mouse recording.
Round 2
2026-10-22Agreed with 24 of 24 outcomes. Independent agent run by CatalystNeuro (same harness, separate context, did not write the notebook). Reviewed: The full executed notebook of the revised run, with the conclusions and the summary, the helper module, the claims file, the README and this record, with the paper's full text. Report in transcripts/000034/verifier-2-report.md. Points not tied to one claim, and what was done: cell 0 said every finding but the recordings' presence rests on running six sorters, against its own table, and now names the fig4a curator counts and the three Kilosort2-only claims; cell 13's label asserted microvolts and now says the scale is not checked; the Summary's 'not with microvolts' was softened; the species summary finding is now backed by a print in cell 3; the streaming finding now gives this run's numbers (0.087 to 0.155 s per chunk, 6.2 to 16.9 h, against the verifier's 0.080 to 0.142 s and 5.6 to 15.5 h); and the round-1 record now says its cell numbers refer to the round-1 notebook.

Method differences

How the tests differed from the paper’s method or data, and the claims each applies to.

The paper's notebook loads a local file, mouse412804_probeC_15min.nwb; the dandiset's mouse recording is matched to it by subject and properties, not by name.recordings-as-described
spikecomparison 0.2.6 and spikeextractors 0.8.4 were run on NumPy 2 with the removed aliases np.float, np.int, np.Inf, np.product and np.in1d restored as the builtins and functions they stood for.fig4a-curators-agree
The curated files carry no sampling rate; their spike times were turned into frames at the mouse recording's 30 kHz, as the paper's loader (NwbSortingExtractor) does with the rate it is given.fig4a-curators-agree

Provenance

Attempt record: 000034-buccino-2020-01
Paper: SpikeInterface, a unified framework for spike sorting
Date: 2026-10-22
Made by: An agent, run by CatalystNeuro
Dataset: DANDI:000034
Version: 0.211030.0713 (doi:10.48324/dandi.000034/0.211030.0713)
Run on: GitHub Actions
Harness: Claude Code (GitHub Actions)
Model: claude-opus-5-5
Protocol: pilot-10

Transcripts of the agent runs

Each run’s prompt, messages and tool calls, with long output cut and the text of the paper left out, and its final report.

← DANDI:000019 DANDI:000035 →