DANDI:001250 · Encore Science

Claims 39

From the paper Preprint · claims file · checked 2026-10-22: ready

ground-truth-neuron-counts
The simulations hold spikes from 1,388 model neurons within 50 um of the Neuropixels probe in the spontaneous regime, 1,836 in the 1-hour evoked regime at 20 kHz and 1,552 in the 10-minute evoked regime at 40 kHz, and every one of them fired at least once in the first 10 minutes.
We detected spikes from 1,388 model neurons during the spontaneous activity condition, 1,836 model neurons during the 1-hour evoked condition sampled at 20 kHz, and 1,552 neurons during the 10 minute evoked condition sampled at 40 kHz, within 50 µm of the Neuropixels probe ... All model neurons fired at least once in the first 10 minutes
Methods, Datasets and simulations, Biophysical model, paragraph 1
How it will be judged
Count the ground truth units of each simulation (the 50 um rule is in the opening comment). Reproduced when the counts are within tolerance and, for each simulation, every unit has at least one spike in the first 600 s; the number of silent units is reported. A simulation the dataset lacks is reported as not testable.
fitted-trace-maximal-amplitude
After gain and noise fitting, the maximal amplitude of the spontaneous Neuropixels simulation (361 uV) matches that of the Marques-Smith recording (362 uV), and the evoked simulation reaches 387 uV.
The spontaneous regime’s maximal amplitude (361 µV) closely matched Marques-Smith’s (362 µV), while the evoked regime produced a slightly higher value (387 µV)
Methods, Fitting amplitude and background noise, paragraph 1
How it will be judged
The paper does not say whether the maximum is taken before or after preprocessing, nor over which duration or sites. Readings, (a) the maximum absolute value of the fitted traces after preprocessing (300 Hz high-pass and common median reference) over cortical sites, as in the code (fig2_M_amplitude_npx, maximum of the absolute trace per site over the loaded trace); (b) the same on the raw fitted traces. Which evoked simulation is meant is not stated; both are shown. Reproduced when, under one reading, the three values are within tolerance.
fig2m-peak-amplitude-increases-with-depth
In both in vivo datasets the median peak amplitude of the traces increased with cortical depth, and the biophysical models reproduced this trend, although their medians differed from in vivo.
In both Marques-Smith and Horvath’s data, median peak amplitude increased with cortical depth ... While the models’ median peak amplitudes differed from in vivo (most p < 0.01, Mann-Whitney U test), all captured the in vivo trend
Results, Validation of recording traces, paragraph 2; Figure 2m, o
How it will be judged
Per site, the peak amplitude is the maximum absolute value of the preprocessed trace (code, fig2_M_amplitude_npx; the paper says "peak amplitudes of the electrode traces"), sites grouped by layer, cortical sites only; first 10 minutes (Methods, Amplitude-to-noise ratio, for all validation metrics; the code reads the whole trace, shown as a variant). For each of NS, E and DS, a Kruskal-Wallis test across layers. The paper does not say how "increased with depth" is judged beyond the test; the criterion takes the test at p < 0.001 and the layer medians rising from layer 1 to layer 6 with at most one inversion between neighbouring layers. Reproduced when this holds for every model; partially reproduced when it holds for some. The in vivo trend and the per-layer Mann-Whitney U tests against M (Neuropixels models) and H (dense model) are reported, with the share of layers at p < 0.01.
fig2np-noise-increases-with-depth
Background noise increased with cortical depth in vivo, and the biophysical models reproduced this trend.
In vivo background noise also increased with cortical depth ( p < 0.001, Kruskall Wallis, Fig. 2n, p ), a trend reproduced by the biophysical models
Results, Validation of recording traces, paragraph 2; Figure 2n, p
How it will be judged
Noise per site as the Methods give it for the fit, the mean absolute deviation of the preprocessed trace (Methods, Fitting amplitude and background noise); the code (src/nodes/validation/noise.py) takes the mean absolute deviation in 1-s segments and keeps the minimum over segments, which is followed. The median absolute deviation (Methods, Amplitude-to-noise ratio) is a variant. First 10 minutes, cortical sites, grouped by layer. Reproduced when, for each of NS, E and DS, a Kruskal-Wallis test across layers has p < 0.001 and the layer medians rise from layer 1 to 6 with at most one inversion between neighbours; partially reproduced when this holds for some models.
fig2q-anr-within-vivo-range
The amplitude-to-noise ratio distributions of the models span a wide dynamic range and lie within the in vivo range, with data between -10 and 10 times the noise level.
The distributions of ANRs observed in vivo and in the models showed a wide dynamic range, with alignment of data points between -10 and 10 times the noise level ( Fig. 2q ), particularly in layers 1, 5, and 6. Despite deviations from in vivo due to difference in peak amplitudes, simulated ANRs remained within the in vivo range
Results, Validation of recording traces, paragraph 2; Figure 2q
How it will be judged
ANR is the preprocessed trace divided by its median absolute deviation, per site, first 10 minutes, cortical sites; the range of all experiments split into 100 bins, the probability per bin per site, averaged over sites (Methods, Amplitude-to-noise ratio). "Within the in vivo range" is not defined; the criterion reads it as the minimum and maximum ANR of each model lying within those of its in vivo reference (M for NS and E, H for DS), widened by 10%. Reproduced when this holds for all three models, and every distribution has data from -10 to 10.
fig2r-power-law-spiking-band
In simulated and real traces, power decreased with frequency in the spiking band (0.3 to 6 kHz) and followed a power law in all layers.
In both the simulated and real traces, power decreased with increasing frequency in the frequency band of spiking activity (0.3 - 6 KHz) and followed a power law in all layers
Results, Validation of recording traces, paragraph 3; Figure 2r
How it will be judged
Welch PSD at 1 Hz resolution with a Hann window (Methods, Spectral composition), of the preprocessed traces, first 10 minutes (code, src/pipes/validation/psd/multiprocess.py), normalised by total power, median over the sites of each layer; a linear regression of log power on log frequency in 300 to 6000 Hz per site (Methods; code eval_freq_scaling). The paper gives no goodness-of-fit rule; the criterion reads "followed a power law" as a negative slope and R2 of at least 0.9 of the log-log fit for the median PSD of every layer of every experiment. Reproduced when this holds for all; the slopes and R2 are reported.
fig2s-v-ap-power-and-scaling-increase-with-depth
In the biophysical models the power in the spike frequency band and the power-law scaling slope in the spiking band increased with cortical depth, as in vivo; the dense probe model had weaker power and slope than in vivo but within the in vivo range.
The biophysical models’ median power in the frequency band of a spike waveform and the scaling slope in the spiking band also increased with cortical depth as in vivo (2s-v, all p < 0.001, Kruskall Wallis) ... Nonetheless the dense probe’s AP power and frequency scaling remained within in vivo range
Results, Validation of recording traces, paragraph 3; Figure 2s-v
How it will be judged
AP power per site is the PSD power in 0.3 to 2 kHz (Results, paragraph 3); the code's revised notebook uses the median PSD in 0.3 to 1 kHz (fig2_R_AP_power_npx_revised), shown as a variant. Both the median PSD value in the band and the summed power in the band are read, since the text says "median power" and "power". The slope alpha is from the 0.3 to 6 kHz fit of fig2r-power-law-spiking-band. For each model (NS, E, DS), a Kruskal-Wallis test across layers. Reproduced when both measures (AP power and alpha) have p < 0.001 in every model, with layer medians rising from layer 1 to 6 with at most one inversion between neighbours; for AP power in 0.3 to 2 kHz the median and summed readings are both run and the outcome is the one both give (methods_underspecified when they differ); the 0.3 to 1 kHz band is a variant, not used for the outcome. Partially reproduced when this holds for one measure or some models. Whether the DS values lie within the range of H's site values is reported.
supp2-no-slow-oscillations
The Horvath traces are dominated by slow (0.5 to 1.5 Hz) and delta (1 to 4 Hz) waves of slow-wave sleep, which the Marques-Smith traces and the biophysical models lack.
In Horvath’s data but not in Marques-Smith’s, the traces were dominated by high-amplitude slow (0.5 - 1.5Hz) and δ waves (1 - 4Hz) characteristic of slow-wave sleep ... The biophysical models did not replicate these oscillations
Results, Validation of recording traces, paragraph 4; Supplementary figure "Validation of raw simulated traces"
How it will be judged
Raw traces (mean subtracted), Welch PSD on the first 10 minutes divided by total power, averaged over sites (legend of the supplementary figure). The paper gives no threshold; the criterion reads "dominated" as the fraction of the 0.5 to 4 Hz power in the 0.5 to 300 Hz power being larger in H than in M and in every model (NS, E, DS) by a factor of 2 or more. Reproduced when this holds.
fig3e-amplitude-decay-with-distance
The peak-to-peak amplitude of a model cell's spikes drops by more than half within 75 um of the soma, and the trace is strongly distorted by noise beyond 200 um.
The peak-to-peak amplitude of the spikes rapidly decayed between nearby electrodes, dropping by more than half within 75 µm ( Fig. 3e ), therefore making the summed extracellular potentials difficult to isolate at 75 µm from the somata. The extracellular traces was strongly distorted by noise beyond 200 µm
Results, Validation of single unit spike waveforms, paragraph 1; Figure 3d, e
How it will be judged
The figure uses the most active layer 5 pyramidal cell, recorded isolated and in the full circuit at 128 electrode positions on a grid (Figure 3d, e legend); the code writes these traces to NWB as biophy_isolated_reyes and biophy_summed_reyes (cell 3754013, 40 s). Spike-triggered average on the isolated trace at each electrode, peak-to-peak amplitude against distance to the soma. Reproduced when the amplitude at every electrode 75 um or more from the soma is below half the amplitude at the nearest electrode. For the 200 um part the paper gives no measure; the spike-triggered average on the summed trace is reported against the trace's MAD at each distance, and that part holds when the peak-to-peak amplitude is below 5 MAD beyond 200 um (a choice made here, as the paper gives none; reported, not used for the outcome). Not testable when the dataset lacks these traces.
fig3c-five-temporal-profiles
Spikes sorted with Kilosort3 from the spontaneous Neuropixels simulation qualitatively match the five temporal profiles reported in humans and cats.
The simulated temporal waveforms qualitatively matched the five shapes reported in cats [ 46 ] and humans [ 9 ]
Results, Validation of single unit spike waveforms, paragraph 1; Figure 3a-c
How it will be judged
A qualitative comparison with panels copied from other papers, with no stated measure. The paper's own measure of profile diversity is the WaveMap clustering of fig3fg-wavemap-six-profiles, on which this claim depends; this claim is judged with that one and otherwise reported as not testable by a fixed rule.
fig3fg-wavemap-six-profiles
WaveMap clustering of multi-electrode spike waveforms identifies six distinct profiles in the simulation, as in the Marques-Smith data.
Consistent with the experimental data, the algorithm identified six distinct profiles from the simulations
Results, Validation of single unit spike waveforms, paragraph 2; Figure 3f, g
How it will be judged
Tolerance 0 because a cluster count is an integer and 10% of 6 is below one. Kilosort3 single units of NS and M (Results, paragraph 1 of the section). Spike-triggered averages; the six nearest sites with the largest average amplitude, ordered by the largest of first peak, trough or second peak; aligned at the trough over 12 ms, reduced to 6 ms; amplitudes scaled to 0 to 1; UMAP (umap-learn 0.5.3) to two dimensions; Louvain clustering (python-louvain 0.16) (Methods, WaveMap). The legend shows three electrodes. The paper does not give UMAP's n_neighbors or the Louvain resolution; the code uses n_neighbors 15 and picks the resolution that maximises modularity over 0.1 to 6.9 in steps of 0.1 (fig3_H_wavemap_shapes), which is followed; its notebook markdown says resolution 1.5, a variant. Which simulation (NS, E or both pooled) is meant is open; the legend of Fig. 3c names the spontaneous regime, so NS is the main reading and E and pooled are variants. Reproduced when NS and M each give six clusters with six electrodes (Methods) and the code's resolution choice, which decide; the 1.5 resolution and the three electrodes of the legend (a display choice) are variants, reported and not used for the outcome. The code's saved outputs printed 7 to 10 clusters for NS and 7 for M.
fig4a-lognormal-firing-rates
The firing rates of single units sorted by Kilosort4 from the spontaneous and evoked simulations are distributed log-normally across layers, as in Marques-Smith's data.
Like Marques-Smith’s sorted unit firing rates, the firing rates from the spontaneous (red dots, Fig. 4a ) and evoked (orange dots) biophysical simulations were distributed log-normally across layers
Results, Validation of firing rates, paragraph 2; Figure 4a
How it will be judged
Kilosort4 single units, first 10 minutes, firing rate as spike count over 600 s, by layer (code fig4_A_B_C_firing_rate_ks4). The paper fits lognormal distributions and judges the match qualitatively. The criterion reads "log-normal" as a Shapiro-Wilk test of log10 rates not rejecting normality at p < 0.05 in layers 5 and 6 (the layers with enough units, Results, paragraph 1) for NS, E and M. Reproduced when this holds in all; partially reproduced when it holds for M and one simulation. Which evoked simulation is open (the code's "npx_evoked" config, 10 minutes); both the 10-minute 40 kHz recording and the first 10 minutes of the 1-hour recording are run.
fig4a-yield-deep-layers
Single-unit yields are low in superficial layers and high in layers 5 and 6; in vivo the yields in these layers are at least 20 times that of layer 4 and no unit is found in layers 1 and 2/3, a trend also seen in the simulated data.
In vivo , the yields in these layers were at least 20 times that in layer 4, and no units were found in layers 1 and 2/3, a trend also reflected in the simulated data
Results, Validation of firing rates, paragraph 1; Figure 4a (unit counts)
How it will be judged
Kilosort4 single units of the first 10 minutes, by layer. Reproduced when, for M, each of layers 5 and 6 has at least 20 times the count of layer 4 and layers 1 and 2/3 have none, and for NS and E the counts of layers 5 and 6 each exceed those of layers 1, 2/3 and 4. "At least 20 times" can be read per layer or for layers 5 and 6 pooled; both are shown, and the claim holds under either. With 0 units in layer 4 the ratio is infinite and holds.
fig4-median-firing-rates
The median firing rate of sorted single units over layers 5 and 6 is 2.8 spikes/s in Marques-Smith's data, 1.2 in the spontaneous and 0.7 in the evoked simulations, and 5 in Buccino's synthetic simulation.
The overall median firing rate averaged over layer 5 and 6 differed but only by a small margin (Marques-Smith: 2.8 ± 1% (95% CI) spikes/secs, spontaneous biophysical simulation: 1.2 ± 0.2%, evoked simulation: 0.7 ± 0.1% and Buccino’s full synthetic simulation: 5 ± 0.4% (95% CI))
Results, Validation of firing rates, paragraph 2; Figure 4b
How it will be judged
Tolerances are the paper's 95% CI half-widths, printed with a stray "%" that the code shows to be spikes/s (fig4_A_B_C_firing_rate_ks4, cell 13 prints 2.85 +- 1.14, 1.15 +- 0.21, 0.655 +- 0.083, 4.92 +- 0.42 Hz). Kilosort4 single units, first 10 minutes. "Median averaged over layer 5 and 6" is read two ways - (a) the median of the pooled layer 5 and 6 rates, as the code does; (b) the mean of the layer 5 median and the layer 6 median. Reproduced when all four values are within tolerance under one reading. Which evoked simulation is open, as in fig4a-lognormal-firing-rates; both are run.
fig4bc-lower-median-narrower-distribution
Compared with in vivo, the biophysical simulations have lower median firing rates and narrower firing rate distributions across layers, while the synthetic model has a lower variance in layer 5 but a higher median.
Specifically, the biophysical simulations had lower median firing rates ( Fig. 4b , p < 0.001, Mann-Whitney) and distributions were narrower ( c, p < 0.001) than in vivo across layers. For reference, the synthetic model distribution also had a lower variance in layer 5 (all p < 0.01) but it displayed higher median firing rate ( p < 0.01)
Results, Validation of firing rates, paragraph 2; Figure 4b, c
How it will be judged
Kilosort4 single units, first 10 minutes, log10 firing rates per layer (Methods, Firing rate). Medians - two-sided Mann-Whitney U test of each simulation's log10 rates against M's, per layer. Variances - 100 bootstrap resamples (with replacement) of each group's log10 rates, the variance of each, and a Mann-Whitney U test of the 100 variances against M's (Methods, Firing rate; code bootstrap_varlogfr). Layers with units in M (5 and 6, and 4 if any). Reproduced when NS and E are lower than M with p < 0.001 in each tested layer for both medians and variances, and S in layer 5 has a lower variance and a higher median than M at p < 0.01; partially reproduced when the medians or the variances part holds alone.
supp4-other-sorters-firing-rates
With other sorters, Kilosort3 gives log-normal firing rate distributions that match in vivo, older sorters give bimodal distributions, and across regimes the simulated medians and variances tend to be lower than in vivo.
While newer sorters like Kilosort3 produced log-normal distributions ( Supplementary Fig. 4a ) that better matched in vivo data (b, p > 0.15), older sorters often generated bimodal distributions ... firing rate medians (b, p < 0.05) and variances ( c, p < 0.01) in simulations tended to be reduced
Results, Validation of firing rates, paragraph 3; Supplementary figure "Validation of firing rate distributions"
How it will be judged
As fig4bc-lower-median-narrower-distribution, for Kilosort, 2, 2.5 and 3 (all units for Kilosort, single units for the others), per layer. The Kilosort3 part is a claim of absence, judged on the paper's test - reproduced when no layer's Mann-Whitney U test of Kilosort3 against M has p <= 0.15 (the code's notebooks supp_fig4_ABC_ks3 run the same per-layer test). Which quantity the p > 0.15 refers to is open - the Results call panel b the medians (and c the variances), while the supplementary legend calls panel a the rate statistics and panel b the variances; both medians and bootstrapped variances are run, and the outcome is the one both give. "Bimodal" has no stated test; reported with Hartigan's dip test as a description, not for the outcome. The "tended to be reduced" part holds when, over the sorter, layer and simulation comparisons, a majority have lower medians (p < 0.05) and lower variances (p < 0.01). Reproduced when both parts hold; partially reproduced when one does.
fig5a-single-unit-fraction-differs
For the spontaneous Neuropixels simulation sorted by Kilosort4, on average 41% of sorted units are single units, and the proportion of single units differs between experiments.
We found that for spontaneous activity recorded with Neuropixels and sorted by Kilosort4, on average 41% of sorted units were ‘single-units’. This proportion differed between experiments
Results, Model predicts low yield, quality and biases, paragraph 2; Figure 5a
How it will be judged
Kilosort4, first 10 minutes, cortical units; fraction = single units / all units. The paper does not say which 5 experiments enter the test (df 4) or how it is formed, and the code holds no such test. The code forms its other tests in this panel as a goodness-of-fit chi-square of the per-experiment single-unit percentages against equal values (scipy chisquare, fig5_A_yield); its experiments in that order are M, NS, NS-2X, E and S. Readings, (a) that goodness-of-fit test on M, NS, NS-2X, E and S; (b) a chi-square contingency test of single against multi-unit counts in the same five; (c) as (b) on M, H, NS, E and DS. "On average 41%" is read three ways - (i) the fraction for NS (the sentence names it); (ii) the mean of NS and NS-2X; (iii) the mean of the M, NS and NS-2X fractions. The paper's own counts favour (iii) - the unit totals it prints (M 200, NS 504, H 358, DS 562, Results paragraph 4, checked against its N values) and N = 1,212 for M, NS and NS-2X give NS-2X 508 units, so NS is 182 / 504 = 36%, NS-2X 193 / 508 = 38% and M 96 / 200 = 48%, whose mean is 40.7%; reading (iii) also reproduces the paper's chi2 = 2.0 of fig5a-single-unit-counts. Reproduced when the fraction is within tolerance and the test has p < 0.001 under one reading.
fig5a-single-unit-counts
The simulations give at least twice as many single units as in vivo (182 and 193 against 96 for Neuropixels, 206 against 93 for the dense probe), yet the fractions of single units are similar.
At least twice more ‘single-units’ were detected from the simulations than in vivo , with the largest difference being 206 vs. 93 for the dense probe ( Fig. 5a , 182 and 193 vs. 96 for Neuropixels). Yet the fractions of ‘single-units’ were similar
Results, Model predicts low yield, quality and biases, paragraph 2; Figure 5a
How it will be judged
Kilosort4, first 10 minutes, cortical single units. The text does not name the two Neuropixels simulations; the code's test with N = 1,212 is on M, NS and NS-2X, and 193 / 182 is the 6% gain of fig5a-downsampling-small-gain, so 182 is NS and 193 NS-2X (code, fig5_A_yield, cells 10 and 19). NS-2X is NS with every second sample kept and the sampling rate left at 40 kHz (Methods, Unit isolation ratio). The fractions are a claim of absence, judged on the paper's test, which the code gives as a goodness-of-fit chi-square of the single-unit percentages against equal values (scipy chisquare); a contingency chi-square of single against multi-unit counts is shown as a variant. Reproduced when the counts are within tolerance and both tests have p >= 0.05; not reproduced when either has p < 0.05.
fig5a-synthetic-mostly-single-units
Most units sorted from Buccino's synthetic traces are single units, four times more than in vivo.
In contrast with the in vivo yield, the majority of units sorted from the synthetic traces were ‘single-units’, four times more than in vivo
Results, Model predicts low yield, quality and biases, paragraph 2; Figure 5a
How it will be judged
In "S 377 vs. 163", 163 is S's multi-unit count - N = 740 is S's 540 units (377 + 163) plus M's 200 units (96 single, 104 multi, Results paragraphs 2 and 4), and the goodness-of-fit chi-square of the single-unit percentages (S 69.8%, M 48.0%) against equal values gives 4.0, the printed value. "Four times more than in vivo" is S's 377 single units against M's 96 (3.9). Kilosort4, first 10 minutes. Reproduced when the two S counts are within tolerance, S has more than half single units, S's single units are 3.6 to 4.4 times M's (four widened by 10%), and the goodness-of-fit chi-square of the single-unit percentages of M and S (code fig5_A_yield, cell 14) has p < 0.05.
fig5a-evoked-more-units
The model predicts that in a stimulus-evoked regime the number of sorted units increases at least threefold and the single-unit yield doubles, while the ratio shifts toward multi-units.
The model predicts that the number of sorted units should increase at least three times in a stimulus-evoked regime. Although the ratio should strongly shift to ‘multi-units’, a key prediction is that the single-unit yield should double
Results, Model predicts low yield, quality and biases, paragraph 3; Figure 5a
How it will be judged
Kilosort4, first 10 minutes, cortical units, E against NS. Reproduced when E has at least 3 times NS's total units, a lower single-unit fraction, and between 1.8 and 2.2 times NS's single units (2 widened by 10%). Which evoked recording is open (the 10-minute 40 kHz or the first 10 minutes of the 1-hour 20 kHz); the code uses the 10-minute sortings of its "npx_evoked" config; both are run, and the claim holds under either.
fig5a-dense-probe-more-multi-units
In vivo, multi-units were twice as frequent with the dense probe as with Neuropixels, which the model did not reproduce; despite its higher-resolution sampling the dense probe did not give a higher single-unit yield than Neuropixels.
These were mostly ‘multi-units’, which were twice more frequent for the dense probe in vivo (M:104, H:265, χ 2 (1, N = 558) = 6.6, p = 0.01). The model did not reproduce this effect (NS:182, DS:206, χ 2 (1, N = 1066) = 0.004, p = 0.9)
Results, Model predicts low yield, quality and biases, paragraph 4; Figure 5a
How it will be judged
Kilosort4, first 10 minutes, cortical units, dense probe counts summed over the three depths. The unit totals are M 200, H 358, NS 504 and DS 562 ("358 vs 200 units and 562 vs. 504 units respectively", which the N values 558 = 200 + 358 and 1066 = 504 + 562 confirm). The test, as in the code (fig5_A_yield, cells 21 and 22), is a goodness-of-fit chi-square of the two experiments' single-unit percentages against equal values; a contingency chi-square of single against multi-unit counts is a variant. Reproduced when M against H has p < 0.05 with H's multi-unit share higher, and NS against DS has p >= 0.05 (absence, on the same test).
fig5a-downsampling-small-gain
Down-sampling the simulated traces by two to shorten spikes improves the Kilosort4 single unit yield only marginally, by 6%.
We applied this approach but observed only a marginal 6% improvement in single-unit yield
Results, Model predicts low yield, quality and biases, paragraph 5; Figure 5a (NS-2X)
How it will be judged
NS-2X as in fig5a-single-unit-counts. Gain = NS-2X single units / NS single units - 1. The 10% default tolerance is 0.006, below one unit's change at these counts (1 / 182 = 0.0055); reproduced when the gain is within 0.054 to 0.066, or when it is positive and below 0.10 as a marginal gain (a reading of "marginal"); both are reported, and the first decides.
fig5b-yield-far-below-theory
Raw single-unit yields per site are an order of magnitude lower than the theoretical yields (active neurons within 50 um per site, NS 6.9, NE 8.4, S 0.65, DS 5.3) and do not differ significantly between probes or sorters.
For the biophysical models, the raw single-unit yields were an order of magnitude lower than the theoretical yields and were not significantly different between probes or spike sorters (5b)
Results, Model predicts low yield, quality and biases, paragraph 6; Figure 5b legend
How it will be judged
Theoretical yield - ground truth neurons within 50 um with at least one spike, counted per layer of their soma, divided by the number of sites in that layer, median over layers (legend; code get_gt_yield_per_site); S is 250 / 384. Sorted yield - single units per layer divided by the sites in that layer, averaged over layers (legend "median yield per electrode, averaged across layers"; the code plots the mean over layers with seaborn's bootstrap 95% CI), for Kilosort4 and the other sorters (Supplementary figure "Single-unit yield and quality ... all tested spike sorters"). "An order of magnitude lower" is read two ways - (a) theoretical / sorted of 10 or more; (b) of 10^0.5 (3.2) or more, the nearest power of ten being 10. Neither the paper nor the code names a test for "not significantly different"; it is reported, with the yields and their CIs, and not used for the outcome. Reproduced when the theoretical yields are within tolerance and the biophysical models' Kilosort4 yields are lower by an order of magnitude under one reading. Partially reproduced when only one of the two parts holds.
fig5b-synthetic-redundant-units
In Buccino's synthetic model, Kilosort4 produces more single units than there are modelled neurons, which indicates redundant units.
In Buccino’s full synthetic model, Kilosort4 produced more ‘single-units’ than the number of modeled neurons, indicating the production of redundant units
Results, Model predicts low yield, quality and biases, paragraph 6; Figure 5b (blue)
How it will be judged
Kilosort4 single units on the first 10 minutes of S. Reproduced when their number exceeds 250.
fig5cd-most-units-degraded
Most single units isolated by Kilosort4 are less than 80% accurate, contain spikes of several true neurons (merged) and miss at least 20% of their matched neuron's spikes; Kilosort4 produces no false positives.
The majority of the isolated ‘single-units’ were less than 80% accurate and degraded across layers ( Fig. 5d , white area) and spike sorters ( Supplementary Fig. 5b ). They contained spikes from multiple true units (’merged unit’) and missed at least 20% of the matched true units spikes ( Fig. 5c , green). Kilosort4 did not produce false positives
Results, Model predicts low yield, quality and biases, paragraph 7; Figure 5c, d
How it will be judged
Kilosort4 single units of NS and E (first 10 minutes), each given its best agreement score with any ground truth neuron and its quality tags (opening comment). Reproduced when, in each simulation, more than half of the single units have a best score below 0.8, more than half are tagged merged, and none is a false positive; partially reproduced when the first part holds and one other fails. The other sorters (Supplementary figure, panel b) are reported. Which evoked recording is open; both are run.
fig5ef-curation-precision-recall
A logistic regression on unit quality metrics classifies good against degraded Kilosort4 single units with 89% and 93% precision and 61% and 55% recall in the spontaneous and evoked simulations.
The model achieved 89% ± 2% (95% CI) and 93% ± 2% precision and 61% ± 3% and 55% ± 2% recall in the spontaneous ( Fig. 5e ) and evoked simulations (f) respectively
Results, Model predicts low yield, quality and biases, paragraph 8; Figure 5e, f
How it will be judged
Tolerances are the paper's 95% CI half-widths. Units - Kilosort4 single units with all metrics computable (182 of 184 in NS). Label - good when the best agreement score is 0.8 or more (Methods, Logistic regression). Features - the quality metrics of ref. 16 named by the code (src/nodes/models/Flc), amplitude_cutoff, firing_range, firing_rate, isi_violations_ratio, rp_contamination, rp_violations, sd_ratio, snr, silhouette and mad_ratio (presence_ratio dropped as constant), z-scored. Model - Binomial GLM with logit link (statsmodels). The Figure 5 legend calls it lasso-regularized but gives no penalty; the Methods (Logistic regression model) describe a GLM with no penalty and the code fits it unpenalised, which is followed. A unit is predicted good when its probability is 0.8 or more (code; the paper is silent). Splits are open - (a) 80% train and 20% test (Methods), (b) 75% and 25% (legend "25% test units", code split_ratio 0.75); 100 random splits each. The value is the median over splits; the CI is 1.96 SD / sqrt(100) (code get_metrics_stats). The split is a method detail on which the Methods and the legend disagree, so both are run and the outcome is the one both give (methods_underspecified when they differ). Which evoked recording is meant is a reading of the reported number; both are run and either may match. Reproduced when all four values are within tolerance under both splits.
fig5g-mad-ratio-and-firing-predict-quality
The new mad_ratio metric has the largest correlation with unit sorting accuracy (R = -0.46), and the firing rate statistics and the spike trough variability metrics (mad_ratio, sd_ratio) are the key significant predictors of a good unit.
We also introduced a new metric, which we called the “mad_ratio” (see Methods), which better reflected contamination by multiple units, displaying the largest correlation with unit sorting accuracy ( R = − 0.46). The model offered the additional insight that two key predictive features of a good unit are its firing rate statistics ( Fig. 5g , including firing rate and range, p < 0.001) and the variability of its spike troughs compared to background noise variability (mad_ratio and sd_ratio, p < 0.001)
Results, Model predicts low yield, quality and biases, paragraph 8; Figure 5g
How it will be judged
NS Kilosort4 single units as in fig5ef-curation-precision-recall. The mad_ratio is not defined in this text's Methods; the code defines it as the mean absolute deviation of the unit's negative-peak spike amplitudes over that of all waveform samples on its extremum channel (get_mad_ratio). The code's R is the Pearson correlation of mad_ratio with the binary good label (fig5_G_curation_metrics_npx_spont, cell 13); the text says "accuracy", so the Pearson correlation with the continuous best agreement score is a second reading. Weights - the unpenalised GLM on all units, z-scored metrics, Wald p-values. Reproduced when R is within tolerance and is the largest in magnitude among the metrics under one reading, and the weights of firing_rate, firing_range, mad_ratio and sd_ratio have p < 0.001.
supp5-cebra-knn-curation
A CEBRA embedding of full waveforms with a k-nearest neighbour classifier gives better recall (82% and 90%) but lower precision (61% and 55%) than the logistic regression in the spontaneous and evoked regimes.
The model achieved better recall (82%, ± 3% and 90% ± 2% for the spontaneous ( Supplementary Fig. 5c ) and evoked (d) regimes respectively) but lower precision (61% ± 3% and 55% ± 3%) than the logistic regression
Results, Model predicts low yield, quality and biases, paragraph 9; Supplementary figure "Single-unit yield and quality ...", panels c, d
How it will be judged
Tolerances are the paper's CI half-widths. Settings from the Methods (cebraspike model) and the code (src/nodes/models/CebraSpike) - Kilosort4 units with more than 25 spikes; the first 25 spike waveforms, +-3 ms on the extremum channel, with the quality metrics appended; good = best score 0.8 or more, poor = merged and partial without good (code); CEBRA offset10-model, cosine distance, 40 iterations, learning rate 0.001, output dimension 3, temperature auto, seed 0; KNN decoder with 20 neighbours, cosine; stratified 80 / 20 splits, 100 seeds; median and 1.96 SD / sqrt(100). The code appends the metrics in sorted unit order while the waveforms keep their own order; a version with the rows aligned is shown as a variant. Reproduced when all four values are within tolerance, under the code's version or the aligned one.
fig6a-synthetic-accuracy-range
On Buccino's synthetic simulation most spike sorters detect most neurons, with accuracies ranging from 50% for herdingspikes to 99% for Kilosort2.
As previously reported, for the synthetic simulation [ 16 ] most spike sorters detected most neurons with accuracies ranging from 50% for herdingspikes to 99% for Kilosort2
Results, Biological factors of sorting failures, paragraph 2; Figure 6a
How it will be judged
The ground truth of S is its 250 units. Accuracy per neuron and the 0.8 rule as in the opening comment; the fraction of neurons with accuracy of 0.8 or more per sorter (Figure 6a legend). Which sorted units enter the best match - the Methods (Sorting accuracy) say only "its best-matching sorted unit", and the code settles it - single units for Kilosort 2 to 4 and all units for Kilosort and herdingspikes (code get_accuracy), which decides; all units for every sorter is shown as a variant. Reproduced when both values are within tolerance and a majority of the six sorters have a fraction above 0.5.
fig6ab-biophysical-accuracy-15-percent
For the biophysical models the sorters well-sort similarly low proportions of ground truth neurons, on average 15% of the neurons within 50 um of the electrodes.
However, the sorters produced similarly low proportions of well-sorted ground truth neurons for the biophysical models ( Fig. 6a ), on average 15% of the neurons within 50 µm of the electrodes across the simulations
Results, Biological factors of sorting failures, paragraph 2; Figure 6a, b
How it will be judged
Figure 6 legend - the 10-minute evoked simulation at 40 kHz and NS; accuracy of 0.8 or more. Per sorter, the well-sorted neurons summed over the ground truths divided by the neurons within 50 um summed (Figure 6b legend). N = 3,190 is 250 + 1,388 + 1,552, so the code sums S, NS and E (fig6_A_B_sorting_accuracy, cell 5); the "biophysical" sum over NS and E alone is a second reading. Best-match units as in fig6a-synthetic-accuracy-range (the code's choice decides). The test, as in the code (cell 12), is a goodness-of-fit chi-square of the six per-sorter percentages against equal values; a contingency chi-square of well-sorted against other neurons per sorter is a variant. Reproduced when the mean over sorters is within tolerance under one reading and the chi-square has p >= 0.05 (the paper's "similarly", on its test).
fig6-dense-probe-lower-accuracy
The dense probe gives five times fewer neurons sorted with above 80% accuracy than the Neuropixels probe, on average 3% against 15% across sorters.
We found that the dense probe produced five times fewer neurons sorted with above 80% accuracy across spike sorters than the Neuropixels probe (on average 3% vs. 15%)
Results, Biological factors of sorting failures, paragraph 3; Figure 6a, b
How it will be judged
DS depths pooled (Figure 6b legend), well-sorted neurons divided by ground truth neurons within 50 um. The 10% default tolerance on 3% is below the resolution of a sum over a few hundred neurons per sorter; reproduced when both values are within tolerance under one reading of fig6ab-biophysical-accuracy-15-percent, or when the Neuropixels value is within tolerance and the ratio of Neuropixels to dense is within 4.5 to 5.5 (five widened by 10%).
fig6c-f-accuracy-predictors
A fractional regression of Kilosort4 sorting accuracy on 21 neuron features fits the data (R2 = 0.42 spontaneous, 0.34 evoked) and finds seven significant features in both regimes; accuracy rises with firing rate and spike spatial extent, is higher for pyramidal cells and in layer 5, and falls with distance to the nearest electrode.
The model provided a good fit to the data ( Fig. 6 , R 2 = 0.42 and R 2 = 0.34 in the spontaneous (c) and evoked (d) regimes respectively) and identified seven significantly predictive features out of 21 in both regimes. Accuracy improved as neuronal firing rates increased and spike spatial extent (or ‘footprint’ [ 6 ]) expanded. It was also higher for pyramidal cells compared to interneurons and in layer 5 compared to other cortical layers. Sorting accuracy decreased when neuron’s distance to the nearest electrode increased
Results, Biological factors of sorting failures, paragraph 4; Figure 6c-f
How it will be judged
Tolerance 0 on the feature count because it is an integer and 10% of 7 is below one. Neurons - ground truth neurons within 50 um; NS and the 1-hour evoked simulation at 20 kHz (Figure 6 legend), Kilosort4 on the first 10 minutes (code). Target - accuracy as the best agreement score with any Kilosort4 unit (code, all units). Features - the 21 of the code (fig6_C_E_fracreg_NS_ks4, cell 4) - firing_rate, synapse_class, etypes bAC, bNAC, bSTUT, cACint, cADpyr, cNAC, cSTUT, dNAC, dSTUT, layers 1, 2/3, 4, 5 and 6, distance to the nearest site, snr, spatial_extent, exc_mini_frequency and dynamics_holding_current, z-scored. Model - Binomial GLM, logit link, statsmodels fit_regularized elastic_net with L1_wt 1, maxiter 100, cnvrg_tol 1e-10 (Methods). The lasso penalty strength is not given anywhere; the code passes none, so statsmodels' alpha 0 applies, which is followed. With alpha 0 no weight is set to zero, which does not match the Methods' account of the lasso selecting features; this is recorded as a deviation, and a penalised fit cannot be run from the paper. R2 - McFadden's pseudo-R2 on test sets of 100 random 75 / 25 splits, median (code); in-sample R2 is a variant. Significance - Wald p < 0.05 of the weights of a one-step unpenalised refit on all neurons (code). Reproduced when both R2 are within tolerance, each regime has seven significant features, and the signs are positive for firing rate, spatial extent, synapse class (excitatory) and layer 5 and negative for distance in both regimes. Partially reproduced when the signs hold but R2 or the counts do not.
supp6-sorted-rates-overestimated
Because sorting favours highly active units, mean firing rates in most layers are overestimated three to four times by sorted spike trains compared with the ground truth.
Notably, the bias for high firing units led to mean firing rates in most layers being overestimated by a factor of three to four from sorted spike trains compared to the ground truth
Results, Biological factors of sorting failures, paragraph 4; Supplementary figure "Ground truth firing rate distribution"
How it will be judged
A reported range of 3 to 4 widened by 10% of each bound gives 2.7 to 4.4 (written as 3.5 +- 0.9 above). NS; true rates of all ground truth neurons within 50 um over the first 600 s, sorted rates of Kilosort4 single units of the first 10 minutes, per layer (legend; code supp_fig7_ABCDE_true_firing_rates_revised, which also bootstraps 100 draws of 100 values). Ratio of the means per layer, assigned by soma layer for true and by site layer for sorted. Reproduced when the ratio lies in 2.7 to 4.4 in a majority of the layers that have sorted units.
fig6gh-selection-bias-layer5-pyramidal
Pyramidal neurons, 91% of the ground truth, are under-represented after sorting by about a factor of two in every layer except layer 5, where they are over-represented by about two; most interneurons are represented in proportion, but rare interneuron types are over-represented by 2.9 to 8.1 by some sorters, mainly Kilosort 1 to 2.5.
Excitatory pyramidal neurons, which made up 91% of the ground truth population ( Fig. 6g ), were consistently underrepresented across all layers by approximately a factor of 2, except in layer 5, where they were over-represented by the same factor ( Fig. 6h , ratios in blue; Fisher’s exact test, two-tailed, p < 0.05). In contrast, most interneurons were proportionally represented relative to the ground truth, except for rare interneuron types, which were over-represented by factors ranging from 2.9 to 8.1 by some spike sorters, particularly Kilosort (1 to 2.5)
Results, Biological factors of sorting failures, paragraph 5; Figure 6g, h
How it will be judged
The 1-hour evoked simulation at 20 kHz (Figure 6 legend). Types are synapse class, layer and electrical type. Ground truth - all neurons within 50 um. Sorted - single units of Kilosort 2 to 4, all units of Kilosort and herdingspikes, each given the synapse class and electrical type of its best-matching neuron (Results). Its layer is the matched neuron's layer, as the Results and the Figure 6h legend say ("assigned the biophysical features of its best ground truth match"), which decides; the code's site layer (fig6_G_H_selection_bias, get_feature_data_for) disagrees with them and is a variant. Which sorting duration is open; the code's table is named "1h" but its generator sorts 600 s; both are run where possible, and the outcome is the one both give (methods_underspecified when they differ). Ratio - share of a type among sorted units / its share in the ground truth. Test - per type and sorter, two-sided Fisher's exact test on [[ground truth of the type, sorted of the type], [ground truth others, sorted others]] (code cell 18), p < 0.05, uncorrected. Reproduced when, for a majority of sorters, the excitatory types of layer 5 have ratios of 1.5 or more and those of the other layers ratios of 0.67 or less, with p < 0.05 for those that differ; and the largest ratio of an interneuron type is within 2.6 to 8.9 for at least one of Kilosort, 2 or 2.5. Partially reproduced when only the excitatory part holds.
fig7a-ring-manifold
The ground truth neural manifold has a ring-like geometry tracking stimulus direction; spike sorting distorts it for all sorters, but the recent sorters (Kilosort2 to 4) still preserve the stimulus mapping.
The ground truth neural manifold exhibited a ring-like geometry ( Fig. 7a ), which continuously reflected changes in the stimulus direction across trials. Spike sorting distorted the manifold’s geometry for all spike sorters, but the most recent sorters still preserved stimulus mapping (from Kilosort2 to 4)
Results, Spike sorting degrades stimulus representation, paragraphs 1 and 2; Figure 7a
How it will be judged
A visual finding with no stated measure. Responses are spike counts per unit in each 200 ms stimulus presentation, 8 directions 45 degrees apart, 50 trials each, 1-hour evoked simulation (Results; Figure 7 legend; code get_evoked_responses), reduced to 3 dimensions by PCA. The criterion reads "ring-like, continuously reflecting direction" as the trial-averaged centroids of the 8 directions, in the plane of the first two PCs, having angular order equal to the direction order (cyclic, either sense), and "preserved mapping" as the same for a sorter's population. Reproduced when the ground truth and Kilosort2, 2.5, 3 and 4 keep the cyclic order.
fig7b-sorting-halves-capacity
Spike sorting significantly reduces the information capacity of the population, on average by 50% across sorters, from 25% above the lower bound.
Distortions in manifold geometry were associated with significant reductions in information capacity, averaging 50% decrease across spike sorters, from 25% above the lower bound ( Fig. 7b , t(199) ranging from -68 to -24, all p < 0.001, t-test)
Results, Spike sorting degrades stimulus representation, paragraph 3; Figure 7b
How it will be judged
Method (Methods, Manifold information geometry; Figure 7 legend; code igeom.py) - 1-hour evoked simulation, 8 directions, 50 trials each, spike counts in each 200 ms presentation; sorted populations are single units of Kilosort 2 to 4 and all units of Kilosort and herdingspikes; neural_manifolds_replicaMFT, kappa 0, n_t 200; each population projected to 200 dimensions (the code uses a Gaussian random projection with unit-norm rows, as in ref. 52); capacity divided by that of the same data with shuffled direction labels. Bootstraps - 200 (legend, code) or 1000 (Methods); the code varies only the projection and shuffle seeds, which is followed, with 200; 1000 is a variant. Test - the code's two-sample t-test (scipy ttest_ind) of the sorter's against the ground truth's values; a one-sample t-test against the ground truth mean (matching the printed df 199) is a variant. "50% decrease from 25% above the lower bound" is read two ways - (a) normalised capacity falls by half of its ground truth value; (b) the excess above the lower bound falls by half (from 0.25 to about 0.125). Reproduced when the ground truth ratio is within tolerance, every sorter is lower with p < 0.001, and the mean decrease is within tolerance under one reading.
fig7f-undersampling-no-loss-selection-bias-loss
Randomly under-sampling ground truth neurons to the sorters' yields does not change information capacity, but also matching the neuron type distribution of the sorted units reduces it by an average 21% across sorters.
Under-sampling ground truth neurons to match sorting yields neither disrupted the manifold’s structure nor its information capacity (example for Kilosort4, Fig. 7d , and all sorters in f, red boxes vs. red line, t(199) ranging from -0.71 to -0.02, all p > 0.47, t-test). However, also matching neuron type distribution to that of sorted units visibly degraded the manifold structure and its information capacity by an averaged 21% across sorters
Results, Spike sorting degrades stimulus representation, paragraph 4; Figure 7d-f
How it will be judged
Capacity as in fig7b-sorting-halves-capacity. Random sample - as many ground truth neurons as the sorter's sorted units, without replacement, a new draw per bootstrap (code get_igeom_stats_for_gt_and_random_sampling). Type-matched sample - for each type (layer, synapse class, electrical type), as many ground truth neurons of that type as the sorter has sorted units of it, without replacement unless too few (code sample_gt_based_on_sorting_distribution). The undersampling part is a claim of absence judged on the paper's t-test (both forms of fig7b-sorting-halves-capacity shown; the code's two-sample test decides) - holds when no sorter's random sample differs from the ground truth at p < 0.05. Type-matched part - every sorter lower at p < 0.001 and the mean decrease within tolerance under one of the two readings of a decrease in fig7b-sorting-halves-capacity. Reproduced when both parts hold; partially reproduced when one does.
fig7g-assignment-bias-capacity
Good and degraded sorted units differ in type distribution; good units have on average 37% lower capacity than the ground truth, and degraded units, matched to the good units' type distribution, a further 12% lower for most sorters.
Unit type distributions differed significantly between biased and good units across spike sorters ( p < 0.001, Fisher exact test with Monte Carlo simulations) ... On average, the information capacity of good units was 37% lower than the ground truth across spike sorters ( Fig. 7g , red boxes vs. red line, t(199) ranging from -0.31 to -15, all p < 0.001, t-test). Degraded units exhibited even lower capacity, averaging a 12% decrease from good units’ capacity across most sorters (Kilosort4, 2.5, 1, and herdingspikes)
Results, Spike sorting degrades stimulus representation, paragraph 5; Figure 7g
How it will be judged
Capacity as in fig7b-sorting-halves-capacity. Good units are tagged good; degraded units are merged or partial and not good (code - "mixed good + overmerger + oversplitter" against "mixed overmerger + oversplitter"). Degraded units resampled per type to the good units' type counts, with replacement (code sample_pop_based_on_ref_distribution; the code then drops duplicates with np.isin and does not reset its arrays between the two classes, both shown as variants of a corrected version, which decides). Type distribution test - a Fisher exact test on the type by class table with Monte Carlo p-values (R fisher.test simulate.p.value, 10^5 replicates, a standard choice). The paper's printed range includes t(199) = -0.31, which gives p of about 0.76, so its "all p < 0.001" and its t range disagree; the per-sorter t and p are reported and the disagreement is a deviation. Reproduced when the type test has p < 0.001 for every sorter, good units are below the ground truth at p < 0.001 for every sorter with a mean decrease within tolerance under one reading of a decrease, and degraded units are below good units for Kilosort4, 2.5, 1 and herdingspikes with a mean decrease within tolerance. Partially reproduced when the first two parts hold.
fig7h-j-manifold-geometry
The capacity loss comes with larger direction manifold radii (6% on average, except a 3% decrease for Kilosort), larger dimensions (6% on average, except a 19% decrease for Kilosort and Kilosort2) and slightly more correlated centroids (1% for Kilosort3, 2, 1 and herdingspikes).
Except for Kilosort which showed a 3% decrease, direction mani-folds’ radii increased by an averaged 6% across sorters ( Fig. 7h , t(199) ranging from -87 to 0.18, all p < 0.001, t-test). Except for Kilosort and Kilosort2 which displayed a 19% decrease in direction manifold’s dimension, dimension also increased by an averaged 6% across sorters (i, t(199) ranging from -59 to 234, all p < 0.001). A marginal but significant increase in center correlation of 1% was observed for Kilosort3, 2, 1, and Herdingspikes
Results, Spike sorting degrades stimulus representation, paragraph 6; Figure 7h-j
How it will be judged
Radius, dimension and centre correlation from neural_manifolds_replicaMFT, as in fig7b-sorting-halves-capacity, relative to the ground truth's. The printed radius range includes t(199) = 0.18, which cannot give p < 0.001, so "all p < 0.001" and the t range disagree; per-sorter t and p are reported. Whether the averages include the excepted sorters is open; both are shown, and the expected values are read over the other sorters. Reproduced when the radius increases for every sorter except Kilosort and the dimension for every sorter except Kilosort and Kilosort2, each at p < 0.001, with mean increases within tolerance, and the centre correlation is higher for Kilosort3, 2, 1 and herdingspikes. Partially reproduced when the directions hold but the means do not.

Pipeline

Paper: done (1 of 1)
Claims: done
Claims check: done (ready)
Attempt: todo
Questions: none
Verified: none