{ "dataset": { "archive": "dandi", "id": "001051", "label": "DANDI:001051" }, "page": "datasets/dandi/001051/", "generated": "2026-11-05T07:41:23+00:00", "caution": "A failure to reproduce is not a finding that the paper is wrong. It may be a failure of the attempt.", "outcome_counts": { "reproduced": 10, "partially_reproduced": 5, "not_reproduced": 4, "blocked": 20 }, "outcome_counts_with_outside_data": { "reproduced": 3, "partially_reproduced": 0, "not_reproduced": 1, "blocked": 2 }, "outcome_counts_elsewhere": { "reproduced": 0, "partially_reproduced": 0, "not_reproduced": 0, "blocked": 0 }, "outcome_counts_links": { "reproduced": 0, "partially_reproduced": 1, "not_reproduced": 0, "blocked": 0 }, "attempts": [ { "id": "001051-bennett-2024-01", "dataset": { "archive": "dandi", "id": "001051", "version": "0.260922.1827", "doi": "10.48324/dandi.001051/0.260922.1827" }, "key": "dandi:001051", "label": "DANDI:001051", "dataset_version": "0.260922.1827", "paper": "10.1016/j.neuron.2024.06.015", "paper_title": "SHIELD: Skull-shaped hemispheric implants enabling large-scale electrophysiology datasets in the mouse brain", "date": "2026-11-01", "kind": "agent", "accountable": "CatalystNeuro", "agent": { "harness": "Claude Code", "model": "claude-opus-5-5", "protocol": "pilot-13" }, "runner": { "platform": "GitHub Actions", "url": "https://github.com/catalystneuro/encore/actions/runs/41270696677" }, "tier": "runs", "deprecated": "", "deprecated_on": null, "entry": "001051-bennett-2024", "entry_title": "SHIELD multi-Neuropixels recordings and alpha events (Bennett et al. 2024)", "description": "Tests the findings of Bennett, Ouellette, Ramirez et al. (2024) on dandiset 001051, which holds 99 change-detection sessions of 27 mice recorded with up to six Neuropixels probes through a SHIELD implant. It checks the paper's session list and unit table against the files, the unit yield, quality and stability across recording days, receptive fields and visual responses, and the alpha events of VISp and MOp with their coherence networks and the striatal subnetworks. The surgery, imaging, histology, MRI and optogenetics material of Figures 3, 5 and 6 is not in the dandiset.", "notebooks": [ { "path": "reproduce_bennett_2024.ipynb", "title": "Reproducing Bennett et al. (2024) from dandiset 001051", "executed": "notebooks/001051-bennett-2024/reproduce_bennett_2024.html" } ], "outcomes": [ { "claim": "link-code-tables", "statement": "The 87 sessions the paper's code lists as passing ephys QC are sessions of the dandiset, and for them the code's units table agrees with the dandiset's units (unit ids, counts per session, quality metrics and receptive-field p values).", "source": "STAR Methods, Key resources table and Data and code availability; DynamicGating_ephys_metrics.ipynb and metadata_tables/units_table.csv at commit 3c09a0dc", "quote": "Analysis code used in this paper is available here", "statistics": [], "kind": "link", "inferred": "", "outcome": "partially_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "All 87 listed sessions are dandiset sessions (by the NWB identifier), and session type, mouse and genotype agree for all 99 sessions. In 81 of 87 sessions the code's units table and the dandiset agree unit by unit (no matched unit differs in max_drift, firing_rate or quality). In 6 sessions (1176989662, 1178460518, 1178693650, 1179911454, 1180266229, 1181096406) 55 to 99.8% of matched units differ in max_drift and firing_rate and 14 to 38% in quality; there the files' spike counts fit the files' firing rates (coefficient of variation of n_spikes / firing_rate 0 in five sessions, 0.076 in 1176989662) but not the code's (5.4 to 11.8, against at most 0.0003 elsewhere), and the RF p values recomputed here are uncorrelated with the code's (r = 0.03 over 251 units, against 0.9990 over 7095 units in the other sessions), so the same unit ids name other units. In 5 of them the code's table has 328 units (133 CP) that are not in the dandiset, each session's on one probe_id of the code's (38 to 114 units) that is not among the file's probe_ids and has no unit in the file, and none with a peak channel of a dandiset unit; the dandiset has no unit missing from the code's table. Where the units agree, recomputed RF significance at 0.01 agrees for 98.8% of units pooled; per session the fraction differing has a median of 0.008 and a maximum of 0.038 and exceeds 1% in 33 of 79 sessions, all 83 differing units having p values of 0.003 to 0.020 in both computations, at the threshold (two bootstraps with different random draws). The dependent claims are tested on all 87 sessions; leaving out the 6 changes their numbers little (shown for the unit counts, drift, QC fraction and yield).", "deviations": [ "A link claim added by the author; the code's notebook and tables are matched to the dandiset by session and unit identifiers." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig3b-surgical-outcomes", "statement": "Of 106 mice entering SHIELD surgery, 79 passed all quality checks, 14 passed after a flag that resolved and 13 failed, a final passing rate of 87.7%, on par with 90% (122 of 135) for the earlier 5-mm cranial window surgery.", "source": "Results, Installation of SHIELD implant and QC evaluation, paragraph 3; Figure 3B", "quote": "Of the 106 mice entering surgery for this study, 79 passed all quality checks with no issues, 14 passed with a flag that eventually resolved, and 13 mice failed ... The final passing rate of 87.7% is on par with the success rate for our previous 5 mm cranial window surgery", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "Surgery records and QC flags for the 106 mice entering surgery are not in the dandiset, which holds only the 27 recorded mice (99 sessions); neither the files, the dandiset metadata nor the code's tables hold them.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig3c-isi-through-implant", "statement": "Intrinsic signal imaging through the implant maps the retinotopic boundaries of VISp and many higher visual areas.", "source": "Results, Installation of SHIELD implant and QC evaluation, paragraph 4; Figure 3C", "quote": "We found that standard procedures successfully mapped retinotopic boundaries of VISp and many higher visual cortical areas (Figure 3C)", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "No intrinsic signal images or sign maps are in the dandiset; the files hold only an ISI-derived area label per unit (isi_label).", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig3d-histology-intact", "statement": "The implanted left hemisphere shows little GFAP activation or loss of DAPI and NeuN staining compared with the unaltered right hemisphere (6 mice, 56 days after surgery).", "source": "Results, Installation of SHIELD implant and QC evaluation, paragraph 5; Figures 3D and S3E", "quote": "found little evidence of GFAP activation or cell loss, indicating that the implant did not compromise the underlying tissue", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "No histology (GFAP, DAPI, NeuN sections) is in the dandiset or the paper's repository.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "tracks-rare-bleeding", "statement": "Inserted probes rarely caused significant intracranial bleeding, in 3 of 547 tracks.", "source": "Results, Installation of SHIELD implant and QC evaluation, paragraph 5", "quote": "We found that inserted probes rarely caused significant intracranial bleeding (3/547 tracks).", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "Probe-track histology is not in the dandiset; the files give CCF positions of channels only, with no record of bleeding.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig3fh-mri-deformation", "statement": "In MRI, the 5-mm cranial window deforms the brain under the window, while the brain under SHIELD implants (250 and 400 micrometre inserts) is close to naive mice; the thicker insert trended toward less deformation.", "source": "Results, Installation of SHIELD implant and QC evaluation, paragraph 6; Figures 3F to 3H", "quote": "revealed clear brain deformation for the non-contoured 5-mm window over the visual cortex ... the area between surface contours for SHIELD mice was more similar to naive mice ... though the thicker design trended toward less deformation", "statistics": [ { "test": "Wilcoxon rank-sum test, area between contours summed over the 4 most posterior sections", "p": 0.05, "p_relation": "<", "corrected": "Benjamini-Hochberg", "n": "naive 3, SHIELD 250 4, SHIELD 400 5, 5-mm window 5 mice" } ], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "No MRI volumes or contours are in the dandiset, and the naive-mouse data of Qiu et al. 2018 would only be the comparison; the measurement itself is missing.", "deviations": [], "assumed": false, "disagreed": false, "data": "with_inputs", "where": null, "proposed": null }, { "claim": "fig3i-learning-similar", "statement": "SHIELD mice and 5-mm window mice learn the change detection task at similar rates (median sessions to criterion 12.5 against 14, p = 0.17).", "source": "Results, Installation of SHIELD implant and QC evaluation, paragraph 7; Figure 3I", "quote": "Mice with either implant had similar learning rates (Figure 3I; median number of sessions to criterion: SHIELD 12.5 \u00b1 2; 5-mm window 14 \u00b1 3; p = 0.17, Wilcoxon rank-sum test)", "statistics": [ { "test": "Wilcoxon rank-sum test", "p": 0.17 } ], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "Training histories (sessions to criterion) of SHIELD and 5-mm window mice are not in the dandiset or the paper's repository; the dandiset holds only the recording sessions.", "deviations": [], "assumed": false, "disagreed": false, "data": "with_inputs", "where": null, "proposed": null }, { "claim": "fig4b-units-areas-per-session", "statement": "A typical 6-probe session records 1,821 non-noise units (SD 341.9) from 28.1 areas (SD 5.1).", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 2; Figure 4B", "quote": "we recorded activity from over 1,800 units spanning ... 25 brain areas (this includes all non-noise units, as described in STAR Methods; 1,821 \u00b1 341.9 units; 28.1 \u00b1 5.1 areas, mean \u00b1 SD", "statistics": [], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "44 sessions of the 87 have in-area units on exactly six probes; they have 1821.0 non-noise units (SD 341.9) and 28.11 areas (SD 5.12), the paper's values; unchanged without the 6 sessions that differ from the code's table.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "figs4ab-insertions-per-mouse", "statement": "Across recording days each mouse received about 20 probe insertions (SD 4.1) into about 10 implant holes (10.4, SD 2.3).", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 2; Figures S4A and S4B", "quote": "we typically made ... probe insertions (20 \u00b1 4.1, mean \u00b1 SD; Figure S4A) targeting ... implant holes (10.4 \u00b1 2.3, mean \u00b1 SD; Figure S4B)", "statistics": [], "kind": "finding", "inferred": "", "outcome": "partially_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "Insertions per mouse, mean 20.28 (SD 3.34) over the 87 sessions (reading a) and 21.41 (SD 3.33) over all sessions (b), within 20 +- 4.1; with only the 467 insertions of the code's table, 18.68. The implant holes per mouse (10.4 +- 2.3) could not be measured, as no file records the hole of an insertion; that part is untested, so the claim holds only in the part that could be tested.", "deviations": [ "Holes per mouse not tested (no hole per insertion in the dandiset or the code's tables)." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "scale-mice-sessions-insertions", "statement": "Stable recordings were made from 25 mice in 87 experiments with 467 probe insertions.", "source": "Discussion, paragraph 2", "quote": "high-quality recordings from 25 mice, totaling 87 experiments and 467 probe insertions", "statistics": [], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "Over the 87 sessions 25 mice, 87 experiments and 507 probes with units; 507 is within 467 +- 46.7. Of these 467 are in the code's units table; the other 40 probes carry 2004 units, 1873 on 'Track not annotated' channels, which the paper's tables leave out. Over all 99 sessions 27 mice and 578 probes with units.", "deviations": [ "The dandiset holds 40 probe recordings in the 87 sessions that the paper's tables leave out; nothing in the files marks them, so insertions are counted as probes with units, with the code's 467 shown." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "penetration-success", "statement": "On average 5.46 of the 6 intended probes per session were successfully inserted.", "source": "STAR Methods, Neuropixels recordings, Probe insertion, paragraph 2", "quote": "Overall, we achieved a penetration success of 5.46 probes per session", "statistics": [], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "507 probes with units over 87 sessions give 5.83 per session (a), 578 over 99 give 5.84 (b), within 5.46 +- 0.546; with the code's 467 insertions, 5.37. The code's probes table lists 467 probes for the 87 sessions and 533 for the 99 (5.37 and 5.38 per session), every one marked inserted, so it records no failed attempt. No count gives 5.46.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig4b-trajectory-consistency", "statement": "Insertions along the same trajectory (same hole and probe orientation) enter the brain at similar CCF locations, with a median absolute deviation of 150 micrometres medial-lateral and 200 micrometres anterior-posterior over 15 trajectories.", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 3; Figure 4B", "quote": "for 15 unique trajectories, we found the median absolute deviation in CCF coordinates was 150 mm in the medial-lateral axis and 200 mm in the anterior-posterior axis", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "metadata_insufficient", "notebook": null, "notes": "The hole and probe orientation of each insertion are not recorded, so insertions cannot be grouped into trajectories; the files give CCF channel positions only.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig4c-drift-discontinuities-rare", "statement": "Discontinuous or abrupt shifts in unit depth in the drift map occurred in 5.1% of insertions (23 of 454).", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 4; Figure 4C", "quote": "We rarely observed discontinuous or abrupt shifts in the depth of units in the drift map (5.1% of insertions; 23/ 454)", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "methods_underspecified", "notebook": null, "notes": "The paper judged drift maps by eye and gives no rule for an abrupt shift, and the files do not hold per-spike depths (only unit peak channels and spike amplitudes), from which drift maps are drawn.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig4d-unit-drift", "statement": "Unit drift with SHIELD is only slightly higher than with the 5-mm window (median 25.1 against 22.82 micrometres).", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 4; Figure 4D", "quote": "found it was only slightly higher than that observed in our previous recordings using the 5-mm insertion window method (SHIELD: 25.1 \u00b1 0.1 mm, 5-mm window: 22.82 \u00b1 0.05; median \u00b1 SEM", "statistics": [], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "Median max_drift 25.05 um over 144,030 SHIELD non-noise units (paper 143,828) and 22.82 over 262,177 VBN non-noise units, within 25.1 +- 0.1 and 22.82 +- 0.05; 25.22 without the 6 differing sessions.", "deviations": [ "SHIELD units include 202 more non-noise units than the paper's count, from the 40 probe recordings the paper's tables leave out (143,828 non-noise units on the 467 insertions in the code's table, 202 on the other 40). The code's own table has 144,068 for the 87 sessions; outside the 6 sessions where it describes other units it and the files both give 135,334." ], "assumed": false, "disagreed": false, "data": "with_inputs", "where": null, "proposed": null }, { "claim": "fig4d-qc-metrics-comparable", "statement": "The distributions of unit quality metrics (presence ratio, amplitude cutoff, ISI violations, SNR, isolation distance, drift, amplitude) are comparable for SHIELD and the 5-mm window.", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 5; Figure 4D", "quote": "We found that the distribution of QC metrics was comparable with both surgical preparations (Figure 4D).", "statistics": [], "kind": "finding", "inferred": "", "outcome": "not_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "Reading (a), the code's Mann-Whitney test, finds all 7 metrics different (p 3.9e-292 to 3.9e-6), as the code's printed output does; reading (b), medians within 10%, holds for 5 of 7 (isi_violations ratio 0.79, amplitude_cutoff 0.89). Both readings give the same outcome. Effect sizes are small (rank-biserial at most 0.07 in absolute value) and the two metrics outside 10% are better for SHIELD.", "deviations": [], "assumed": false, "disagreed": false, "data": "with_inputs", "where": null, "proposed": null }, { "claim": "qc-pass-fraction-similar", "statement": "The fraction of units passing the standard quality filter is very similar for SHIELD (41.1%) and the 5-mm window (40.7%).", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 5", "quote": "the percentage of units passing our standard quality metrics filter (see STAR Methods) was very similar for the SHIELD versus 5-mm window implants (% units passing quality metrics criteria: SHIELD: 41.1, 5-mm window: 40.7)", "statistics": [], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "41.09% of SHIELD and 40.69% of VBN non-noise units pass the QC filter.", "deviations": [], "assumed": false, "disagreed": false, "data": "with_inputs", "where": null, "proposed": null }, { "claim": "fig4e-unit-yield-stable", "statement": "About 300 units are recorded per probe on each day, and unit yield does not differ across the four recording days with SHIELD (p = 0.48).", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 6; Figure 4E", "quote": "On each recording day, we measured the activity of roughly 300 units from each probe (Figure 4E). ... unit yield was not significantly different across days for SHIELD mice (p = 0.48, Kruskal-Wallis H test)", "statistics": [ { "test": "Kruskal-Wallis H test", "p": 0.48, "n": "129, 126, 121 and 91 insertions on days 1 to 4" } ], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "Over all 500 insertions with non-noise units, medians 308, 302, 287.5 and 277 per day and Kruskal-Wallis p = 0.333; over the 467 insertions of the code's table (129, 126, 121 and 91, the figure's counts) medians 314, 305.5, 295 and 303 and p = 0.477. Every median is within 300 +- 30.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig4e-vbn-yield-drops", "statement": "With the 5-mm window, unit yield decreased on the second recording day (p = 6e-22).", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 6; Figure 4E", "quote": "In contrast to the 5-mm window preparation, for which unit yield decreased on the second recording day (p = 6e\u201322, Kruskal-Wallis H test)", "statistics": [ { "test": "Kruskal-Wallis H test", "p": 6e-22, "n": "459 and 446 VBN insertions on days 1 and 2" } ], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "VBN medians 321 (459 insertions) and 261 (446) on days 1 and 2; Kruskal-Wallis p = 1.2e-24 for day 1 against day 2 (a) and 1.2e-19 over the code's six groups (b).", "deviations": [], "assumed": false, "disagreed": false, "data": "with_inputs", "where": null, "proposed": null }, { "claim": "fig4e-firing-rate-stable", "statement": "The average firing rate of recorded units is similar across the four recording days (p = 0.21).", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 6; Figure 4E", "quote": "Additionally, the average firing rate of recorded units was similar across days (p = 0.21 Kruskal-Wallis H test).", "statistics": [ { "test": "Kruskal-Wallis H test", "p": 0.21, "n": "23, 24, 22 and 18 sessions on days 1 to 4" } ], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "Kruskal-Wallis p = 0.223 over 23, 24, 22 and 18 sessions (text 0.21, code 0.26).", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig4ef-rf-fraction-stable", "statement": "A consistent fraction of VISp units has significant spatial receptive fields on each of the four recording days (p = 0.38).", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 7; Figures 4E and 4F", "quote": "we measured a consistent fraction of units with well-defined spatial receptive fields (RFs) (Figure 4E and 4F; p = 0.38 Kruskal-Wallis H test)", "statistics": [ { "test": "Kruskal-Wallis H test", "p": 0.38, "n": "12, 11, 12 and 11 sessions on days 1 to 4" } ], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "With RF p values recomputed here and p < 0.01, median fractions 0.900, 0.877, 0.913 and 0.900 over 12, 11, 12 and 11 sessions, Kruskal-Wallis p = 0.671; at 0.05, p = 0.368; with the code's p values, 0.864 and 0.813.", "deviations": [ "RF p values are recomputed with the AllenSDK chi-square bootstrap (1000 resamplings) with their own seed; they match the code's table (r = 0.9990) where units agree." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig4f-rf-match-retinotopy", "statement": "VISp receptive fields recorded through adjacent holes on successive days lie where the ISI-based retinotopic map predicts.", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 7; Figure 4F", "quote": "the region of visual space preferred on each day matched the expected location indicated by the ISI-based retinotopic map", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "The ISI retinotopic maps and the insertion images needed to predict each insertion's retinotopic location are not in the dandiset.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig4g-visual-responses-similar", "statement": "The population response of VISp units to an image is highly similar across the four recording days.", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 7; Figure 4G", "quote": "In addition, the stimulusevoked temporal dynamics were highly similar across the 4 recording days (Figure 4G).", "statistics": [], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "595, 482, 534 and 446 VISp units with significant RFs on days 1 to 4 (12, 11, 12 and 11 sessions and mice); Kruskal-Wallis p = 0.343 for the per-unit response 0 to 250 ms (a) and 0.70 for the time of peak (b); the day population responses correlate at r = 0.977 to 0.991.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "behavior-stable-across-days", "statement": "Task performance is stable across the four recording days despite repeated insertions (median rewards 59.5, 63.5, 56.5, 60.5; p = 0.96).", "source": "Results, High-quality multi-Neuropixels recordings, paragraph 8", "quote": "we found that mouse behavioral performance on the visual task was stable across the 4 recording days despite repeated probe insertions (median rewards received: day 1, 59.5; day 2, 63.5; day 3, 56.5; day 4, 60.5; Kruskal-Wallis test: p = 0.96)", "statistics": [ { "test": "Kruskal-Wallis test", "p": 0.96 } ], "kind": "finding", "inferred": "", "outcome": "partially_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "No reading finds a difference across days (Kruskal-Wallis p 0.731 to 0.817), so the claim of stability holds. The medians do not all match: over the 87 sessions (a) all rewards 73, 65, 61, 53.5 and (b) earned rewards 67, 59, 55, 48 (days 2 and 3 within tolerance in each); over all sessions (a) 68, 63.5, 60, 56.5 (days 2 to 4) and (b) 62.5, 57.5, 54, 50.5 (days 1 to 3). Medians fall from day 1 to day 4 in every reading. Hit trials exceed earned rewards because of the no-reward block. The paper's medians end in .5 on days 1 and 3, which needs an even number of sessions those days; the 87-session set has 23 on day 1 and the 99-session set 25 on day 3, so neither set is the paper's.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "figs4i-insertion-suppression", "statement": "Activity is transiently suppressed after probe insertion and recovers with a time constant of 25 to 30 minutes.", "source": "STAR Methods, Neuropixels recordings, Probe insertion, paragraph 1; Figure S4I", "quote": "we find that activity is transiently suppressed directly after probe insertion and gradually recovers with a time constant of 25-30 minutes (Figure S4I)", "statistics": [], "kind": "finding", "inferred": "", "outcome": "not_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "Activity recovers over the session (normalised rate 0.826 in minute 1, 0.999 at minute 30, 1.077 at the end, reading a), but the fitted time constant over the common 77.3 min from the first stimulus is 36.0 min (SE 2.3) under (a) and 35.4 min (SE 2.3) under (b), outside 22.5 to 33 min; per-session fits give a median of 34.8 min (IQR 23.1 to 52.5). Shorter windows give longer time constants, 41.2 to 42.2 min over the shortest active block (64 min), 45.9 to 47.1 at 60 min, 65.2 to 84.0 at 40 to 50 min, and no recovery at 30 min (tau unbounded); per-session fits over each session's active block give a median of 39.8 min (IQR 25.7 to 66.8), 26 of 87 sessions inside the range. No window gives a pooled tau inside the range, so the outcome does not rest on the window.", "deviations": [ "The fit (a - b exp(-t / tau) on 1-minute means, least squares) and the 87 sessions (the paper has 85) are not given by the paper.", "The fit window is not given (the S4I legend plots the change over time from experiment start); the main fit uses the longest window all sessions share, 77.3 min from the first stimulus, which runs past the active block. Windows of 30 to 64 min and per-session active blocks are shown; all give tau above 33 min." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "figs4de-retraction-reduces-drift", "statement": "Retracting probes 100 micrometres after insertion reduces unit drift (unit means 24.48 against 30.25 micrometres; session means 24.23 against 30.56).", "source": "Figure S4D and S4E legends; STAR Methods, Probe insertion, paragraph 1", "quote": "Units from sessions where probes were retracted had lower drift (mean = 24.48 \u03bcm) than those from sessions where probes were not retracted (mean = 30.25 \u03bcm; Kolmogrov-Smirnov test, p < .001)", "statistics": [ { "about": "unit drift, retracted against not retracted sessions", "test": "Kolmogorov-Smirnov test", "p": 0.001, "p_relation": "<", "n": "25,896 and 76,851 units" } ], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "metadata_insufficient", "notebook": null, "notes": "No file, and none of the code's tables, records which sessions had probes retracted 100 um; the 23 retracted sessions cannot be identified. The drift and yield measures are in the files.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "figs4f-retraction-yield-similar", "statement": "Retracted and non-retracted sessions record similar numbers of good units (means 1125 against 1280, p = 0.318).", "source": "Figure S4F legend", "quote": "Number of good units recorded per session is similar across retracted and non-retracted sessions. ... Distributions did not differ significantly (Kolmogrov-Smirnov test, p = 0.318).", "statistics": [ { "test": "Kolmogorov-Smirnov test", "p": 0.318 } ], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "metadata_insufficient", "notebook": null, "notes": "No file, and none of the code's tables, records which sessions had probes retracted 100 um; the 23 retracted sessions cannot be identified. The drift and yield measures are in the files.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "figs4gh-sharpened-probes", "statement": "Sharpened probes show drift comparable to unsharpened probes and a lower insertion failure rate (p = 0.0004).", "source": "Figure S4G and S4H legends", "quote": "Sharpened Neuropixels probes show comparable drift to unsharpened probes. ... Sharpened Neuropixels probes have a lower insertion fail rate than unsharpened probes (p = 0.0004, Fisher\u2019s exact test).", "statistics": [ { "about": "insertion failure rate, sharpened against unsharpened", "test": "Fisher's exact test", "p": 0.0004, "n": "32 insertions before and 112 after sharpening" } ], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "metadata_insufficient", "notebook": null, "notes": "No file records which probes were sharpened or the failed insertion attempts (failed probes leave no record, or records of unannotated probes); the code's tables do not either.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "figs3d-duragel-yield-stable", "statement": "Dura-Gel can stay on the implant for at least 2 weeks without visible brain deterioration or a drop in unit yield.", "source": "Results, Dual-hemisphere SHIELD implant, paragraph 1 (end, page 10); Figures S3B and S3D", "quote": "Dura-Gel can remain in place for at least 2 weeks without visible signs of brain deterioration (Figure S3B) or a drop in unit yield (Figure S3D).", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "No session records Dura-Gel use or days since its application, and the 24 Dura-Gel insertions cannot be identified; the paper used Dura-Gel for the optotagging and bilateral experiments, whose mice are not in the dandiset.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig5-bilateral-recordings", "statement": "Simultaneous Neuropixels recordings from both hemispheres are feasible with the dual-hemisphere implant.", "source": "Results, Dual-hemisphere SHIELD implant, paragraph 1; Figures 5C and 5D", "quote": "In preliminary experiments, we demonstrate the feasibility of simultaneous, bilateral Neuropixels recordings with the SHIELD method (Figures 5C and 5D).", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "No dual-hemisphere session is in the dandiset; no annotated unit lies more than 550 um right of the midline (largest z 6250 against midline 5700), and those that do are in midline structures (LSr, ACB, STR, TTd).", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig6bc-cortical-silencing", "statement": "Blue light through the implant in a VGAT-ChR2 mouse silences visually evoked activity of VISp regular-spiking units at the recording site, with activity near baseline 2 mm away, and the effect depends on distance and power.", "source": "Results, SHIELD is compatible with optogenetics, paragraph 2; Figures 6B and 6C", "quote": "Stimulusevoked activity of regular spiking (RS) units in VISp was silenced when a 2-mW laser spot was targeted to the recording location but returned to near baseline at a distance of 2 mm (Figure 6B). The dependence of this silencing effect on distance and laser power (Figure 6C)", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "The VGAT-ChR2 mouse is not in the dandiset; its genotypes are Sst-IRES-Cre;Ai32, Vip-IRES-Cre;Ai32, Ai32 and wild type, and the optotagging tables hold the Sst and Vip ChR2 tagging at 0.95 to 2.05 power levels, not the cortical silencing protocol.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig6ef-optotagging-mrn", "statement": "Red light through the implant evokes reliable short-latency spikes from ChrimsonR-tagged Vglut2 midbrain units, found up to 4 mm below the brain surface.", "source": "Results, SHIELD is compatible with optogenetics, paragraph 3; Figures 6E and 6F", "quote": "Photo-stimulation with pulses (10 ms) of red light (638 nm) evoked reliable, short-latency spiking responses from units recorded in the midbrain (Figure 6E). Such optotagged units were found up to 4 mm below the surface of the brain", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "data_missing", "notebook": null, "notes": "The Vglut2-IRES-Cre mouse with ChrimsonR in MRN is not in the dandiset (see genotypes); no session has 638 nm, five-pulse 20 Hz trains.", "deviations": [], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig7a-mop-higher-frequency", "statement": "VISp alpha events are rhythmic at 3 to 5 Hz, and MOp events occur in a slightly higher band, 5 to 7 Hz.", "source": "Results, SHIELD recordings reveal multi-regional subnetworks, paragraphs 3 and 4; Figure 7A", "quote": "we observed rhythmic 3\u20135 Hz waves of activity in VISp (Figure 7A, left) ... These MOp alpha-like events occurred in a slightly higher frequency band (5\u20137 Hz)", "statistics": [], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "Under frequency reading (b), the autocorrelogram spectrum peak, VISp events at 4.0 Hz and MOp at 4.5 Hz (medians), MOp higher by the one-sided rank-sum test (p 5.3e-7 to 0.0016) in all 8 L, E and set combinations, which the criterion accepts. The 2 s autocorrelogram gives a 0.5 Hz grid on which the MOp median sits on the lower edge of the widened 4.5 to 7.7 Hz range (22.7% of L2 E2 MOp values are exactly 4.5 Hz); with the spectrum zero-padded to 0.049 Hz steps the medians are 4.05 to 4.10 Hz (VISp) and 4.64 to 4.74 Hz (MOp), inside the ranges, with MOp higher at p 7.1e-7 to 0.0038 in all 8. Under reading (a), the inverse of the median interval between maxima, VISp 4.17 to 4.21 Hz and MOp 4.35 to 4.55 Hz, and MOp is not significantly higher (p 0.078 to 0.76). 1469 to 1704 qualifying VISp events and 80 to 122 MOp events. The detector finds synthetic sinusoidal rates of 3 to 5 Hz only (none at 2.5 Hz or below or 5.5 Hz or above), so a 5 to 7 Hz MOp band cannot appear under either reading; this follows from the event rule of the claims file.", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results.", "Reading (b) restricts the spectrum peak to 1 to 20 Hz, a standard choice; the zero-padded spectrum (2048 points) is a standard interpolation shown as a variant." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig7b-low-arousal", "statement": "VISp and MOp alpha events occur when the mouse is immobile and the pupil constricted, with pupil diameter about 25% smaller than at random session times.", "source": "Results, SHIELD recordings reveal multi-regional subnetworks, paragraph 5; Figure 7B", "quote": "VISp and MOp events tended to occur during periods of lower arousal, when the animal was immobile and the pupil was constricted (during alpha events, pupil diameter was ... 25% smaller relative to randomly chosen session times)", "statistics": [], "kind": "finding", "inferred": "", "outcome": "partially_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "At qualifying events the pupil is smaller (Wilcoxon p at most 9.1e-15) and running lower (0.03 to 1.16 against 4.1 to 7.7 cm/s at random times, p at most 8.0e-6) for VISp and for MOp in every combination. The change is not 25%: VISp -12.9 to -15.4% (area, mean), -7.5 to -8.8% (diameter); MOp -30.8 to -36.5% (area), -17.3 to -20.9% (diameter); pooled -14.0 to -16.3% (area). No reading falls within -22.5 to -27.5%. The running and pupil streams share the stimulus clock as far as the data check shows: running slows after hits against misses from a median 0.225 s after the change, and the pupil constricts after white against black flashes from a median 0.35 s, with small contrasts before time 0. Alignment of at least one stream is not shown in 12 of the 99 sessions (2 without a running contrast, as they have no hit or no miss trials, 2 without eye tracking, 5 with a weak running contrast and 3 with a weak pupil contrast).", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results.", "Diameter is the square root of window-mean area; two sessions without eye tracking contribute no pupil values." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig7b-visp-offset-locked", "statement": "VISp alpha events are often triggered by the offset of visual stimuli, while the timing of MOp events is not related to visual stimuli.", "source": "Results, SHIELD recordings reveal multi-regional subnetworks, paragraph 5; Figure 7B, right", "quote": "The VISp events were often triggered by the offset of visual stimulation (Figure 7B) ... The timing of MOp events was not related to visual stimuli (Figure 7B).", "statistics": [], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "In all combinations VISp events peak 100 to 150 ms after stimulus offset (KS against uniform p 9.2e-233 or less; chi-square p printed as 0) and MOp events do not differ from uniform (chi-square p 0.41 to 0.53, KS p 0.28 to 0.62).", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results.", "The stimulus before an event is the last active image presentation, as the code's stimulus-trials table holds active presentations." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "figs6-replay", "statement": "Activity during post-stimulus VISp alpha events is weakly but significantly correlated with activity during the preceding stimulus (r = 0.77, p = 1.8e-6), as if events replay recent stimulation.", "source": "Results, SHIELD recordings reveal multi-regional subnetworks, paragraph 5; Figure S6", "quote": "Activity during post-stimulus alpha events was weakly but significantly correlated with activity during the preceding stimulus, indicating that alpha events tend to replay recent sensory stimulation (Figure S6; r = 0.77", "statistics": [ { "test": "correlation", "statistic": "r = 0.77", "p": 1.8e-06 } ], "kind": "finding", "inferred": "", "outcome": "not_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "Across sessions (7 under L1, as in the paper, 8 under L2; four points each) the correlation of evoked and event responses is r = 0.21 to 0.63 over 32 combinations of the readings and windows, below 0.693 in all; p < 0.05 in 18 of 32. Events are assigned to the last presentation before them, as the criterion says, which takes events that start during an image (473 of 1656 L2 E2 events over all sessions) and after the active block (64); keeping only events 0 to 0.5 s after the offset, the post-stimulus events of the S6 legend, gives r = 0.19 to 0.53, p < 0.05 in 14 of 32, none within tolerance. The per-unit variant gives r = 0.46 to 0.67 (0.52 to 0.66 with post-offset events only).", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results.", "Rows are normalised by their maximum, a standard choice; sessions enter when each of the four images precedes a qualifying event." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig7cd-visp-coupling", "statement": "During VISp events, VISp is tightly coupled to LP and VISl (sharp CCG peaks near zero lag and a 3 to 7 Hz coherence peak), but not to PO or SSp.", "source": "Results, SHIELD recordings reveal multi-regional subnetworks, paragraph 6; Figures 7C and 7D", "quote": "During VISp events, VISp activity was tightly coupled to the secondary visual thalamus (LP) and higher-order lateral visual cortex (VISl), as evidenced by sharp peaks in the CCG at near-zero lag. VISp was not strongly coupled to somatosensory thalamus (PO) or SSp. Indeed, coherence analysis revealed a prominent peak in the 3\u20137 Hz band between VISp population spiking activity and LP and VISl, but not PO or SSp (Figure 7D)", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "methods_underspecified", "notebook": "reproduce_bennett_2024.ipynb", "notes": "The CCG part holds in every combination (LP and VISl peak at 0 to 20 ms, heights 0.335 to 0.508, above PO and SSp at 0.055 to 0.092). The coherence part holds under R1 in all 8 combinations, but under R2 only with L2 E1: with L1 SSp is also significant (+0.008 to +0.011, q 0.0032 and 2.5e-5), with L2 E2 SSp is (+0.007, q 0.048 and 0.036; PO +0.011 at q 0.051). Every part is tested in every combination. The readings disagree, which the claims file's header makes methods underspecified. Open for the authors is how the random-time coherence is formed (the Methods' wording reads as R1, the code does R2).", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig7cd-mop-coupling", "statement": "During MOp events, MOp is coupled to PO and SSp but not to LP or VISl.", "source": "Results, SHIELD recordings reveal multi-regional subnetworks, paragraph 6; Figures 7C and 7D", "quote": "Conversely, MOp showed the opposite pattern: strong coupling to PO and SSp, but not to LP or VISl.", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "methods_underspecified", "notebook": "reproduce_bennett_2024.ipynb", "notes": "The criterion needs MOp-PO to be significantly coherent. Under E1 (8 of 16 combinations) PO is untested (fewer than 25 events), so coherence_part is False there. Under E2 in all 8 combinations MOp is significantly coherent with PO (+0.15 to +0.20) and SSp (+0.19 to +0.25) and not with LP, and the CCGs of PO and SSp peak at 0 to 20 ms (0.20 to 0.33) against 0.03 to 0.06 for LP and VISl. VISl is untested under L1 and under L2 E1, and coherence_part counts an untested VISl as passing, so every part is tested and holds only under L2 E2 (4 combinations; VISl +0.005 and +0.027, q 0.89 and 0.34). The readings disagree, which the claims file's header makes methods underspecified. Changed from reproduced after verification round 3 agreed with the change proposed after round 2.", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results.", "Under E1 the PO part, and under L1 and L2 E1 the VISl part, are untested (fewer than 25 events); all four parts are tested only under L2 E2 (4 of the 16 combinations)." ], "assumed": false, "disagreed": true, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig7e-lags-straddle-zero", "statement": "In the 3 to 7 Hz band, time lags between VISp or MOp and their associated regions straddle zero; neither consistently leads or follows.", "source": "Results, SHIELD recordings reveal multi-regional subnetworks, paragraph 7 (top of page 13); Figure 7E", "quote": "Time lags between VISp and MOp and their associated regions in the 3\u20137 Hz band straddled zero lag, indicating that these areas did not consistently lead or follow activity in the other network nodes (Figure 7E).", "statistics": [], "kind": "finding", "inferred": "", "outcome": "not_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "On the code's test 94 of 102 pair results differ from zero after correction; only VISp to VISl under L1 does not. Median lags are -23 to +14 ms and the fraction of positive lags 0.12 (MOp to PO) to 0.97 (VISp to VISal), over 25 to 1143 events per pair. Whether the claim means that the median lag does not differ from zero (the criterion) or that the distribution of lags spans zero is asked of a reader (questions/001051.yaml).", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig7f-visp-network", "statement": "Beyond visual cortex and thalamus, VISp alpha events enhance coherence of VISp with ACA, RSP and CP.", "source": "Results, SHIELD recordings reveal multi-regional subnetworks, paragraph 8 (page 13; the text cites it as Figure 6F); Figure 7F", "quote": "In addition to visual corticothalamic regions, VISp alpha events enhanced coherence between VISp and medial cortical structures anterior cingulate area (ACA) and RSP, as well as CP", "statistics": [ { "test": "Wilcoxon signed-rank test, event against random coherence", "p": 0.05, "p_relation": "<", "corrected": "Benjamini/Hochberg", "n": "34 CCF regions" } ], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "methods_underspecified", "notebook": "reproduce_bennett_2024.ipynb", "notes": "ACA, RSP, CP, LGd, LP and VISl are each significant and positive in 14 of 16 combinations when any subdivision counts and 8 of 16 when all must (22 of 32 combination and mode rows); under L1 E2 R1 CP is not (q = 0.1), and with all subdivisions required ACAv or RSPagl or RSPd fail under R1. SSp has no subdivisions in these files; RSP has RSPagl, RSPd and RSPv, ACA has ACAd and ACAv, under both L1 and L2. The readings disagree.", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig7f-mop-network", "statement": "MOp alpha events enhance coherence of MOp with SSp, PO, MRN and the deep layers of the superior colliculus (SCiw).", "source": "Results, SHIELD recordings reveal multi-regional subnetworks, paragraph 8 (page 13; the text cites it as Figure 6F); Figure 7F", "quote": "Conversely, MOp events enhanced coherence between MOp and SSp/PO, as well as motor-related midbrain structures MRN and deep layers of the superior colliculus (SCiw).", "statistics": [ { "test": "Wilcoxon signed-rank test, event against random coherence", "p": 0.05, "p_relation": "<", "corrected": "Benjamini/Hochberg", "n": "34 CCF regions" } ], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "methods_underspecified", "notebook": "reproduce_bennett_2024.ipynb", "notes": "Under E2, SSp, PO, MRN and SCiw are significant and positive with L1 (R1 and R2) and L2 R2; with L2 R1 MRN is not (q = 0.11); under E1 PO, MRN and SCiw have too few events. MRN and SCiw each come from one session. The readings disagree.", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig7-hippocampus-not-engaged", "statement": "Neither VISp nor MOp alpha events engage the hippocampus.", "source": "Results, SHIELD recordings reveal multi-regional subnetworks, paragraph 4; Figures 7A and 7F", "quote": "Neither visual nor motor events appeared to engage the hippocampus.", "statistics": [], "kind": "finding", "inferred": "", "outcome": "not_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "For VISp events DG is significantly and positively coherent in every combination (+0.012 to +0.032), with CA1, CA3, ProS, SUB or POST also under R2, and ProS under L1 E2 R1 and SUB under L2 E2 R1; for MOp events CA1 and DG under L2 E1 R2, and CA1 (+0.025, q = 0.044) under L1 E2 R2. 20 of 32 combinations and references have a significant positive hippocampal pair. The hippocampal effects are small next to the visual areas (+0.06 to +0.39).", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig8bc-cp-rf-location", "statement": "CP units with significant spatial receptive fields lie mostly in the medial dorsal striatum, midway along the anterior-posterior extent of CP.", "source": "Results, Fine-scale alpha rhythm coupling between visual cortex and striatum, paragraph 2; Figures 8B and 8C", "quote": "These units were biased toward the medial dorsal striatum mid-way along the anteriorposterior extent of the CP", "statistics": [], "kind": "finding", "inferred": "", "outcome": "partially_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "688 of 2376 QC CP units (52 sessions, 21 mice) have significant RFs. They are more dorsal than CP-noRF units (median y 3150 against 3500 um, one-sided p = 1.9e-58) but more lateral (median z 4075 against 4350 um, two-sided p = 1.8e-33). Their median AP coordinate, 5388 um, is in the posterior part of the CP sampled (3700 to 6500 um), where the RF fraction rises from 0.10 to 0.45 across AP terciles.", "deviations": [ "RF significance recomputed here (link-code-tables); the CCF z axis orientation is checked against VISp (3050) and midline structures (5300)." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig8d-cp-rf-larger-response", "statement": "CP units with receptive fields respond more strongly to the task's full-field images than CP units without receptive fields.", "source": "Results, Fine-scale alpha rhythm coupling between visual cortex and striatum, paragraph 3; Figure 8D", "quote": "CP-RF units showed significantly larger visual responses to this stimulus compared with CPnoRF units (Figure 8D).", "statistics": [ { "test": "Wilcoxon rank-sum test per time point", "p": 0.05, "p_relation": "<", "corrected": "Benjamini-Hochberg", "n": "80 CP-RF and 205 CP-noRF units" } ], "kind": "finding", "inferred": "", "outcome": "partially_reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "In the five sessions CP-RF units respond more than CP-noRF units in 53 bins of 0 to 250 ms (first at 55 ms), CP-RF higher in all, none before onset, with the Methods' QC, the criterion's units. The unit counts there, 70, 162 and 218, miss the expected 80 +- 8 and 205 +- 20.5 for CP-RF and CP-noRF (VISp 218 is within 239 +- 23.9). With the code's filter (presence ratio above 0.9, noise units kept and split by their RF p values) the counts are 79, 206 and 239, within one unit of the paper's, with 78 significant bins, all CP-RF higher. The effect holds under both; the expected counts only under the variant, so the claim holds in part. Whether the criterion should take the code's filter is asked of a reader (questions/001051.yaml).", "deviations": [ "The unit counts match only with the code's filter (presence ratio above 0.9 and no noise filter), which the claims file names as a variant; the effect holds with both. Noise CP units now have RF p values, so the variant splits them by RF as the code does." ], "assumed": false, "disagreed": true, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig8d-response-latencies", "statement": "CP-RF responses lag VISp by about 20 ms, and CP-noRF responses lag CP-RF by about another 20 ms.", "source": "Results, Fine-scale alpha rhythm coupling between visual cortex and striatum, paragraph 3; Figure 8D", "quote": "CP-RF lagged behind VISp responses by ... 20 ms, whereas CP-noRF had slower responses that lagged behind CP-RF units by an additional 20 ms", "statistics": [], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "methods_underspecified", "notebook": "reproduce_bennett_2024.ipynb", "notes": "With latency as the first time the population mean reaches half its peak (b), CP-RF lags VISp by 16 ms and CP-noRF lags CP-RF by 20 ms, within 20 +- 5; with the time of peak (a) the lags are 34 and 9 ms, outside. Both filters (the Methods' QC and the code's) give the same values. The paper says only that CP-RF lagged VISp by about 20 ms and that CP-noRF had slower responses, and gives no latency definition; the claims file lists both readings without preferring either. The readings give different outcomes and nothing in the paper picks one, so the claim is methods underspecified. Half-peak latency had been adopted as an assumption after it was seen to match; changed from reproduced after verification round 3 agreed with the change proposed after round 2. Question for the authors - which latency measure was used.", "deviations": [ "Both latency readings of the claims file were run, half-peak time of the population mean (b) and time of the population peak (a); the paper gives no measure." ], "assumed": false, "disagreed": true, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig8e-cp-rf-more-modulated", "statement": "Triggered on VISp alpha events, CP-RF firing rates are more modulated than CP-noRF rates, particularly in the troughs of the oscillation.", "source": "Results, Fine-scale alpha rhythm coupling between visual cortex and striatum, paragraph 4; Figure 8E", "quote": "We found that CP-RF firing rates were significantly more modulated, particularly during the troughs of the oscillation when the visual thalamo-cortical network is suppressed.", "statistics": [ { "test": "Wilcoxon rank-sum test per time point", "p": 0.05, "p_relation": "<", "corrected": "Benjamini-Hochberg", "n": "439 events" } ], "kind": "finding", "inferred": "", "outcome": "reproduced", "blocked_by": null, "notebook": "reproduce_bennett_2024.ipynb", "notes": "In every combination (387 to 493 events, 4 or 5 sessions) CP-RF and CP-noRF rates differ in 112 to 222 bins by the rank-sum test, 24 or 25 at the troughs of the VISp rate, and the CP-RF range over 0 to 1 s is larger (1.96 to 2.11 against 1.18 to 1.34 Hz; 0.50 to 0.56 against 0.30 to 0.32 normalised). The code's signed-rank variant gives 98 to 216 bins.", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null }, { "claim": "fig8f-cp-rf-higher-coherence", "statement": "During VISp alpha events, CP-RF units are more coherent with VISp in the 3 to 7 Hz band than CP-noRF units (p = 3.8e-14, 439 events).", "source": "Results, Fine-scale alpha rhythm coupling between visual cortex and striatum, paragraph 4; Figure 8F", "quote": "As predicted, CP-RF units displayed higher coherence with VISp in the 3\u20137 Hz band, indicating that alpha events coordinate functionally related cortico-striatal subnetworks.", "statistics": [ { "test": "Wilcoxon signed-rank test", "p": 3.8e-14, "n": "439 events" } ], "kind": "finding", "inferred": "", "outcome": "blocked", "blocked_by": "methods_underspecified", "notebook": "reproduce_bennett_2024.ipynb", "notes": "Under R2 (the code) CP-RF coherence with VISp exceeds CP-noRF (0.096 to 0.110 against 0.042 to 0.052, paired Wilcoxon p 6.5e-11 to 1.2e-14); under R1 it does not (p 0.36 to 0.75). The readings disagree. The code's L2 E2 readings give 439 events, the paper's count, and p = 1.9e-10 under R2.", "deviations": [ "Run under the claims file's 16 combinations of area membership (L1, L2), event rule (E1, E2), random-time coherence (R1, R2) and session set (a, b); population windows are cut from one 10 ms grid per session. Events are detected on the smoothed rate, and the autocorrelogram threshold, CCGs and coherence use the unsmoothed rate, a detail taken from the code's printed counts quoted in the claims file's header (1653 VISp and 113 MOp qualifying events, code cell 12), which the unsmoothed autocorrelogram matches (1643 to 1704 and 119 to 122 under E2) and the smoothed one does not (3035 to 3426 and 152 to 170). The Benjamini-Hochberg family of the pair test holds brain areas only (not CP-RF, CP-noRF, VIS or unannotated labels); this changes significance for 4 of 734 pair results." ], "assumed": false, "disagreed": false, "data": "dataset", "where": null, "proposed": null } ], "findings": [ { "kind": "metadata_error", "statement": "In 6 of the 87 sessions the paper analysed (1176989662, 1178460518, 1178693650, 1179911454, 1180266229, 1181096406) the units under each unit id differ from the paper's units table at the cited code commit (max_drift and firing_rate differ for 55 to 99.8% of units, quality for 14 to 38%, and receptive-field p values recomputed from the files are uncorrelated with the table's; the files' spike counts fit the files' firing rates but not the table's). The dandiset seems to hold a different spike sorting or unit assignment for these sessions than the one the paper used.", "found": null, "status": null }, { "kind": "data_missing", "statement": "In 5 of those sessions (1176989662, 1178693650, 1179911454, 1180266229, 1181096406) the paper's units table has 328 units, 133 of them in CP, that the session files do not hold. Each session's are on one probe of the table (probe_id 88, 46, 121, 133 and 139, with 38 to 114 of these units) whose probe_id is not among the file's and on which the file has no unit; none of their peak channels is a peak channel of a file's unit.", "found": null, "status": null }, { "kind": "selection_undocumented", "statement": "The 87 sessions passing ephys QC are listed only in the paper's code; the files and the dandiset metadata do not mark them, and the 12 other sessions are not explained. In the 87 sessions the files also hold 40 probe recordings (2004 units, 1873 on unannotated channels) that the paper's tables leave out, and nothing marks them.", "found": null, "status": null }, { "kind": "metadata_insufficient", "statement": "No file records the implant hole or probe orientation of an insertion, which probes were sharpened, which sessions had probes retracted 100 um, Dura-Gel use, or failed insertion attempts, so Figures 4B (trajectories), S4B, S4D to S4H and S3D cannot be tested.", "found": null, "status": null }, { "kind": "data_missing", "statement": "The dandiset holds none of the Figure 3 material (surgery records, ISI images, histology, MRI, training histories), the dual-hemisphere sessions of Figure 5, or the VGAT-ChR2 and Vglut2-Cre mice of Figure 6; its 27 mice are all single-hemisphere change-detection mice.", "found": null, "status": null }, { "kind": "data_missing", "statement": "Sessions 1187475832 and 1202438738 have no eye tracking. In the others the running and pupil streams follow the stimulus and trial clock (running slows after hits from a median 0.225 s after the change, the pupil constricts after white flashes from a median 0.35 s), but in 5 sessions for running and 3 for the pupil the contrasts are too weak to show it, and 2 sessions (1173189336, 1230732154) have no running contrast, as they have no hit or no miss trials; in all, alignment of at least one stream is not shown in 12 of the 99 sessions.", "found": null, "status": null }, { "kind": "paper_inconsistency", "statement": "The Methods describe 8 natural images in the change detection task; every session holds four image stimuli (three natural images and im115_r at 0.7 contrast) besides omitted flashes.", "found": null, "status": null }, { "kind": "methods_unavailable", "statement": "For the alpha-event analyses (Figures 7, 8E, 8F and S6), please confirm (1) how the random-time coherence was formed, both areas at the random time (as the Methods read) or the reference area at the event against the other area at the random time (as the code does); (2) whether area membership used the ISI labels and exact CCF labels or the code's substring match (which puts POST units into PO and VISpm into VISp); (3) the event rule, four maxima (Methods) or three in a window with more than three (code). The outcomes of fig7cd-visp-coupling, fig7f-visp-network, fig7f-mop-network and fig8f-cp-rf-higher-coherence depend on these choices; the code's choices reproduce the 439 events of Figure 8F.", "found": null, "status": null }, { "kind": "methods_unavailable", "statement": "Please state the latency measure of Figure 8D (half-peak time and time of peak give different lags, so the claim is blocked here), the fit, its window and the normalisation behind the 25 to 30 min recovery time constant of Figure S4I (fits here give 35 to 36 min over 77 min and longer time constants over shorter windows), and the windows and row normalisation of Figure S6F.", "found": null, "status": null } ], "reconciliation": null, "verifications": [ { "date": "2026-11-01", "kind": "agent", "by": "Independent agent run by CatalystNeuro (same harness, separate context, did not write the notebook)", "reviewed": "Round 1. The first executed notebook with the conclusion cells and the summary removed, the helper module and the claims file, with the paper. The verifier queried the archive metadata for the blocked claims and tested find_events and rf_pvalues on synthetic inputs. Its report had no attempt record and gave its own outcome for each claim; 43 of 46 agree with the record as it then stood.", "agreed": 43, "of": 46, "disagreements": [ { "claim": "fig7cd-mop-coupling", "outcome": "blocked", "reason": "Holds in all 8 E2 combinations, coherence and CCG. Under E1, PO has fewer than 25 events and is not tested (methods_underspecified). An area that is absent (fewer than 25 events) counts as not significant in the negative parts; MOp-VISl under L1 is never tested." }, { "claim": "fig8d-cp-rf-larger-response", "outcome": "partially_reproduced", "reason": "The response test holds: 53 significant bins, all with CP-RF higher. Under the Methods QC filter the counts are 70 and 162, outside 80 +- 8 and 205 +- 20.5; only the code's presence 0.9 variant gives 76, 209 and 239. Noise CP units have no RF p value, because rf_pvalues covers only good units, so they fall into CP-noRF in the variant." }, { "claim": "fig8d-response-latencies", "outcome": "blocked", "reason": "Half-peak reading: lags 16 and 20 ms, within tolerance. Peak reading: 34 and 9 ms (QC) or 43 and 0 ms (variant), outside it (methods_underspecified)." } ] }, { "date": "2026-11-01", "kind": "agent", "by": "Independent agent run by CatalystNeuro (same harness, separate context, did not write the notebook)", "reviewed": "Round 2. The full executed notebook with the conclusion cells and the summary, the helper module, the README, the attempt record and the claims file, with the paper. The verifier agreed with 44 of the 46 outcomes and would block fig7cd-mop-coupling and fig8d-response-latencies as methods_underspecified.", "agreed": 44, "of": 46, "disagreements": [ { "claim": "fig7cd-mop-coupling", "outcome": "blocked", "reason": "Would block as methods_underspecified. The criterion's 'reproduced' condition needs MOp-PO to be significant. Cell 78 prints coherence_part False in all 8 E1 combinations, because PO is absent there; VISl is also untested in all L1 combinations and under L2 E1. All four parts are tested only under L2 E2, 4 of the 16 combinations. The claims file's rule is that every alpha claim is judged under each combination of readings, and the claim is methods_underspecified when they disagree; here the E1 readings do not reproduce the claim, so they disagree with E2. 'No tested part contradicts' is true, but the criterion does not use that test." }, { "claim": "fig8d-response-latencies", "outcome": "blocked", "reason": "Would block as methods_underspecified. The half-peak assumption is stated, but it is weakly grounded. The paper says only that the 'response time course ... lagged' and that CP-noRF had 'slower responses', which does not pick an onset measure over a peak measure. The claims file offers both readings without preferring either. The half-peak reading was adopted after it turned out to be the one that matches: peak lags are 34 and 9 ms, half-peak lags 16 and 20 ms (cell 99)." } ] }, { "date": "2026-11-01", "kind": "agent", "by": "Independent agent run by CatalystNeuro (same harness, separate context, did not write the notebook)", "reviewed": "Round 3. The two changes of outcome proposed after round 2 (fig7cd-mop-coupling and fig8d-response-latencies) and the analysis behind them in the re-executed notebook, with the claims file and the paper. The verifier agreed with both.", "agreed": 2, "of": 2, "disagreements": [] } ] } ], "entries": [ { "name": "001051-bennett-2024", "type": "reproduction", "title": "SHIELD multi-Neuropixels recordings and alpha events (Bennett et al. 2024)", "description": "Tests the findings of Bennett, Ouellette, Ramirez et al. (2024) on dandiset 001051, which holds 99 change-detection sessions of 27 mice recorded with up to six Neuropixels probes through a SHIELD implant. It checks the paper's session list and unit table against the files, the unit yield, quality and stability across recording days, receptive fields and visual responses, and the alpha events of VISp and MOp with their coherence networks and the striatal subnetworks. The surgery, imaging, histology, MRI and optogenetics material of Figures 3, 5 and 6 is not in the dandiset.", "datasets": [ "dandi:001051" ], "papers": [ "10.1016/j.neuron.2024.06.015" ], "derived_from": [], "notebooks": [ { "path": "reproduce_bennett_2024.ipynb", "title": "Reproducing Bennett et al. (2024) from dandiset 001051", "executed": "notebooks/001051-bennett-2024/reproduce_bennett_2024.html" } ] } ] }