EM sections and synapse counts of the MICrONS volume (MICrONS Consortium 2025)¶

Paper. The MICrONS Consortium (2025). Functional connectomics spanning multiple areas of mouse visual cortex. Nature 640, 435-447. doi:10.1038/s41586-025-08790-w. The claims file, claims/000402/10.1038-s41586-025-08790-w.yaml, was written from the open access version of record (PMC11981939).

Data. The two claims here are data: elsewhere: the EM imagery and the synapse tables of the MICrONS volume, which the BossDB open data bucket serves without an account (s3://bossdb-open-data/iarpa_microns/minnie/). The imagery of each subvolume is in neuroglancer precomputed format (minnie35/em, minnie65/em); the helper reads its sharded chunks directly. The synapse tables are gzipped CSV files, streamed once and counted without being kept. Dandiset 000402 (version 0.230307.2132), the entry's dataset, holds no EM data; the helper names it because this notebook belongs to the entry on it. The other analyses of the entry are in reproduce_microns_2025.ipynb.

This notebook was added in the second revision, after the verifier's second round. Both claims had been verified as blocked (too_large); a third verification round agreed with the outcomes found here, which are now recorded.

Random numbers. None are used.

Claim Data Tested here
em-sections elsewhere yes, from the imagery's metadata and its sections with data
synapse-detection elsewhere the two counts; precision, recall and F1 are not tested (no manual annotations found)

Data check¶

In [1]:
import time
import zlib

import numpy as np
import pandas as pd

import microns

pd.set_option("display.width", 200)
pd.set_option("display.max_colwidth", 120)
# The files on the BossDB bucket that this notebook and the attempt record cite, with their sizes
listing = pd.concat([microns.boss_list(p) for p in (
    "minnie65/synapse_graph/", "minnie35/synapse_graph/", "functional_data/two_photon_structural_stacks/",
    "functional_coregistration/", "proofreading_status/", "minnie65/em/info", "minnie35/em/info")])
listing = listing[~listing.key.str.endswith("/")]
listing["MB"] = (listing.bytes / 1e6).round(2)
print(listing[["key", "MB"]].to_string(index=False))
# EM imagery: every scale of each subvolume
for vol in ("minnie35", "minnie65"):
    info = microns._session.get(f"{microns.BOSS_ROOT}{vol}/em/info", timeout=60).json()
    print(f"\n{vol}/em: {info['data_type']}, {info['num_channels']} channel; scales (nm per voxel, offset, size, chunk, encoding):")
    for s in info["scales"]:
        print(f"  {s['key']:>15}  offset {s['voxel_offset']}  size {s['size']}  chunk {s['chunk_sizes'][0]}  {s['encoding']}")

                                                                                                                          key       MB
                        iarpa_microns/minnie/minnie65/synapse_graph/cave_exports/v1078/synapses_pni_2_v1_filtered_view.csv.gz 20045.32
                    iarpa_microns/minnie/minnie65/synapse_graph/cave_exports/v1078/synapses_pni_2_v1_filtered_view_header.csv     0.00
                        iarpa_microns/minnie/minnie65/synapse_graph/cave_exports/v1181/synapses_pni_2_v1_filtered_view.csv.gz 20045.49
                    iarpa_microns/minnie/minnie65/synapse_graph/cave_exports/v1181/synapses_pni_2_v1_filtered_view_header.csv     0.00
                                iarpa_microns/minnie/minnie65/synapse_graph/cave_exports/v1300/connections_with_nuclei.csv.gz  2996.97
                            iarpa_microns/minnie/minnie65/synapse_graph/cave_exports/v1300/connections_with_nuclei_header.csv     0.00
                        iarpa_microns/minnie/minnie65/synapse_graph/cave_exports/v1300/synapses_pni_2_v1_filtered_view.csv.gz 20045.98
                    iarpa_microns/minnie/minnie65/synapse_graph/cave_exports/v1300/synapses_pni_2_v1_filtered_view_header.csv     0.00
                               iarpa_microns/minnie/minnie65/synapse_graph/cave_exports/v661/mat661_soma_soma_connections.csv   560.57
                                            iarpa_microns/minnie/minnie65/synapse_graph/cave_exports/v661/mat661_synapses.csv 24535.17
                                                               iarpa_microns/minnie/minnie65/synapse_graph/synapses_pni_2.csv 51020.16
                                                                  iarpa_microns/minnie/minnie35/synapse_graph/assigned.csv.gz 13806.24
                iarpa_microns/minnie/functional_data/two_photon_structural_stacks/two_photon_stack_17797_9_19_resized_v1.tiff 10005.72
                        iarpa_microns/minnie/functional_data/two_photon_structural_stacks/two_photon_stack_17797_9_19_v1.tiff  1250.79
iarpa_microns/minnie/functional_data/two_photon_structural_stacks/two_photon_structural_stacks_technical_documentation_v1.pdf     0.10
                                                iarpa_microns/minnie/functional_coregistration/func_unit_em_match_release.csv     0.01
                                              iarpa_microns/minnie/proofreading_status/proofreading_status_public_release.csv     0.06
                                                                                        iarpa_microns/minnie/minnie65/em/info     0.01
                                                                                        iarpa_microns/minnie/minnie35/em/info     0.01

minnie35/em: uint8, 1 channel; scales (nm per voxel, offset, size, chunk, encoding):
           8_8_40  offset [13824, 13824, 7904]  size [212992, 180224, 6944]  chunk [128, 128, 16]  raw
         16_16_40  offset [6912, 6912, 7904]  size [106496, 90112, 6944]  chunk [64, 64, 64]  jpeg
         32_32_40  offset [3456, 3456, 7904]  size [53248, 45056, 6944]  chunk [64, 64, 64]  jpeg
         64_64_40  offset [1728, 1728, 7904]  size [26624, 22528, 6944]  chunk [64, 64, 64]  jpeg
       128_128_80  offset [864, 864, 3952]  size [13312, 11264, 3472]  chunk [64, 64, 64]  jpeg
      256_256_160  offset [432, 432, 1976]  size [6656, 5632, 1736]  chunk [64, 64, 64]  jpeg
      512_512_320  offset [216, 216, 988]  size [3328, 2816, 868]  chunk [64, 64, 64]  jpeg
    1024_1024_640  offset [108, 108, 494]  size [1664, 1408, 434]  chunk [64, 64, 64]  jpeg
   2048_2048_1280  offset [54, 54, 247]  size [832, 704, 217]  chunk [64, 64, 64]  jpeg

minnie65/em: uint8, 1 channel; scales (nm per voxel, offset, size, chunk, encoding):
           8_8_40  offset [13824, 13824, 14816]  size [212992, 180224, 13088]  chunk [128, 128, 16]  raw
         16_16_40  offset [6912, 6912, 14816]  size [106496, 90112, 13088]  chunk [64, 64, 64]  jpeg
         32_32_40  offset [3456, 3456, 14816]  size [53248, 45056, 13088]  chunk [64, 64, 64]  jpeg
         64_64_40  offset [1728, 1728, 14816]  size [26624, 22528, 13088]  chunk [64, 64, 64]  jpeg
       128_128_80  offset [864, 864, 7408]  size [13312, 11264, 6544]  chunk [64, 64, 64]  jpeg
      256_256_160  offset [432, 432, 3704]  size [6656, 5632, 3272]  chunk [64, 64, 64]  jpeg
      512_512_320  offset [216, 216, 1852]  size [3328, 2816, 1636]  chunk [64, 64, 64]  jpeg
    1024_1024_640  offset [108, 108, 926]  size [1664, 1408, 818]  chunk [64, 64, 64]  jpeg
   2048_2048_1280  offset [54, 54, 463]  size [832, 704, 409]  chunk [64, 64, 64]  jpeg

Each subvolume's imagery has 9 scales. The z axis counts 40 nm sections at the finest four scales (8 to 64 nm per pixel in x and y) and merges sections at the coarser ones, so 64 nm per pixel (64_64_40) is the coarsest scale that keeps every section. The files cited in the attempt record are listed with their sizes: the synapse table exports of subvolume 65 (versions 1078, 1181 and 1300, about 20 GB each), the unversioned table synapses_pni_2.csv (51 GB), the connection table of version 1300, the synapse table of subvolume 35 (assigned.csv.gz, 13.8 GB), the structural stacks, the 2021 match release and the 2021 proofreading release.

em-sections¶

The block was cut into 27,972 sections of 40 nm, 26,652 were imaged by five autoTEMs, and an 800 um region of sections 7,931 to 27,904 was processed, split into two subvolumes of about 35% and 65% of the sections.

Criterion: the released imagery and its metadata give the sections of each subvolume (7,931 to 14,815 and 14,816 to 27,904). The processed region is 27,904 - 7,931 + 1 sections (expected 19,974, tolerance 0), and the share of subvolume 35 is 35 +- 3.5%. Counts of sections cut and imaged are reported if the released metadata give them.

The metadata give each subvolume's bounds as whole chunks of 64 sections at the 64 nm scale (voxel_offset and size above). The first and last section that hold data are read from the imagery: at the 64 nm scale, the chunk layers at each end of each subvolume are sampled on a grid of every 8th chunk in x and y (52 x 44 = 2,288 chunks a layer), and a section has data when any sampled chunk has a non-zero voxel in it. The grid is a choice of this attempt; at the last section of subvolume 65 it is checked against a sparser grid and against the 16 nm scale.

In [2]:
started = time.time()
sc35, sc65 = microns.em_scale("minnie35"), microns.em_scale("minnie65")
b = {}
for vol, s in (("minnie35", sc35), ("minnie65", sc65)):
    z0, nz = s["voxel_offset"][2], s["size"][2]
    b[vol] = (z0, z0 + nz - 1)
    print(f"{vol} metadata: sections {z0} to {z0 + nz - 1} ({nz})")
lo, hi = b["minnie35"][0], b["minnie65"][1]
print(f"Metadata, union: {lo} to {hi}, {hi - lo + 1} sections; overlap of the two subvolumes: "
      f"{b['minnie35'][1] - b['minnie65'][0] + 1} sections ({b['minnie65'][0]} to {b['minnie35'][1]})")
print(f"Metadata, share of subvolume 35 with the split at the start of subvolume 65: "
      f"{100 * (b['minnie65'][0] - lo) / (hi - lo + 1):.2f}%")

# Sections with data at both ends of each subvolume
layers = {"minnie35": (0, (14752 - sc35["voxel_offset"][2]) // 64, (14816 - sc35["voxel_offset"][2]) // 64),
          "minnie65": (0, (27808 - sc65["voxel_offset"][2]) // 64, (27872 - sc65["voxel_offset"][2]) // 64)}
has = {}
for vol, gzs in layers.items():
    for gz in gzs:
        z0, n, counts = microns.em_sections_with_data(vol, gz, step=8)
        for i, c in enumerate(counts):
            has[(vol, z0 + i)] = int(c)
        print(f"{vol}, sections {z0}-{z0 + len(counts) - 1}, {n} chunks sampled; sampled chunks with data per section:")
        print("  ", counts.tolist())
first = {v: min(z for (vv, z), c in has.items() if vv == v and c > 0) for v in layers}
last = {v: max(z for (vv, z), c in has.items() if vv == v and c > 0) for v in layers}
for v in layers:
    print(f"{v}: first section with data {first[v]}, last {last[v]} (the last sampled section is {max(z for (vv, z) in has if vv == v)})")
print(f"Elapsed: {time.time() - started:.0f} s")

minnie35 metadata: sections 7904 to 14847 (6944)
minnie65 metadata: sections 14816 to 27903 (13088)
Metadata, union: 7904 to 27903, 20000 sections; overlap of the two subvolumes: 32 sections (14816 to 14847)
Metadata, share of subvolume 35 with the split at the start of subvolume 65: 34.56%

minnie35, sections 7904-7967, 2288 chunks sampled; sampled chunks with data per section:
   [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 737, 865, 909, 902, 893, 947, 921, 943, 893, 961, 894, 942, 1009, 905, 918, 932, 887, 894, 905, 907, 878, 890, 892, 912, 893, 889, 892, 913, 905, 906, 900, 886, 899, 900, 884, 893, 907]

minnie35, sections 14752-14815, 2288 chunks sampled; sampled chunks with data per section:
   [940, 939, 915, 911, 947, 940, 935, 925, 944, 938, 933, 942, 929, 918, 927, 948, 932, 920, 945, 931, 944, 929, 930, 935, 966, 920, 917, 928, 970, 901, 923, 1417, 1417, 1407, 1393, 1393, 1375, 1369, 1341, 1341, 1338, 1332, 1286, 1244, 1239, 1239, 1239, 1239, 1230, 1228, 1228, 1228, 1226, 1126, 1126, 1126, 1120, 1120, 1117, 1113, 1086, 1086, 1086, 1086]

minnie35, sections 14816-14847, 2288 chunks sampled; sampled chunks with data per section:
   [1083, 1079, 1009, 1004, 993, 985, 961, 973, 970, 968, 974, 962, 963, 963, 978, 969, 965, 961, 983, 965, 965, 973, 965, 970, 965, 957, 882, 881, 891, 882, 880, 893]

minnie65, sections 14816-14879, 2288 chunks sampled; sampled chunks with data per section:
   [0, 0, 0, 0, 0, 0, 0, 0, 0, 968, 973, 960, 963, 963, 978, 969, 965, 961, 983, 965, 965, 973, 965, 970, 965, 957, 882, 881, 891, 882, 880, 893, 874, 890, 878, 897, 875, 894, 884, 888, 880, 879, 887, 877, 884, 895, 874, 886, 898, 881, 889, 886, 880, 891, 884, 877, 875, 883, 873, 882, 894, 875, 874, 882]

minnie65, sections 27808-27871, 2288 chunks sampled; sampled chunks with data per section:
   [608, 586, 593, 574, 586, 586, 577, 585, 575, 587, 594, 599, 580, 583, 583, 588, 590, 577, 592, 583, 583, 592, 588, 584, 585, 577, 576, 583, 577, 576, 580, 583, 580, 584, 583, 580, 572, 578, 580, 577, 583, 585, 577, 577, 567, 584, 582, 580, 589, 592, 576, 581, 581, 575, 569, 585, 590, 582, 590, 580, 591, 582, 583, 589]

minnie65, sections 27872-27903, 2288 chunks sampled; sampled chunks with data per section:
   [573, 588, 588, 596, 572, 581, 583, 588, 596, 583, 581, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
minnie35: first section with data 7931, last 14847 (the last sampled section is 14847)
minnie65: first section with data 14825, last 27882 (the last sampled section is 27903)
Elapsed: 138 s
In [3]:
# Checks of the last section of subvolume 65: a sparser grid at 64 nm, and 16 chunks at 16 nm
z0, n, counts = microns.em_sections_with_data("minnie65", layers["minnie65"][2], step=16)
print(f"64 nm, every 16th chunk ({n} chunks): last section with data {z0 + int(np.where(counts > 0)[0].max())}")
s16 = microns.em_scale("minnie65", "16_16_40")
grid = [-(-s // c) for s, c in zip(s16["size"], s16["chunk_sizes"][0])]
gz = (27872 - s16["voxel_offset"][2]) // 64
lasts = []
for x in range(grid[0] // 4, 3 * grid[0] // 4, grid[0] // 8):
    for y in range(grid[1] // 4, 3 * grid[1] // 4, grid[1] // 8):
        a = microns.em_chunk("minnie65", s16, (x, y, gz))
        if a is not None and (a > 0).any():
            lasts.append(s16["voxel_offset"][2] + gz * 64 + int(np.where((a.reshape(a.shape[0], -1) > 0).any(axis=1))[0].max()))
print(f"16 nm, 16 chunks in the centre of the volume: {len(lasts)} with data; last section with data in each: {sorted(set(lasts))}")

64 nm, every 16th chunk (572 chunks): last section with data 27882

16 nm, 16 chunks in the centre of the volume: 8 with data; last section with data in each: [27882]
In [4]:
start, end = first["minnie35"], last["minnie65"]
n_data = end - start + 1
print(f"Sections with data: subvolume 35 {first['minnie35']} to {last['minnie35']}, subvolume 65 {first['minnie65']} to {last['minnie65']}")
print(f"Processed region from the imagery: {start} to {end}, {n_data} sections (expected 19,974, from 7,931 to 27,904)")
print(f"Processed region from the metadata bounds: {lo} to {hi}, {hi - lo + 1} sections")
rows = [("imagery, subvolume 35 up to the first section of subvolume 65", first["minnie65"] - start, n_data),
        ("imagery, all sections of subvolume 35 with data", last["minnie35"] - first["minnie35"] + 1, n_data),
        ("metadata bounds, split at the start of subvolume 65", b["minnie65"][0] - lo, hi - lo + 1),
        ("claims file split (7,931 to 14,815 of 7,931 to 27,904)", 14815 - 7931 + 1, 27904 - 7931 + 1)]
for name, k, n in rows:
    print(f"  share of subvolume 35, {name}: {k} of {n} = {100 * k / n:.2f}% (expected 31.5 to 38.5)")
print("Sections cut (27,972) and imaged (26,652): not in the metadata above.")

Sections with data: subvolume 35 7931 to 14847, subvolume 65 14825 to 27882
Processed region from the imagery: 7931 to 27882, 19952 sections (expected 19,974, from 7,931 to 27,904)
Processed region from the metadata bounds: 7904 to 27903, 20000 sections
  share of subvolume 35, imagery, subvolume 35 up to the first section of subvolume 65: 6894 of 19952 = 34.55% (expected 31.5 to 38.5)
  share of subvolume 35, imagery, all sections of subvolume 35 with data: 6917 of 19952 = 34.67% (expected 31.5 to 38.5)
  share of subvolume 35, metadata bounds, split at the start of subvolume 65: 6912 of 20000 = 34.56% (expected 31.5 to 38.5)
  share of subvolume 35, claims file split (7,931 to 14,815 of 7,931 to 27,904): 6885 of 19974 = 34.47% (expected 31.5 to 38.5)
Sections cut (27,972) and imaged (26,652): not in the metadata above.

Result: partially reproduced. The imagery of subvolume 35 starts at section 7,931, as the paper says. The imagery of subvolume 65 ends at section 27,882, not 27,904, in every sampled chunk with data, at 64 nm and at 16 nm, so the region with data spans 7,931 to 27,882, 19,952 sections against the expected 19,974 (tolerance 0). The metadata bounds, whole chunks of 64 sections, give 7,904 to 27,903, 20,000 sections, which also differs. The share of subvolume 35 is 34.55% to 34.67% by the imagery and 34.56% by the metadata bounds, within 35 +- 3.5%. The two subvolumes overlap: the imagery of subvolume 35 runs to 14,847 and that of subvolume 65 starts at 14,825, while the claims file splits them at 14,815 and 14,816. The metadata do not give the sections cut or imaged. The 8 nm scale was not checked.

synapse-detection¶

Automated detection found 524 million synaptic clefts (186 million in subvolume 35, 337 million in subvolume 65), with 96% precision and 89% recall against 8,611 manually annotated synapses in 70 subvolumes, and 98% partner assignment accuracy on 191 held-out synapses.

Criterion: counts from the released synapse tables (expected 337 +- 33.7 million in subvolume 65 and 186 +- 18.6 million in subvolume 35). Precision and recall can be tested only if the manual annotations of the 70 test subvolumes are public.

The paper names no materialization version. Subvolume 65 is counted on the export of version 1181, the version the paper's code reads (claims file header, "its code uses version 1181"); the file has no header line, so each line is a synapse. Subvolume 35 has one table on the bucket, minnie35/synapse_graph/assigned.csv.gz (2021), whose first line is a header. Each file is streamed once; a broken stream resumes at the byte reached.

In [5]:
started = time.time()
syn = {}
for name, path, header in (("subvolume 65, version 1181", "minnie65/synapse_graph/cave_exports/v1181/synapses_pni_2_v1_filtered_view.csv.gz", 0),
                           ("subvolume 35", "minnie35/synapse_graph/assigned.csv.gz", 1)):
    print(name, path)
    head = microns._range(microns.BOSS_ROOT + path, 0, 4096)
    print("  first line:", zlib.decompressobj(16 + zlib.MAX_WBITS).decompress(head).split(b"\n")[0][:200].decode())
    lines, got = microns.count_csv_gz_lines(microns.BOSS_ROOT + path, report_gb=5)
    syn[name] = lines - header
    print(f"  {got / 1e9:.2f} GB read, {lines} lines, {syn[name]} synapses")
s65, s35 = syn["subvolume 65, version 1181"], syn["subvolume 35"]
print(f"\nSubvolume 65: {s65 / 1e6:.1f} million (expected 337 +- 33.7): {abs(s65 / 1e6 - 337) <= 33.7}")
print(f"Subvolume 35: {s35 / 1e6:.1f} million (expected 186 +- 18.6): {abs(s35 / 1e6 - 186) <= 18.6}")
print(f"Both: {(s65 + s35) / 1e6:.1f} million (the paper's total, 524 million)")
print("Manual annotations of the 70 test subvolumes: none among the files listed above.")
print(f"Elapsed: {time.time() - started:.0f} s")

subvolume 65, version 1181 minnie65/synapse_graph/cave_exports/v1181/synapses_pni_2_v1_filtered_view.csv.gz
  first line: 106675816,142854,244590,18572,142890,244657,18573,142946,244642,18573,508,84396795031127211,864691136041487043,84467163775312026,864691135568681196

  5 of 20.0 GB, 84 million lines

  10 of 20.0 GB, 168 million lines

  15 of 20.0 GB, 253 million lines

  20 of 20.0 GB, 337 million lines

  20.05 GB read, 337312429 lines, 337312429 synapses
subvolume 35 minnie35/synapse_graph/assigned.csv.gz

  first line: cleft_segid,centroid_x,centroid_y,centroid_z,bbox_bx,bbox_by,bbox_bz,bbox_ex,bbox_ey,bbox_ez,presyn_segid,postsyn_segid,presyn_x,presyn_y,presyn_z,postsyn_x,postsyn_y,postsyn_z,size

  5 of 13.8 GB, 68 million lines

  10 of 13.8 GB, 135 million lines

  13.81 GB read, 186268896 lines, 186268895 synapses

Subvolume 65: 337.3 million (expected 337 +- 33.7): True
Subvolume 35: 186.3 million (expected 186 +- 18.6): True
Both: 523.6 million (the paper's total, 524 million)
Manual annotations of the 70 test subvolumes: none among the files listed above.
Elapsed: 760 s

Result: partially reproduced. The version 1181 export of subvolume 65 holds 337.3 million synapses and the table of subvolume 35 holds 186.3 million, both within tolerance of the paper's 337 and 186 million, and together 523.6 million against 524 million. Precision, recall and F1 are not tested: no manual synapse annotations were found on the bucket. The 98% partner assignment accuracy in the statement has no expected quantity and is not tested either.

Summary¶

Claim Data Outcome
em-sections elsewhere partially reproduced (share of subvolume 35 within tolerance; 19,952 sections with data against 19,974)
synapse-detection elsewhere partially reproduced (both counts within tolerance; precision, recall and partner accuracy not tested)

A failure to reproduce here may be a failure of this attempt, not of the paper: see the reasoning and the differences from the paper's method.

How the analysis differs from the paper's. The first and last sections are read from the imagery at 64 nm per pixel on a grid of sampled chunks, not from a list of sections, which the release does not give. The paper names no version of the synapse table; subvolume 65 is counted at version 1181 and subvolume 35 on its only public table.

What was found about the data. The imagery of subvolume 65 ends at section 27,882, 22 sections before the paper's 27,904, and the imagery of the two subvolumes overlaps from 14,825 to 14,847. The metadata bounds are whole chunks and do not give the processed sections.