DANDI:000223

we conducted 19 paired spine Ca 2+ imaging and HD-MEA recording experiments in 10 different cell cultures ... based on 19 recordings with a mean of 102 ± 43 units per recording ... (in one of recordings two spines were chosen)

Results, 3.4 Performance evaluation on experimental data, paragraphs 1 and 2; Table 1

The 20 files hold 19 distinct recordings (two files are the same blob) from 10 subjects, with 20 spine and shaft ROI pairs, two of them in one recording. Units per recording are 101.7 with a population SD of 42.6 (sample SD 43.7). Every per-row unit count is within 10% of Table 1; rows 16 and 18 have 59 and 101 units against 58 and 100, and in each of these files one unit has no spikes, so the counts of units with spikes equal Table 1's. Each subject id is taken to be one culture, which the unit counts per subject support.

Moreover, the R value for the best-matching unit ( best-R ) shows the largest increase with respect to the next (ordered) R value among all units

Results, 3.1 Identifying the presynaptic unit of a dendritic target spine, last paragraph; Figure 1(e)

With the carried-forward reading, the gap between best-R and the second R is the largest of the consecutive gaps for 7 of 20 spines (rows 1, 2, 3, 4, 6, 13 and 14), so the unnamed example can be one of them. For the other 13 spines it is not (rank 2 to 45). A weak test, as the claims file says.

we found that putative connections were rejected in three different scenarios ... The best-R was smaller than either one of the two thresholds or both thresholds

Results, 3.4 Performance evaluation on experimental data, paragraph 2; Figure 5(a)

All three scenarios occur among the rejected spines under every reading and region. With Table 1's quantiles and region 0, 5 spines fall below the test threshold only, 4 below the GT threshold only and 4 below both (Table 1, 4, 1 and 6); the other readings and regions give 5 to 9, 2 to 4 and 3 to 4.

Note that for some recordings the test threshold was larger than the GT threshold and vice versa for other recordings

Results, 3.4 Performance evaluation on experimental data, paragraph 2; Figure 5(a)

With Table 1's quantiles and region 0, the test threshold is above the GT threshold for 14 spines and below it for 6 (Table 1, 15 and 5); the other readings and regions give 15 to 18 and 2 to 5.

We found that both the spike contrast ( R =-0.737, P =0.0002, Spearman’s rank correlation, figure 5(c) ) ... of a network correlated with the classification performance as measured by the maximum F -score of the respective network

Results, 3.4 Performance evaluation on experimental data, paragraph 3; Figure 5(c)

What the paper reports

• Spearman's rank correlation, R = -0.737, p = 2 × 10⁻⁴ (n = the rows of Table 1 (its 20 spine rows reproduce the reported R and P))

Over the 20 rows, spike contrast (all units, imaging span, the combination of table1-network-measures) against the maximum F of noise region 0 and the Methods' classification gives Spearman R = -0.731 (P = 0.0003); over 19 recordings -0.726 (P = 0.0004). Every combination, region and classification gives -0.726 to -0.784 with P at most 0.0004, within -0.737 ± 0.074.

Not quoted: The per-spine values are given only in Table 1 (column best_R); Figure 5(a) plots them. No sentence of the text states them.

17 of 20 spines have a best-R within 10% of Table 1 under at least one reading of p_thresh (96 to 99) and hyperactive strategy, with the Methods' 5-point smoothing; 19 of 20 with the code's lowess (frac 0.0002), and 14 of 20 with the code's default pair (97, percentile strategy). Row 13 reaches at most 0.143 against 0.160 (0.146 with the code's smoothing), row 14 at most 0.354, just below the bound of 0.355 for 0.394, and row 16 at most 0.537 against 0.676. Over the carried-forward readings best-R spans 0.137 to 0.762 (Table 1, 0.148 to 0.765). The assignment of files to rows 14 and 15 (two recordings of subject 3112 with 133 units each) and to rows 16 and 20 (subjects 2748 and 2765, 59 units each) is not fixed by culture and unit count; it was taken from the spike contrast. The outcome does not rest on it, as swapping rows 16 and 20 gives 16 matches, swapping rows 14 and 15 gives 17, and swapping both gives 16, all within 15 to 19. Within a spine best-R moves widely across the eight readings (row 15, 0.320 to 0.701; row 7, 0.269 to 0.605), so a match under some reading is a weak test, and the later claims inherit a reading chosen for being nearest to Table 1; their outcomes are conditional on that choice. With every chain of close bursts merged into one (Methods 2.6) in place of the code's single pass, 16 of 20 carried-forward best-R values match, against 17 (row 20 moves from 0.293 to 0.340).

How the test differed from the paper’s method

• The convolved traces are computed at the frame times as a sum over spikes of the same causal exponential kernel, not with elephant at every 20 kHz sample; for three units of one recording the largest difference from elephant is 0.13 to 0.34 Hz on peaks of 19 to 45 Hz (R above 0.99997).
• Units whose convolved trace is constant over the kept frames have an undefined R and are left out of the ranking (one unit in each of rows 16 and 18); the code's np.argsort would rank such a NaN first.
• Bursts touching the start or end of the recording, which the code cannot handle (it would raise an error), are dropped, as Methods 2.6 says ("Edge cases, where data began or ended during a burst, were excluded").
• Table 1 rows are matched to files by subject (culture) and unit count, as the claims file says; where two assignments fit equally (rows 14 and 15; rows 16 and 20) the one whose spike contrast is nearer to Table 1 is taken, and the swapped assignments are shown (16 to 17 matches, the same outcome). In the file with two spines, the first ROI pair is taken as row 9 and the second as row 10. The paper gives no identifier that settles either.
• The lowess-smoothed population counts are rounded to 9 decimals before they are compared with the integer burst baseline. In every recording some smoothed values equal the baseline up to rounding error (within 1e-9) and lie above it unrounded, so without rounding the burst edges, and with them every later number, depend on the last bits of floating-point arithmetic, which can differ between machines. This is the likely reason the run the round-2 verifier read differed from the author's run, which this machine still reproduced unchanged before the fix. The paper's code compares the unrounded values.
• The ROI pairs are taken in the order spine, shaft, as notebook 1 of the code draws and loads them ("the drawing order has to be spine1, adjacent shaft of spine1, ..."); the spine ROI is the smaller of each pair in 19 pairs and equal in one.

Not quoted: The per-recording values are given only in Table 1; the procedure is in Results 3.3, Methods 2.8 and Figure 4.

Under the Methods' classification (the claims file's main reading) the maximum F is within 10% of Table 1 for 15 of 19 recordings, with each of the three noise regions; rows 6, 14 and 15 are higher here (0.669 to 0.674, 0.586 and 0.575 to 0.578 against 0.599, 0.485 and 0.520) and row 17 lower (0.477 to 0.481 against 0.537). With the code's classification 17 of 19 match. The optimal quantiles differ from Table 1's; Q_test is 0.939 to 0.970 here for every recording where Table 1 has 0.545 to 0.960, and the pair of quantiles lies within 10% of Table 1's for 0 or 1 of 19 recordings. Each maximum is unique, except row 7 with region 2 (5 maxima). Recording 3 (rows 9 and 10) was run with each spine's noise estimate; the two give 0.777 to 0.785, against 0.795. The share of ground-truth surrogates whose best unit is not their own unit, which Methods 2.8 calls rare, is 0.064 to 0.439 (0.064 to 0.126 for rows 1 to 5; 0.383 to 0.439 for rows 8, 14, 15, 16, 17 and 20), nearly the same in the three regions. Recordings with a larger share have lower maximum F (Spearman 0.89) and a larger gap between the two classifications (up to 0.067, row 14), so this share is why the two readings give 15 and 17 of 19. Whether the paper's share was as large cannot be told from the paper. Rows 16 and 18 keep a unit with no spikes; without it, max F changes by at most 0.003 under either classification (0.003 for row 16 under the code's).

How the test differed from the paper’s method

• The background region of the noise estimate is not in the dataset; three 6 by 6 pixel squares near each spine were placed by a rule fixed before any R was computed (see acceptance-9-of-20), and each was run.
• The release probabilities are a fresh draw with the Methods' parameters (seeded), as the claims file's main reading.
• The test-R quantiles are computed from the R tensor with weights, which gives the same values as np.nanquantile on the pooled 1'000 by 1'000 test-Rs of the code's compute_test_r.

We obtained typically large F -scores across experiments (mean = 0.692 ± 0.136 based on 19 recordings with a mean of 102 ± 43 units per recording and 1’000 generated surrogate spine Ca 2+ response traces per unit)

Results, 3.4 Performance evaluation on experimental data, paragraph 1

Under the Methods' classification, over 19 recordings (recording 3 once, with row 9's estimate), the mean maximum F is 0.696 and the population SD 0.121 with noise region 0 (mean 0.695 to 0.696 and SD 0.120 to 0.121 over the regions), so the mean is within tolerance and the SD just below the 0.1224 bound. With the code's classification the mean is 0.665 to 0.666 and the SD 0.134, both within tolerance. F-scores are large across recordings, as the paper states; the spread under the main reading is slightly smaller than reported. The two classifications differ by the misidentified surrogates (6 to 44% of them, table1-optimal-thresholds), so whether the SD falls inside the tolerance depends on that share.

How the test differed from the paper’s method

• As for table1-optimal-thresholds.

For 9 out of 20 spines, the best-matching unit passed the acceptance test, and a monosynaptic connection was confirmed

Results, 3.4 Performance evaluation on experimental data, paragraph 1; Table 1, columns GT_thr, test_thr and accept?; Figure 5(a)

With Table 1's quantiles (reading a), 7 spines are accepted with each of the three noise regions and 18 of 20 decisions agree with Table 1; rows 3 and 16, accepted in Table 1, are rejected here, in each of five seeds and regions. Row 3 has best-R 0.221 (Table 1, 0.217) but a test threshold of 0.249 against 0.190 (region 0); row 16 has best-R 0.537 (Table 1, 0.676) and a GT threshold of 0.684. With the optimal quantiles found here (reading b), 7, 7 and 6 are accepted (18, 18 and 17 decisions agree). All readings and regions fall in the claims file's band for a partial reproduction (17 to 19 decisions), so the outcome does not depend on the noise region. The stored release probabilities and a Bernoulli draw per spike also give 7 accepted and 18 agreeing. GT thresholds are within 10% of Table 1 for 12 or 13 spines and test thresholds for 10 or 11. Rows 9 and 10, two spines of one recording, have identical thresholds in Table 1 (0.507 and 0.639); here each spine has its own noise estimate and the thresholds differ (0.497 and 0.649 against 0.483 and 0.632 with region 0). The two spines share their best (115) and second-best (44) units, and with row 9's noise estimate row 10's thresholds equal row 9's in every region, so Table 1 fits one noise estimate per recording. Row 3's GT threshold is near Table 1's (0.112 to 0.121 against 0.112), but its test-Rs reach 0.245 to 0.254 at Q_test 0.96, against 0.190. Row 16's GT-Rs have median 0.71, so its GT threshold at Q_GT 0.424 is 0.683 to 0.684 against 0.383, and its best-R is 0.537, not 0.676. Both distributions are nearly the same in the three regions, so neither mismatch comes from the noise region. Rows 6 and 18 keep their Table 1 decisions in five seeds. In row 10, background squares 0 and 2 sit on bright processes (noise SD 126 and 96, against 39 for square 1), against the aim of the placement rule; row 10's decision is the same with each square.

How the test differed from the paper’s method

• The dataset holds no background ROI for the noise estimate (Methods 2.7, "We selected a background ROI close to the spine"). As the claims file asks, three background squares were placed near each spine before any R was computed, by a fixed rule in the helper (6 by 6 pixels, at least 4 pixels from any ROI, mean image intensity at or below the 25th percentile of the squares in the 96 by 96 pixel window around the spine, the three nearest to the spine). Square size, margin, window and percentile are this attempt's choices. Each region was run.
• Spine pixel traces are read from the image series with the spine ROI mask of the file; their mean equals the stored ROI trace exactly in every recording.

the accepted connections had significantly larger best-R values (0.584 ± 0.149) compared to the rejected ones (0.331 ± 0.124) (Mann-Whitney U =12.0, n 1 =9, n 2 =11, P <0.05)

Results, 3.4 Performance evaluation on experimental data, paragraph 2; Figure 5(b)

What the paper reports

• Mann-Whitney U, U = 12.0, p < 0.05 (n = 9 accepted and 11 rejected spines)

With this attempt's classification (7 accepted, 13 rejected, reading a, every region), accepted spines have best-R 0.622 (SD 0.081) and rejected 0.337 (SD 0.137); both means are within tolerance. The Mann-Whitney test gives the smaller U = 5 (86 for the accepted group), two-sided p = 0.0005 and one-sided p = 0.0002, so the accepted group is higher with P below 0.05. U is not within 12 ± 1.2; with 7 and 13 spines in place of 9 and 11 the statistic is on a different scale. Reading b with region 2 gives U = 10 (6 and 14 spines, two-sided p = 0.0064).

it is important to note that the value of the best-R alone was not a predictor for the outcome of the test. In fact, as one can see from figure 5(b) , the acceptance test rejected connections with R values as high as ~0.6, while in another case an R value of ~0.2 was enough for acceptance

Results, 3.4 Performance evaluation on experimental data, paragraph 2; Figure 5(b)

The largest best-R of a rejected spine is 0.627 (row 17), within tolerance of about 0.6, and it exceeds the smallest accepted best-R, so no cut on best-R separates the two groups. The smallest accepted best-R is 0.515 (row 2), not about 0.2; row 3, accepted in Table 1 with 0.217, is rejected here. Reading b with region 2 gives a largest rejected best-R of 0.673 (row 9).

and the OUT/IN ratio ( R =0.672, P =0.0012, Spearman’s rank correlation, figure 5(c) ) of a network correlated with the classification performance as measured by the maximum F -score of the respective network

Results, 3.4 Performance evaluation on experimental data, paragraph 3; Figure 5(d) (the text cites figure 5(c) for both; the legend places this one in 5(d))

What the paper reports

• Spearman's rank correlation, R = 0.672, p = 0.0012 (n = the rows of Table 1 (its 20 spine rows reproduce the reported R and P))

Over the 20 rows, OUT/IN against maximum F gives Spearman R = 0.585 (P = 0.007) with noise region 0 and the Methods' classification, and 0.598 (P = 0.007) over 19 recordings, both below the tolerance of 0.672 ± 0.067 (bound 0.605). Region 2 gives 0.582 and 0.595, also below. Region 1 gives 0.615 and 0.623, and the code's classification 0.625 to 0.645 over 20 rows, within tolerance. Under the claims file's main (Methods) classification the outcome therefore depends on the noise region; region 0, the background square nearest the spine, was adopted as the best reading of the Methods' "a background ROI close to the spine". Over 19 recordings, which leave out row 10 (whose squares 0 and 2 sit on processes), region 0 still gives 0.598. The correlation is positive and significant under every combination, region and classification (R 0.582 to 0.754, P at most 0.0072). The OUT/IN values themselves differ from Table 1's (table1-network-measures).

How the test differed from the paper’s method

• Assumption. The noise region is the background square nearest the spine (region 0), from Methods 2.7 ("close to the spine"); regions 1 and 2 are shown. The question is put to the paper's authors in the findings.

Not quoted: The per-recording values are given only in Table 1; the measures are defined in Results 3.4, paragraph 3.

Spike contrast is within 10% of Table 1 for all 19 recordings with all sorted units, whole recording or imaging span (16 of 19 without the hyperactive units). OUT/IN is within 10% for at most 6 of 19 recordings (rows 1, 2, 9, 14, 15 and 19) under any combination of unit set, span and burst extension, so both measures match for 6 recordings, below the 10 the claims file asks for a partial reproduction. OUT/IN here is mostly higher than in Table 1 (row 8, 0.419 against 0.060; row 6, 0.762 against 0.189). With the bursts of each p_thresh and strategy, 10 of 19 recordings can be matched one by one, and row 8 reaches at least 0.139 under every reading. The spike contrast part of the claim holds; the OUT/IN values could not be recovered with the burst detection the paper describes. For rows 14, 15, 16 and 20 the spike contrast also decided the row-to-file assignment (table1-best-r), so its match there is not independent. The code's single merging pass drops every burst after the second in a chain of three or more bursts closer than 75 ms; with the carried-forward reading it loses bursts in 7 of 19 recordings (1 to 6, and 18 in row 20). Merging every chain, as Methods 2.6 says, changes OUT/IN by at most 0.18 (row 2, 2.194 to 2.016), and the code's lowess smoothing with the same pair by at most 0.25 (row 3, 2.293 to 2.547); each still matches 6 of 19.

How the test differed from the paper’s method

• Spike contrast is computed with elephant 1.2.1 (printed in the notebook), where the paper used 0.10.0; the spike_contrast function and its helper are the same in the two versions apart from the docstring (compared in preparing this attempt).
• OUT/IN uses the bursts of the carried-forward reading of table1-best-r.