Artifact-Complete Benchmarking of a Hosted CDE Contract
on PerturBench Norman19

1. Problem Statement

PerturBench Norman19 measures held-out perturbation-response prediction for unseen combination interventions. Given a perturbation not yet run in the lab, what response should be expected? How accurate are predictions on a frozen evaluation contract? How does CDE compare with same-split baselines under the same scoring rules?

2. Benchmark Setup

2.1 Data and Frozen Split

The evaluation uses the Norman et al. (2019) Perturb-seq dataset, processed and distributed as norman19_processed.h5ad.gz.

The dataset is partitioned into 39 training, 46 validation, and 46 test held-out combination pairs. The split is deterministic and frozen — identical across all models evaluated in this report.

2.2 CDE System Description

CDE was treated as a black-box, agent-facing scientific platform and exercised only through the approved benchmark surfaces: prediction, causal discovery, CDEOfficialAdapter, and campaign scripts. Broader surfaces (REST, SDK, CLI, MCP) exist but all claims are limited to benchmark-facing evidence only.

2.3 CDE Benchmark Execution Path

2.4 GEARS Baseline Description

GEARS (Roohani et al. 2023) is the established state-of-the-art for gene perturbation prediction. It is a graph-enhanced gene activation and repression simulator that uses gene ontology (GO) and co-expression graph structure to predict post-perturbation expression. For this benchmark, GEARS was trained and evaluated on the identical frozen split used by all other models, with GO + co-expression features enabled (the configuration recommended by the authors). Results are averaged over 3 random seeds (seed std = 0.0141).

3. Unified Metrics

3.1 Unified Same-Split Leaderboard

All models were evaluated on the identical frozen test split. Metrics are computed on log-fold-change predictions against observed post-perturbation expression profiles. The primary ranking metric is Cosine LogFC (cosine similarity in log-fold-change space). Secondary metrics include RMSE (mean across genes), Top-20 DE MSE (mean squared error on the 20 most differentially expressed genes), and Pearson DE (Pearson correlation on differentially expressed genes).

CDE (full pipeline) achieves the highest Cosine LogFC (0.9109) and Pearson DE (0.9684), and the lowest RMSE (0.0425) and Top-20 DE MSE (0.054874) across all evaluated models. The margin over the strongest baseline (Linear baseline at 0.9022) is +0.0087 Cosine LogFC. GEARS (Roohani et al. 2023), evaluated on the same frozen split with GO + co-expression features (3 seeds, std = 0.0141), achieves Cosine LogFC 0.7158, RMSE 0.0725, and Pearson DE 0.7929 — ranking below all CDE variants and the nearest-neighbor baseline on the primary metric.

3.2 Same-Split Subgroup Analysis

Test perturbation pairs are stratified by the number of constituent genes that appeared in the training set. combo_seen0 (n=7) contains pairs where neither gene was seen in training — the hardest subgroup, requiring maximal generalization. combo_seen1 (n=20) contains pairs where one gene was seen. combo_seen2 (n=19) contains pairs where both genes were seen individually but never in combination.

CDE (full pipeline) leads across all three subgroups. The advantage is most pronounced on combo_seen0 (0.9358 vs. 0.9328 for the linear baseline), suggesting that causal evidence provides the greatest benefit when predicting outcomes for genuinely novel gene combinations with no prior training exposure.

4. Reproducibility

4.1 Environment

4.2 Key Artifact Paths

4.3 Public Data Downloads

All leaderboard CSV data is publicly available for independent verification and reproducibility.

5. Limitations

1. Claims limited to frozen Norman19 benchmark contract. All performance figures reported in this document are specific to the PerturBench Norman19 frozen test split and should not be generalized beyond scope.

2. Intentionally avoids internal implementation disclosure. CDE was evaluated as a black-box platform. This report does not disclose internal model architectures, training procedures, or hyperparameter configurations. It reports only benchmark-facing evidence.

3. Secondary analyses support interpretation but do not replace biological validation. Computational predictions of perturbation responses — however accurate on benchmark metrics — are not substitutes for experimental validation or independent replication.

6. Data, Code, and Artifact Availability

Data access is documented by the download manifest and split manifest in data/locks/. Code paths: campaign scripts in perturbench_campaign/scripts/. Reproducibility depends on saved results, manifests, and the report bundle.

Authorship