OpenADMET CYP challenge: method report

Summary

• Regression (pIC50 for CYP1A2, CYP2C9, CYP2D6, CYP3A4). The plain mean, with equal weights, of eight regressors per isoform: four LightGBM models, two CatBoost models and two multitask graph neural networks (Chemprop D-MPNN) trained from scratch on the challenge targets plus 17 auxiliary targets. A post-hoc level placement, fitted on an internal pseudo-test set, shifts CYP1A2 and maps CYP2D6 onto an expected score distribution, preserving ranks.
• TDI (CYP2D6 and CYP3A4, time-dependent inhibition). One pooled LightGBM classifier over both isoforms with an isoform indicator. Decision thresholds chosen to maximise out-of-fold MCC. This file was chosen on 24 September 2026 and has not changed since.
• How choices were made. Every candidate, rule and threshold was written down, with its pass or fail criterion, before the data it would be judged on. Alternatives that lost are listed under What did not work.
• What was not used. No proprietary data. No test label exists to be used, and no value reconstructed by third parties from the public leaderboard entered any prediction or any choice. Test structures, without labels, enter in two places only: the similarity profile used to filter the pseudo-test set P, and the below-4.0 fraction checked by the placement. Public leaderboard values of the organisers' reference entries were used only as a check on the internal benchmark, never as a target.

Data

The overlap between the external data and the 750 test molecules, by the first 14 characters of the InChIKey, is 1 molecule for AID 1851 and 3 for ChEMBL. The auxiliary models are cross-fitted in 5 folds grouped by InChIKey, so a molecule present in the external data never receives a prediction from a model that saw it. The 4 structures that coincide with test molecules are removed from every training set of the graph networks.

Representations

Regression models

Eight members, one weight of 1/8 each.

Heads of the graph networks. The 4 scored pIC50 values (loss weight 1) and 17 auxiliary heads (weight 0.3 each): the TDI-condition pIC50 (4), the log2 fold change of the single-concentration screen (4), the ChEMBL pIC50 (4) and the PubChem AID 1851 maximum response clipped to the range from −150 to 50 (5). Every target is standardised per head and missing labels are masked. The training table has 34,155 structures and 138,066 labels. The weight of 0.3 and the clipping range were taken from the published method of another participating team, not chosen on our data.

Hyperparameters of the second network. They were chosen before any benchmark prediction, on tuning molecules that are not in the benchmark: 15% of the 4,137 training molecules outside the benchmark that have at least one scored label (about 620, drawn with a fixed seed), with models trained without them and without the 768 benchmark molecules. Thirteen configurations were tried with one seed: the stock one and twelve drawn at random from depth 3, 4 or 5, message width 300 or 600, feed-forward width 300 or 600 with 1 or 2 layers, dropout 0, 0.1 or 0.2, and 50 or 100 epochs. The one with the lowest macro ST-RAE on the tuning molecules was kept (0.6711, against 0.7874 for the stock configuration). The benchmark labels entered no choice. No other component was tuned.

Ensemble and level placement

The ensemble is a plain mean. Each step was written down before it was run and checked with a paired bootstrap by molecule on the internal benchmark (2,000 resamples). Lower ST-RAE is better; "macro" averages the four isoforms.

Alone, the second network is better than the first (macro ST-RAE 0.609 against 0.678, difference −0.069 [−0.112; −0.032]). Putting it in place of the first changes nothing measurable (interval [−0.007; +0.006]); next to it, as an eighth member, it helps, because the two networks err differently enough to compensate. The out-of-fold predictions of the networks are honest by construction: for fold k, every row of a fold-k molecule is removed from training, auxiliary labels included.

Level placement. The benchmark ranks methods but says nothing about the absolute level of the test, so a post-hoc map is fitted on the internal pseudo-test set P (next section) with a procedure and an adoption rule written before the data. For the networks, the predictions on P come from models trained without the 201 P molecules: all of them come from the single-concentration screen, and a model that had seen their screen readout would be optimistic on P, while no test molecule has one. Two isoforms adopt a map, three families being allowed (none, affine, quantile map with shrinkage):

• CYP1A2: a shift, p' = p − 0.42;
• CYP2D6: a rank-preserving quantile map towards the expected score population, shrinkage k = 0.75;
• CYP2C9 and CYP3A4: unchanged.

The expected population is simulated from the design of the test set as we understand it from the challenge (analogues of strong training actives) using only the single-concentration library and training labels; no test structure or label and no leaderboard information enters it. One adoption condition compares the fraction of test predictions below 4.0 with that population and was evaluated on the v6 test predictions.

On P, the macro ST-RAE after placement is 0.665 for v6, against 0.675 for the seven-model version (v5, built but not submitted), 0.697 for the six-model version and 0.715 for the first placement (v3). By isoform, v6 gives CYP1A2 0.710, CYP2C9 0.770, CYP2D6 0.588 and CYP3A4 0.592, better on all four than the seven-model version. These comparisons come from unpaired runs on 201 molecules: they are used as a guard (the new version must not be worse), not as evidence of a gain.

The Spearman correlation between the v6 columns and the earlier seven-model version is 0.998 (CYP1A2), 0.998 (CYP2C9), 0.992 (CYP2D6) and 0.999 (CYP3A4); between v6 and the v3 regression file that was on the leaderboard before v6 it is 0.972, 0.984, 0.926 and 0.990.

TDI model

One classifier, C_pool, trained on the rows of both isoforms with an isoform indicator column. Folds are by molecule, so the two rows of a molecule share a fold. Inputs: the same descriptors as the regression plus the 14 auxiliary columns; regression outputs do not enter the TDI. LightGBM defaults, seed 20260924.

The thresholds maximise the out-of-fold MCC: 0.0794 for CYP2D6 and 0.28557 for CYP3A4. The MCC with the choice of variant also nested is 0.131 [−0.014; 0.275] on CYP2D6 and 0.421 [0.337; 0.504] on CYP3A4; the mean of the two is 0.276. The honest figure for CYP2D6 is 0.131, and its interval includes zero. The out-of-fold AUC is 0.693 on CYP2D6 and 0.779 on CYP3A4. The MCC optimum of a weak classifier on imbalanced classes labels 480 of the 750 test molecules (64%) positive for CYP2D6, against a true prevalence of 22% in the validation molecules. This follows from the rule written beforehand; it was not adjusted after the fact.

Validation

• Internal benchmark. 768 training molecules that have an active analogue elsewhere in the training set (Tanimoto 0.50–0.80), 5 folds, paired bootstrap by molecule with 2,000 resamples. Out-of-fold ST-RAE of the v6 eight-model mean: CYP1A2 0.629, CYP2C9 0.559, CYP2D6 0.838, CYP3A4 0.365, macro 0.598. Spearman: 0.698, 0.668, 0.570, 0.862, mean 0.699. The benchmark ranks methods, and it is optimistic by about 0.11 on level: the tutorial LightGBM scores 0.780 on it and 0.893 on the public leaderboard. It says nothing about placement. Every candidate that passed was also chosen because it passed this same benchmark, so its gains are optimistic.
• Pseudo-test set P, declared. 201 molecules from 148 parent compounds. The universe is the single-concentration library (4,376 molecules with a log2 fold change on all four enzymes) minus molecules whose SMILES is in the test. The parents are training molecules with a measured pIC50 of at least 6.0 on CYP1A2, CYP2C9 or CYP3A4. A molecule enters if it is an analogue of a parent (Tanimoto 0.50–0.80) and its maximum similarity to the training set falls between the 5th and 95th percentile of the same quantity on the test structures (structures only, no labels). Labels are of two kinds: measured (a dose-response exists in the training set; the prediction is out-of-fold) and imputed from the screen readout. Folds are grouped by parent; intervals come from a paired bootstrap by parent. P is used only for placement.
• Controls. The rebuilt v6 matches the evaluated ensemble with a maximum difference of 1.8e−15; the placement reproduces the below-4.0 fractions recorded in the placement stage; the file has 750 rows with names, order and SMILES identical to the official test file; the four official validators pass with an empty error list.

What did not work

Candidates compared with the best single LightGBM component (macro ST-RAE 0.6677 on the benchmark), paired 95% interval of the difference:

Tried on 5 and 6 October 2026 against the seven-model mean (0.6152), each with its rule written before the run:

For the TDI, a three-component ensemble (C_pool, TabICL, CheMeleon, Platt-calibrated, with thresholds at the expected prevalence) lost to the first recipe on nested MCC, −0.092 [−0.184; −0.006]; its score was not worse (AUC 0.682 against 0.693 on CYP2D6, 0.795 against 0.779 on CYP3A4), the threshold was. A predicted "TDI shift" (TDI-condition pIC50 minus direct pIC50) was worse than C_pool as a classifier on its own (AUC −0.161 [−0.222; −0.101] on CYP3A4) and, as an extra feature, lowered the CYP3A4 AUC (−0.024 [−0.047; −0.004]). A threshold band tried on the out-of-fold scores did not pass its gate (difference in macro MCC −0.0374 [−0.1176; +0.0327]). The public Octant dataset turned out to be another processing of the challenge TDI training set (1,250 of 1,340 identifiers shared, r = 0.96 with the TDI-condition pIC50), so it adds no new data.

Reproducibility

The project repository is currently private and its release is not decided; the commit that holds everything described here is f60ed6e. Each stage has a rule file written before its script and an outcome file after it, under esperimenti/uscite/<stage>/. The v6 files are in invii/v6/ and the order of the commands is:

PYTHONPATH=pacchetti/cb OMP_NUM_THREADS=4 .venv/bin/python esperimenti/106_insieme_v4.py
PYTHONPATH=pacchetti/cb OMP_NUM_THREADS=4 .venv/bin/python esperimenti/107_invio_v4.py
.venv/bin/python esperimenti/108_chemprop_mt.py tabella
open -a Terminal esperimenti/108_corsa.command       # first network: 21 models
open -a Terminal esperimenti/119_iper_corsa.command  # second network: tuning, then 15 benchmark models
.venv/bin/python esperimenti/120_iper_valuta.py
open -a Terminal esperimenti/123_iper_w1_corsa.command   # second network without P and on all data
.venv/bin/python esperimenti/124_iper_w1_valuta.py
.venv/bin/python esperimenti/125_invio_v6.py

CPU only, OMP_NUM_THREADS=4; seeds 20260924, 1 and 2 for the networks. The component predictions come from esperimenti/100_maplight.py, esperimenti/101_catboost_v1.py and the pipeline of the first submission. The frozen recipe for the tree-based part (definitivo/RICETTA-v4.md, run with definitivo/corri-v4.sh) rebuilds it from scratch bit for bit on three Apple-silicon Macs (M4 Pro, M2, M1), CatBoost included (definitivo/confronto-tre-mac-v4.txt: 16 fingerprints, 0 different). The two graph networks have not been rebuilt on other machines: they were trained on the CPU of the M4 Pro, and on that machine the same seed gives the same predictions.

Limits

• The benchmark is optimistic on level by about 0.11, and the candidates that passed were selected on it.
• P has 201 molecules from 148 parents; its comparisons are unpaired and used as a guard. The gain of the placement came, in the stage that measured it, from molecules with imputed labels, and the expected population uses the same imputer. This was measured for the first placement and not measured again since.
• The expected population assumes the test was built as the challenge describes it. If it was not, the placement of CYP2D6 in particular can be wrong.
• The search for the second network's hyperparameters is small: 13 configurations, one seed, about 620 tuning molecules.
• TDI on CYP2D6: MCC 0.131 with an interval that includes zero, from 38 positive validation molecules; the file labels 64% of the test positive.
• The graph networks are reproduced on one machine only.
• The ChEMBL release number was not recorded in the project files; the download date is.
• The code is not public at the time of writing.

Final submission (v7)

Declarations and references

Proprietary data: no. Pre-trained models in the submitted files: none; every model is trained from scratch (TabICL, CheMeleon and Uni-Mol were evaluated and did not enter). External public data: PubChem AID 1851 and ChEMBL, as above.

• Veith H. et al. Comprehensive characterization of cytochrome P450 isozyme selectivity across chemical libraries. Nat. Biotechnol. 27, 1050–1055 (2009).
• Zdrazil B. et al. The ChEMBL Database in 2023. Nucleic Acids Res. 52, D1180–D1192 (2024).
• Heid E. et al. Chemprop: a machine learning package for chemical property prediction. J. Chem. Inf. Model. 64, 9–17 (2024).
• Graff D. E. et al. Chemprop v2: an efficient, modular machine learning package for chemical property prediction. J. Chem. Inf. Model. 66, 28–33 (2026), the major version used here (2.3.1).
• Notwell J. H., Wood M. W. ADMET property prediction through combinations of molecular fingerprints. arXiv:2310.00174 (2023), the MapLight recipe.
• Prokhorenkova L. et al. CatBoost: unbiased boosting with categorical features. NeurIPS (2018).
• Ke G. et al. LightGBM: a highly efficient gradient boosting decision tree. NeurIPS (2017).
• Zhou G. et al. Uni-Mol: a universal 3D molecular representation learning framework. ICLR (2023).
• Qu J., Holzmüller D., Varoquaux G., Le Morvan M. TabICL: a tabular foundation model for in-context learning on large data. ICML (2025), arXiv:2502.05564.
• Burns J. W. et al. Deep learning foundation models from classical molecular descriptors (CheMeleon). arXiv:2506.15792 (2025).