Does the knowledge graph schema matter for clinical outcome prediction?

Jhee and colleagues generated 10,000 synthetic patients modelled on 552 people with a ruptured intracranial aneurysm and represented them as knowledge graphs following two schemas, the Swiss Personalized Health Network (SPHN) schema and CARE-SM, a model built on the Semanticscience Integrated Ontology. They asked three questions. Do graph models outperform tabular models? Does the schema matter? Does the way time is encoded matter? They reported that a relational graph convolutional network (R-GCN) with a literal encoder outperformed tabular models (AUC 0.91 against at most 0.71), that SPHN outperformed CARE-SM (0.91 against 0.50), and that time encodings performed alike.

The task is three-class node classification of the outcome at the end of the stay (discharge home, rehabilitation or death), evaluated with stratified 10-fold cross-validation and weighted one-versus-rest AUC.

We reran the public pipeline (TeamHeKA/neurovasc, commit 1619d99) with its pinned software and the original protocol: the same folds, one ninth of each training set for validation, 2,000 full-batch epochs and the checkpoint with the best validation accuracy. Every change to the original code is marked in the repository. We then corrected the errors listed below and compared each configuration with tabular models on the same patients and folds.

Real patients

From MIMIC-IV v3.1 we selected adults with a first admission coded for non-traumatic subarachnoid haemorrhage. At landmarks 24, 48 and 72 hours after admission, patients still alive and in hospital entered the analysis (1,388, 1,287 and 1,190 patients). Features used only information recorded before the landmark, namely age, sex, Glasgow Coma Scale, vital signs, laboratory values and the presence and timing of eight treatments. The outcome was discharge home (560 at 24 hours), to rehabilitation or care (530), or death including hospice (298).

We built SPHN and CARE-SM graphs following the original templates, an SPHN variant with time relations, a compiled CARE-SM graph with shortcut edges from each patient to each feature, and a graph using the released SPHN 2026.1 classes and properties with ATC, ICD-10-PCS, LOINC and SNOMED CT codes. Before training, every graph was checked automatically. Every numeric value had to reach the model, every feature had to lie within the model's reach, and no outcome edge could be present. Patient-specific nodes started from zero embeddings to prevent memorisation.

To test whether ontology meaning helps, we varied only the class and code nodes shared across patients. They started from random vectors, from SapBERT embeddings of their labels (also shuffled across nodes as a control) or from OWL2Vec* embeddings of the SPHN schema extended with the code hierarchies. Alternatively, their ancestors were added as graph edges, from the ATC and ICD-10-PCS codes themselves or from the OMOP vocabularies, which add SNOMED CT and LOINC ancestors.

Models were compared on the same ten folds with paired t-tests, and we report mean differences with 95% confidence intervals. Only aggregate results left the secure environment, and counts below 11 are never reported.

Three observations explain the published differences. The published tabular results reproduce only on an earlier version of the synthetic table, so the graph advantage reflected different data. The five-layer CARE-SM model failed because the numeric values were dropped, not because of depth or over-smoothing, since the corrected five-layer model performs well. Merging the CARE-SM graphs into one graph lost no information, because the features read back from it give the same performance as the original table.

The original authors later shared their CARE-SM training notebook. It trains CARE-SM with the same three-layer model as SPHN, with an embedding size of 100 and its own fold seed. Run with these settings, the published graph gives 0.502 ± 0.018, matching the published 0.50, while the graph with its values restored gives 0.822 ± 0.008 with four layers and 0.886 ± 0.006 with unit-scale embeddings.

Age alone predicts the synthetic outcome as well as any model (0.914 ± 0.007), and SPHN delivers it on a dedicated edge. Relabelling that edge lowers SPHN, so part of its lead came from one predicate. The care events carry no outcome signal by construction, which is why the time encodings could not differ.

Why age is a sufficient predictor

The reason lies in how the synthetic patients were generated. The generator in the original repository works in four steps.

Age is the only continuous variable in the sum and is measured in years, while most of the others are 0/1 codes, so the sum, and hence the outcome, is close to age cut at two thresholds. The data show this directly. Mean age is 43.0, 60.1 and 74.8 years for discharge home, rehabilitation and death, with a spread of only 7 to 9 years within each class, and a rule with cut-offs near 50 and 70 years classifies 78 percent of patients correctly, the same accuracy as the graph model on SPHN (0.781 ± 0.013). Variables that enter the sum with visible weight carry some signal (sex alone reaches an AUC of about 0.76), whereas length of stay, the Glasgow Coma Scale and the number of medical acts carry none. Mixing unstandardised values also pushes some variables outside clinical ranges, for example Glasgow Coma Scale values up to 52 on a scale that ends at 15. The care pathway events are generated afterwards, independently of everything else.

We infer the dominance of age from the generator code and the data. The mixing weights are stored in a serialised file in the original repository that we did not load. The figures in this subsection were computed from data/syn_data.csv at commit 1619d99.

Unlike the synthetic data, age alone predicted the outcome weakly while the care events and their timing carried signal. Compiling CARE-SM and adding time relations to SPHN changed nothing. Passing messages in both directions did not change the template SPHN graph, but the graph built with released SPHN terms performed better and matched gradient boosting. Because that graph differs from the template in several ways at once, including its layout and depth, the gain cannot be attributed to the released terms themselves.

None of the five ways of supplying ontology content changed performance by more than 0.002 AUC, and the differences are smaller than the variation between seeds. The two embeddings do carry different information. OWL2Vec* places heart rate next to the other vital-sign results and the drugs together, while SapBERT groups labels by wording. On these features the models use the graph layout and the numeric values rather than the meaning of the terms. The feature set is small and dense, so related codes have little need to share strength, and settings with many sparse codes may behave differently.

The large schema effect in the original study came from how the graphs were built and preprocessed. Once corrected, the schema made at most a small difference, and graph models performed as well as, but not better than, tabular models on the same data. The original graphs used locally minted identifiers, and no hierarchy or reasoning entered the models, so the comparison was one of graph structures rather than of ontologies. Our experiments with released terms, standard codes, embeddings and hierarchies support this reading.

Limitations. The real-data analysis covers one centre and one outcome definition. MIMIC-IV has no structured aneurysm severity score, and treatments partly reflect severity, so their signal is prognostic rather than causal. We did not rerun the TransE and RDF2Vec baselines of the original study, whose code is not in its repository.

All code, pinned environments and per-fold results are in the repository, and every number on this page is computed from those results by site/build.py. The synthetic data and original code are available from TeamHeKA/neurovasc. MIMIC-IV requires credentialed access through PhysioNet. The OMOP-based hierarchy needs a vocabulary export from Athena, which includes licensed SNOMED CT and LOINC content.

The original authors' repository notes the discrepancy between the tabular and graph data. We shared these findings with the original authors before publication and thank them for their responses.