Results (July 2019)

TADPOLE's first phase is complete and we have evaluated all the prize-eligible submissions.

Watch the live announcement on YouTube: https://www.youtube.com/watch?v=BFS9Sr0lhuM

Check below for a description of the evaluation dataset and the overall rankings.

General observations

• There was no clear "one-size-fits-all" winner.
• Data-driven approaches for both feature selection and prediction of target variables generally performed well.
• Many teams combined different types of algorithms to produce forecasts:
1. Most used statistical regression;
2. Some used generic machine learning techniques that are robust and can work well for other problems; and
3. Some used disease progression models that are specifically tailored for the current problem of disease prediction.
• Forecasts were very good for clinical diagnosis and ventricle volume -- on the other hand, predicting ADAS turned out to be very difficult -- no team was able to generate forecasts that were significantly better than random guessing 
• Meta-analysis results: the most-important features that helped improve predictions were DTI & CSF for clinical diagnosis, and "augmented features" for Ventricle volume prediction.
• Throughout this page we will refer to ADAS-Cog 13 as simply ADAS.

TADPOLE Prize winners

Overall Results

Legend:

• MAUC – Multiclass Area Under the Curve
• BCA – Balanced Classification Accuracy
• MAE – Mean Absolute Error
• WES – Weighted Error Score
• CPA – Coverage Probability Accuracy for 50% Confidence Interval
• ADAS – Alzheimer's Disease Assessment Scale Cognitive (13)
• VENTS – Ventricle Volume
• RANK (overall) – We first compute the sum of ranks from MAUC, ADAS MAE and VENTS MAE, then derive the final ranking from these sums of ranks. For example, the top entry has the lowest sum of ranks from these three categories.

Note (14 June 2019): The rankings in each prize category can be found by ordering according to Diagnosis MAUC, and ADAS and Ventricle MAE. The overall rankings below require valid submissions for every target variable. 

Overall scores — longitudinal dataset D2

The results on the D2 dataset suggest that we do not have a clear winner on all categories. While Frog had the best overall submission with the lowest sum of ranks, for each performance metric individually we had different winners: Frog (clinical diagnosis MAUC of 0.931), ARAMIS-Pascal (clinical diagnosis BCA of 0.850), FortuneTellerFish-Control (ADAS MAE and WES of 4.7), VikingAI-Sigmoid (ADAS CPA of 0.02), EMC1-Std/EMC1-Custom (ventricle MAE of 0.41 and ventricle WES or 0.29), and DIKU-ModifiedMri-Std/ DIKU-ModifiedMri-Custom (ventricle CPA of 0.01).

Overall scores — cross-sectional dataset D3

Here, most submissions have worse performance compared to the equivalent predictions on the D2 longitudinal dataset, due to the lack of longitudinal, multimodal data. GlassFrog-Average had the best overall rank and obtained a diagnosis MAUC of 0.897, ADAS MAE of 5.86 and a Ventricle MAE of 0.68 (% ICV). For diagnosis prediction, IBM-OZ-Res obtained the highest clinical diagnosis scores: MAUC of 0.905 and BCA of 0.830. For ADAS predictions, CyberBrains had the best MAE and WES of 4.72. ATRI-Biostat-MA obtained the best ADAS CPA of 0.04. For Ventricle prediction, GlassFrog-LCMEM-HDR had a MAE of 0.48 (% ICV) and the best WES of 0.38, while the 6 DIKU submissions obtained the best CPA of 0.05.

Additional entries

In addition to the standard predictions and the benchmarks, we also included two consensus predictions by taking the mean (ConsensusMean) and median (ConsensusMedian) over all predictions from all participants. For D2 predictions, the ConsensusMedian submission obtained the best overall rank, obtaining MAUC of 0.925 in diagnosis prediction (second-best), 5.12 error on ADAS-Cog 13 MAE (ninth-best) and 0.38 on Ventricles MAE, the best result in this category for D2. On the other hand, ConsensusMean ranked 3rd overall on D2, with diagnosis MAUC of 0.920 (fourth-best), ADAS-Cog 13 MAE of 3.75, the best prediction in this category, and Ventricle MAE of 0.48 (rank 16). For ADAS-Cog 13 and Ventricle volume prediction, the best consensus methods reduced the error by 11% and 8% respectively compared to the best prediction from participants or benchmarks.

In order to test whether the best results have not been obtained by chance due to randomness in the test set, we evaluated n=62 (as many as number of entries) randomly perturbed predictions from the simplest benchmark, BenchmarkLastVisit, and computed the best results obtained by any of these predictions. These are shown as RandomisedBest, and obtain high scores especially for ADAS-Cog 13, ranking 3rd with a final MAE of 4.52. High performance scores are also obtained for Ventricles, ranking 14 with an MAE of 0.47, a 14% increase in error from the best forecast, while for diagnosis prediction a lower MAUC score of 0.797 is obtained, ranking 43rd. This suggests that the entries with higher MAE than RandomisedBest should be interpreted with care, as the scores and ranks could be high due to randomness in the test set. This is particularly relevant for ADAS-Cog 13 predictions, where only the BenchmarkMixedEffects and ConsensusMean got better results, suggesting all other methods are not able to predict the ADAS-Cog 13 any better than random guessing based on the last available measurement.

It is worth mentioning that, while drafting the manuscript, we discovered that dropping APOE as a covariate in the BenchmarkMixedEffectsAPOE model considerably decreases the error in ADAS prediction, so we included it as an additional entry for scientific interest.  

Additional entries for D2

Additional entries for D3

Confidence Intervals

Below are confidence intervals (CIs) computed for every submission, based on 50 bootstraps of the test set D4. The first figure (Fig. 1) shows CIs based on forecasts from D2, while the second (Fig. 2) shows CIs for forecasts on D3.

Fig 1. Confidence intervals for forecasts based on the longitudinal D2 prediction set.

Fig 2. Confidence intervals for forecasts based on the cross-sectional D3 prediction set.

Meta-analysis

To understand which types of features and algorithms yielded higher performance, we show here associations between predictive performance and feature selection methods, different types of features, methods for data imputation, and methods for forecasting of target variables (diagnosis, ADAS and ventricles). For each type of feature/method and each target variable (clinical diagnosis, ADAS and Ventricles), we show the distribution of estimated coefficients from a general linear model, derived from the approximated inverse hessian matrix at the maximum likelihood estimator. From this analysis we removed outliers, defined as submissions with ADAS MAE higher than 10 and Ventricle MAE higher than 0.15 (%ICV). For all plots, distributions to the right of the gray dashed vertical line are associated with better performance. 

The results in Fig. 3 below show trends that indicate what aspects of the methods could be associated with better performance. For feature selection, methods that perform manual selection of features are associated with better predictive performance in ADAS13 and Ventricles. In terms of feature types, including features from many modalities was generally associated with an increase in overall performance, except for FDG (for all target variables). Moreover, augmented features correlate with overall performance improvements especially for ventricle prediction. In terms of data imputation methods, while some differences can be observed, no clear conclusions can be drawn currently. In terms of prediction models, we notice that neural networks are more significantly associated with increased performance in ventricle prediction, while disease progression models are associated with decreased performance in prediction or clinical diagnosis and ventricles. However, given the small number of methods tested (<50) and the large number of degrees of freedom (n=21), these results should be interpreted with care.

Fig 3. Associations between the prediction of clinical diagnosis, ADAS and Ventricle volume and different strategies of (top) feature selection, (upper-middle) types of features, (lower-middle) data imputation strategies and (bottom) prediction methods for the target variables. For each type of feature/method (rows) and each target variable (columns), we show the distribution of estimated coefficients from a general linear model. Positive coefficients, where distributions lie to the right of the dashed vertical line, indicate better performance than baseline (vertical dashed line). For ADAS and Ventricle prediction, we flipped the sign of the coefficients, to consistently show better performance to the right of the vertical line.

Demographics of D1-D4 datasets

Summary of TADPOLE datasets D1-D4. Each subject has been allocated to either Control, MCI or AD group based on diagnosis at the first available visit within each dataset. The bottom table contains the number of visits with data available, by modality. For example, in D4 there were a total of 150 visits where an MRI scan was undertaken, which represented a total of 64% of all visits analysed across all subjects in D4. 

Description of Algorithms

Summary

We had a total of 33 participating teams, who submitted a total of 58 forecasts from D2, 34 forecasts from D3, and 6 forecasts from custom prediction sets. A total of 8 D2/D3 submissions from 6 teams did not have predictions for all three target variables, so we only computed the performance metrics for the available target variables. Another 3 submissions lacked confidence intervals for either ADAS or ventricle volume, which we imputed using default low-width confidence ranges of 2 for ADAS and 0.002 for Ventricles/ICV. 

Table 1 below summarizes the methods used in the submissions in terms of feature selection, handling of missing data, predictive models for clinical diagnosis and ADAS/Ventricles biomarkers, as well as training and prediction times. Condensed descriptions of each submitted method can be found here, while even more detailed descriptions are here (original files submitted by participants). 

Table 1. Summary of methods used in the TADPOLE submissions. Keywords: SVM – Support Vector Machine, RF – random forest, LSTM – long short-term memory network, NN – neural network, RNN – recurrent neural network, SMNSR - Sparse Multimodal Neighbourhood Search Regression, DPM – disease progression model, KDE – kernel density estimation, LDA – linear discriminant analysis, SM – slope model, ECOC - error-correcting output codes, SVD – singular value decomposition (*) Augmented features

Participant statistics

Locations of participating teams

Team categories

Prediction methods

Organised by:  

Prize sponsors: