#### Results (July 2019)

TADPOLE's first phase is complete and we have evaluated all the prize-eligible submissions.

Watch the live announcement on YouTube: https://www.youtube.com/watch?v=BFS9Sr0lhuM

Check below for a description of the evaluation dataset and the overall rankings.

#### General observations

- There was no clear "one-size-fits-all" winner.
- Data-driven approaches for both feature selection and prediction of target variables generally performed well.
- Many teams combined different types of algorithms to produce forecasts:
- Most used statistical regression;
- Some used generic machine learning techniques that are robust and can work well for other problems; and
- Some used disease progression models that are specifically tailored for the current problem of disease prediction.
- Forecasts were very good for clinical diagnosis and ventricle volume -- on the other hand,
**predicting ADAS turned out to be very difficult**-- no team was able to generate forecasts that were significantly better than random guessing **Meta-analysis results**: the most-important features that helped improve predictions were DTI & CSF for clinical diagnosis, and "augmented features" for Ventricle volume prediction.- Throughout this page we will refer to ADAS-Cog 13 as simply ADAS.

### TADPOLE Prize winners

Category |
Team |
Members |
Institution |
Country |
Prize |

Overall best | Frog | Keli Liu, Paul Manser, Christina Rabe | Genentech | USA | £5000 |

Clinical status | Frog | Keli Liu, Paul Manser, Christina Rabe | Genentech | USA | £5000 |

Ventricle volume | EMC1 | Vikram Venkatraghavan, Esther Bron, Stefan Klein | Erasmus MC | Netherlands | £5000 |

Best university team | Apocalypse | Manon Ansart | ICM, INRIA | France | £5000 |

High-School (best) | Chen-MCW | Gang Chen | Medical College Wisconsin | USA | £5000 |

High-School (runner up) | CyberBrains | Ionut Buciuman, Alex Kelner, Raluca Pop, Denisa Rimocea, Kruk Zsolt | Vasile Lucaciu College | Romania | £2500 |

Overall best D3 prediction | GlassFrog | Steven Hill, Brian Tom, Anais Rouanst, Zhiyue Huang, James Howlett, Steven Kiddle, Simon R. White, Sach Mukherjee, Bernd Taschler | Cambridge University | UK | £2500 |

#### Overall Results

Legend:

- MAUC – Multiclass Area Under the Curve
- BCA – Balanced Classification Accuracy
- MAE – Mean Absolute Error
- WES – Weighted Error Score
- CPA – Coverage Probability Accuracy for 50% Confidence Interval
- ADAS – Alzheimer's Disease Assessment Scale Cognitive (13)
- VENTS – Ventricle Volume
- RANK (overall) – We first compute the sum of ranks from MAUC, ADAS MAE and VENTS MAE, then derive the final ranking from these sums of ranks. For example, the top entry has the lowest sum of ranks from these three categories.

Note (14 June 2019): The rankings in each prize category can be found by ordering according to Diagnosis MAUC, and ADAS and Ventricle MAE. The overall rankings below require valid submissions for **every** target variable.

#### Overall scores — longitudinal dataset D2

RANK |
TEAM NAME | MAUC | BCA | ADAS MAE | ADAS WES | ADAS CPA | VENTS MAE | VENTS WES | VENTS CPA |
---|---|---|---|---|---|---|---|---|---|

1.0 | Frog | 0.931 | 0.849 | 4.850 | 4.74000 | 0.440 | 0.44690 | 0.327 | 0.470 |

2.0 | EMC1-Std | 0.898 | 0.811 | 6.050 | 5.40000 | 0.450 | 0.41160 | 0.286 | 0.430 |

3.0 | VikingAI-Sigmoid | 0.875 | 0.760 | 5.200 | 5.11000 | 0.020 | 0.45340 | 0.354 | 0.200 |

4.0 | EMC1-Custom | 0.892 | 0.798 | 6.050 | 5.40000 | 0.450 | 0.41160 | 0.286 | 0.430 |

5.0 | CBIL | 0.897 | 0.803 | 5.660 | 5.65000 | 0.370 | 0.46250 | 0.463 | 0.090 |

6.0 | Apocalypse | 0.902 | 0.827 | 5.570 | 5.57000 | 0.500 | 0.51720 | 0.517 | 0.500 |

7.0 | GlassFrog-Average | 0.902 | 0.825 | 5.260 | 5.27000 | 0.260 | 0.68120 | 0.601 | 0.330 |

8.0 | GlassFrog-SM | 0.902 | 0.825 | 5.770 | 5.92000 | 0.200 | 0.52340 | 0.335 | 0.200 |

9.0 | BORREGOTECMTY | 0.866 | 0.808 | 5.900 | 5.82000 | 0.390 | 0.42990 | 0.374 | 0.400 |

10.0 | EMC-EB | 0.907 | 0.805 | 6.750 | 6.66000 | 0.500 | 0.44660 | 0.403 | 0.480 |

11.5 | lmaUCL-Covariates | 0.852 | 0.760 | 6.280 | 6.29000 | 0.280 | 0.41550 | 0.407 | 0.110 |

11.5 | CN2L-Average | 0.843 | 0.792 | 5.310 | 5.31000 | 0.350 | 0.49030 | 0.490 | 0.330 |

13.0 | VikingAI-Logistic | 0.865 | 0.754 | 6.020 | 5.91000 | 0.260 | 0.45340 | 0.354 | 0.200 |

14.0 | lmaUCL-Std | 0.859 | 0.781 | 6.300 | 6.33000 | 0.260 | 0.42070 | 0.411 | 0.090 |

15.5 | CN2L-RandomForest | 0.896 | 0.792 | 5.730 | 5.73000 | 0.420 | 0.71210 | 0.712 | 0.410 |

15.5 | FortuneTellerFish-SuStaIn | 0.806 | 0.685 | 4.810 | 4.81000 | 0.210 | 0.48590 | 0.486 | 0.180 |

17.0 | CN2L-NeuralNetwork | 0.783 | 0.717 | 5.360 | 5.36000 | 0.340 | 0.44090 | 0.441 | 0.270 |

18.0 | BenchmarkMixedEffectsAPOE | 0.822 | 0.749 | 4.750 | 4.75000 | 0.360 | 0.56640 | 0.566 | 0.400 |

19.0 | Tohka-Ciszek-RandomForestLin | 0.875 | 0.796 | 6.030 | 6.03000 | 0.150 | 0.56010 | 0.560 | 0.370 |

20.0 | BGU-LSTM | 0.883 | 0.779 | 6.090 | 6.12000 | 0.390 | 0.60250 | 0.600 | 0.230 |

21.0 | DIKU-GeneralisedLog-Custom | 0.878 | 0.790 | 5.400 | 5.40000 | 0.260 | 1.05350 | 1.054 | 0.050 |

22.0 | DIKU-GeneralisedLog-Std | 0.877 | 0.790 | 5.400 | 5.40000 | 0.260 | 1.05350 | 1.054 | 0.050 |

23.0 | CyberBrains | 0.823 | 0.747 | 5.160 | 5.16000 | 0.240 | 0.62250 | 0.623 | 0.120 |

24.0 | AlgosForGood | 0.847 | 0.810 | 5.460 | 5.11000 | 0.130 | 0.69390 | 3.310 | 0.190 |

25.0 | lmaUCL-halfD1 | 0.845 | 0.753 | 6.530 | 6.51000 | 0.310 | 0.44020 | 0.421 | 0.130 |

26.0 | BGU-RF | 0.838 | 0.673 | 6.330 | 6.10000 | 0.350 | 0.50170 | 0.382 | 0.260 |

27.0 | Mayo-BAI-ASU | 0.691 | 0.624 | 4.980 | 4.98000 | 0.320 | 0.51710 | 0.517 | 0.400 |

28.0 | BGU-RFFIX | 0.831 | 0.673 | 6.330 | 6.10000 | 0.350 | 0.50170 | 0.382 | 0.260 |

29.0 | FortuneTellerFish-Control | 0.834 | 0.692 | 4.700 | 4.70000 | 0.220 | 1.37860 | 1.379 | 0.500 |

30.0 | GlassFrog-LCMEM-HDR | 0.902 | 0.825 | 6.340 | 6.21000 | 0.470 | 1.65760 | 1.593 | 0.410 |

31.0 | SBIA | 0.776 | 0.721 | 7.100 | 7.38000 | 0.400 | 0.44100 | 0.311 | 0.130 |

32.0 | Chen-MCW-Stratify | 0.848 | 0.783 | 6.480 | 6.24000 | 0.230 | 1.01010 | 1.004 | 0.110 |

33.0 | Rocket | 0.680 | 0.519 | 5.810 | 5.71000 | 0.340 | 0.64420 | 0.644 | 0.290 |

34.5 | Chen-MCW-Std | 0.836 | 0.778 | 6.480 | 6.24000 | 0.230 | 1.01010 | 1.004 | 0.110 |

34.5 | BenchmarkSVM | 0.836 | 0.764 | 6.820 | 6.82000 | 0.420 | 0.85700 | 0.842 | 0.500 |

36.0 | DIKU-ModifiedMri-Custom | 0.807 | 0.670 | 6.440 | 6.44000 | 0.270 | 0.92360 | 0.924 | 0.010 |

37.0 | DIKU-ModifiedMri-Std | 0.806 | 0.670 | 6.440 | 6.44000 | 0.270 | 0.92360 | 0.924 | 0.010 |

38.0 | DIVE | 0.708 | 0.568 | 7.100 | 7.10000 | 0.340 | 0.48890 | 0.489 | 0.130 |

39.0 | ITESMCEM | 0.680 | 0.657 | 6.260 | 6.26000 | 0.350 | 0.91870 | 0.919 | 0.430 |

40.0 | BenchmarkLastVisit | 0.774 | 0.792 | 7.050 | 7.05000 | 0.450 | 0.62560 | 0.611 | 0.470 |

41.0 | Sunshine-Conservative | 0.845 | 0.816 | 7.900 | 7.90000 | 0.500 | 1.11700 | 1.117 | 0.500 |

42.0 | BravoLab | 0.771 | 0.682 | 8.220 | 8.22000 | 0.490 | 0.58100 | 0.581 | 0.410 |

43.0 | DIKU-ModifiedLog-Custom | 0.807 | 0.670 | 6.440 | 6.44000 | 0.270 | 1.17450 | 1.175 | 0.060 |

44.0 | DIKU-ModifiedLog-Std | 0.806 | 0.670 | 6.440 | 6.44000 | 0.270 | 1.17450 | 1.175 | 0.060 |

45.0 | Sunshine-Std | 0.825 | 0.771 | 7.900 | 7.90000 | 0.500 | 1.11700 | 1.117 | 0.500 |

46.0 | Billabong-UniAV45 | 0.720 | 0.616 | 9.220 | 8.82000 | 0.290 | 1.09160 | 0.994 | 0.450 |

47.0 | Billabong-Uni | 0.718 | 0.622 | 9.220 | 8.82000 | 0.290 | 1.09160 | 0.994 | 0.450 |

48.0 | ATRI-Biostat-JMM | 0.779 | 0.710 | 12.880 | 69.62000 | 0.350 | 1.94860 | 5.118 | 0.330 |

49.0 | Billabong-Multi | 0.541 | 0.556 | 27.010 | 19.90000 | 0.460 | 1.07010 | 1.070 | 0.450 |

50.0 | ATRI-Biostat-MA | 0.741 | 0.671 | 12.880 | 11.32000 | 0.190 | 1.84430 | 5.274 | 0.230 |

51.0 | BIGS2 | 0.455 | 0.488 | 11.620 | 14.65000 | 0.500 | 1.19950 | 1.118 | 0.070 |

52.0 | Billabong-MultiAV45 | 0.527 | 0.530 | 28.450 | 21.22000 | 0.470 | 1.12940 | 1.075 | 0.470 |

53.0 | ATRI-Biostat-LTJMM | 0.636 | 0.563 | 16.070 | 74.65000 | 0.330 | 1.80460 | 5.010 | 0.260 |

- | Threedays | 0.921 | 0.823 | - | - | - | - | - | - |

- | ARAMIS-Pascal | 0.876 | 0.850 | - | - | - | - | - | - |

- | IBM-OZ-Res | 0.868 | 0.766 | - | - | - | 1.14530 | 1.145 | 0.500 |

- | Orange | 0.774 | 0.792 | - | - | - | - | - | - |

- | SMALLHEADS-NeuralNet | 0.737 | 0.605 | 13.870 | 13.87000 | 0.410 | - | - | - |

- | SMALLHEADS-LinMixedEffects | - | - | 8.090 | 7.94000 | 0.040 | - | - | - |

- | Tohka-Ciszek-SMNSR | - | - | 5.870 | 5.87000 | 0.140 | - | - | - |

The results on the D2 dataset suggest that we do not have a clear winner on all categories. While Frog had the best overall submission with the lowest sum of ranks, for each performance metric individually we had different winners: Frog (clinical diagnosis MAUC of 0.931), ARAMIS-Pascal (clinical diagnosis BCA of 0.850), FortuneTellerFish-Control (ADAS MAE and WES of 4.7), VikingAI-Sigmoid (ADAS CPA of 0.02), EMC1-Std/EMC1-Custom (ventricle MAE of 0.41 and ventricle WES or 0.29), and DIKU-ModifiedMri-Std/ DIKU-ModifiedMri-Custom (ventricle CPA of 0.01).

#### Overall scores — cross-sectional dataset D3

RANK |
TEAM NAME | MAUC | BCA | ADAS MAE | ADAS WES | ADAS CPA | VENTS MAE | VENTS WES | VENTS CPA |
---|---|---|---|---|---|---|---|---|---|

1.0 | GlassFrog-Average | 0.897 | 0.826 | 5.860 | 5.57000 | 0.250 | 0.67800 | 0.548 | 0.240 |

2.0 | GlassFrog-LCMEM-HDR | 0.897 | 0.826 | 6.570 | 6.56000 | 0.340 | 0.47870 | 0.377 | 0.240 |

3.0 | GlassFrog-SM | 0.897 | 0.826 | 5.770 | 5.77000 | 0.190 | 0.82390 | 0.551 | 0.070 |

4.0 | Tohka-Ciszek-RandomForestLin | 0.865 | 0.786 | 4.920 | 4.92000 | 0.100 | 0.83400 | 0.834 | 0.350 |

7.0 | VikingAI-Logistic | 0.876 | 0.768 | 5.940 | 5.91000 | 0.220 | 1.04070 | 1.011 | 0.180 |

7.0 | Rocket | 0.865 | 0.771 | 5.270 | 5.14000 | 0.390 | 1.05590 | 1.056 | 0.270 |

7.0 | lmaUCL-Std | 0.854 | 0.698 | 6.950 | 6.93000 | 0.050 | 0.81320 | 0.814 | 0.220 |

7.0 | lmaUCL-Covariates | 0.854 | 0.698 | 6.950 | 6.93000 | 0.050 | 0.81320 | 0.814 | 0.220 |

7.0 | lmaUCL-halfD1 | 0.854 | 0.698 | 6.950 | 6.93000 | 0.050 | 0.81320 | 0.814 | 0.220 |

10.0 | EMC1-Std | 0.705 | 0.567 | 6.290 | 6.19000 | 0.470 | 0.79690 | 0.616 | 0.480 |

11.0 | SBIA | 0.779 | 0.782 | 6.630 | 6.43000 | 0.400 | 0.82340 | 0.748 | 0.180 |

13.0 | BGU-LSTM | 0.877 | 0.776 | 6.750 | 6.17000 | 0.390 | 1.11450 | 0.786 | 0.170 |

13.0 | BGU-RFFIX | 0.877 | 0.776 | 6.750 | 6.17000 | 0.390 | 1.11450 | 0.786 | 0.170 |

13.0 | BGU-RF | 0.877 | 0.776 | 6.750 | 6.17000 | 0.390 | 1.11450 | 0.786 | 0.170 |

15.0 | BravoLab | 0.813 | 0.730 | 8.020 | 8.02000 | 0.470 | 0.63980 | 0.640 | 0.420 |

16.5 | BORREGOTECMTY | 0.852 | 0.748 | 6.440 | 5.86000 | 0.460 | 1.13690 | 1.016 | 0.490 |

16.5 | CyberBrains | 0.830 | 0.755 | 4.720 | 4.72000 | 0.210 | 1.53550 | 1.536 | 0.500 |

18.0 | ATRI-Biostat-MA | 0.799 | 0.772 | 7.390 | 6.63000 | 0.040 | 0.92770 | 0.969 | 0.100 |

19.5 | EMC-EB | 0.869 | 0.765 | 7.710 | 7.91000 | 0.500 | 1.02960 | 1.070 | 0.490 |

19.5 | DIKU-GeneralisedLog-Std | 0.798 | 0.684 | 6.990 | 6.99000 | 0.170 | 0.95410 | 0.954 | 0.050 |

21.0 | DIKU-GeneralisedLog-Custom | 0.798 | 0.681 | 6.990 | 6.99000 | 0.170 | 0.95410 | 0.954 | 0.050 |

22.5 | DIKU-ModifiedLog-Std | 0.798 | 0.688 | 7.100 | 7.10000 | 0.170 | 0.95030 | 0.950 | 0.050 |

22.5 | DIKU-ModifiedMri-Std | 0.798 | 0.688 | 7.100 | 7.10000 | 0.170 | 0.95030 | 0.950 | 0.050 |

24.5 | DIKU-ModifiedLog-Custom | 0.798 | 0.691 | 7.100 | 7.10000 | 0.170 | 0.95030 | 0.950 | 0.050 |

24.5 | DIKU-ModifiedMri-Custom | 0.798 | 0.691 | 7.100 | 7.10000 | 0.170 | 0.95030 | 0.950 | 0.050 |

26.0 | Billabong-Uni | 0.704 | 0.626 | 6.690 | 6.69000 | 0.380 | 0.98340 | 0.983 | 0.480 |

27.0 | Billabong-UniAV45 | 0.703 | 0.620 | 6.690 | 6.69000 | 0.380 | 0.98340 | 0.983 | 0.480 |

28.0 | ATRI-Biostat-JMM | 0.794 | 0.781 | 8.450 | 8.12000 | 0.340 | 0.96580 | 1.450 | 0.370 |

29.0 | CBIL | 0.847 | 0.780 | 10.990 | 11.65000 | 0.490 | 1.12070 | 1.121 | 0.390 |

30.0 | BenchmarkLastVisit | 0.785 | 0.771 | 6.970 | 7.07000 | 0.420 | 1.17360 | 0.635 | 0.110 |

31.0 | Billabong-MultiAV45 | 0.682 | 0.603 | 9.300 | 9.30000 | 0.430 | 1.09140 | 1.091 | 0.490 |

32.0 | Billabong-Multi | 0.681 | 0.605 | 9.300 | 9.30000 | 0.430 | 1.09140 | 1.091 | 0.490 |

33.0 | ATRI-Biostat-LTJMM | 0.732 | 0.675 | 12.740 | 63.98000 | 0.370 | 1.16570 | 1.073 | 0.400 |

34.0 | BenchmarkSVM | 0.494 | 0.490 | 10.010 | 10.01000 | 0.420 | 1.15450 | 1.178 | 0.500 |

35.0 | DIVE | 0.512 | 0.498 | 16.660 | 16.74000 | 0.410 | 1.41590 | 1.416 | 0.340 |

- | IBM-OZ-Res | 0.905 | 0.830 | - | - | - | 1.76790 | 1.768 | 0.500 |

Here, most submissions have worse performance compared to the equivalent predictions on the D2 longitudinal dataset, due to the lack of longitudinal, multimodal data. GlassFrog-Average had the best overall rank and obtained a diagnosis MAUC of 0.897, ADAS MAE of 5.86 and a Ventricle MAE of 0.68 (% ICV). For diagnosis prediction, IBM-OZ-Res obtained the highest clinical diagnosis scores: MAUC of 0.905 and BCA of 0.830. For ADAS predictions, CyberBrains had the best MAE and WES of 4.72. ATRI-Biostat-MA obtained the best ADAS CPA of 0.04. For Ventricle prediction, GlassFrog-LCMEM-HDR had a MAE of 0.48 (% ICV) and the best WES of 0.38, while the 6 DIKU submissions obtained the best CPA of 0.05.

#### Additional entries

In addition to the standard predictions and the benchmarks, we also included two consensus predictions by taking the mean (ConsensusMean) and median (ConsensusMedian) over all predictions from all participants. The ConsensusMedian submission got particularly strong results in diagnosis prediction, obtaining the second-best MAUC of 0.925. On the other hand, ConsensusMean ranked lower on the D2 prediction task. In order to test whether the best results have been obtained because the algorithms are genuinely good at predictions and not by chance, we also evaluated 62 (as many as the number of D2 entries) randomly perturbed predictions from the simplest benchmark, BenchmarkLastVisit, and computed the best results obtained by any of these predictions. These are shown as RandomisedBest, and obtain high scores especially for ADAS, ranking 2nd with a final MAE of 4.52. High performance scores are also obtained for Ventricles, ranking 14 with an MAE of 0.47, a 14% increase in error from the best prediction, while for diagnosis prediction a lower MAUC score of 0.797 is obtained, ranking 43rd. This suggests that the entries scoring below the scores of RandomisedBest should be interpreted with care, as the scores and ranks could be high due to randomness in the test set. This is particularly relevant for ADAS predictions.

It is worth mentioning that, while drafting the manuscript, we discovered that dropping APOE as a covariate in the BenchmarkMixedEffectsAPOE model considerably decreases the error in ADAS prediction, so we included it as an additional entry for scientific interest.

##### Additional entries for D2

RANK |
TEAM NAME | MAUC | BCA | ADAS MAE | ADAS WES | ADAS CPA | VENTS MAE | VENTS WES | VENTS CPA |
---|---|---|---|---|---|---|---|---|---|

1.0 | ConsensusMedian | 0.925 | 0.857 | 5.120 | 5.01000 | 0.280 | 0.38260 | 0.329 | 0.090 |

2.0 | BenchmarkMixedEffects | 0.846 | 0.706 | 4.190 | 4.19000 | 0.310 | 0.56180 | 0.562 | 0.500 |

3.0 | RandomisedBest | 0.797 | 0.803 | 4.520 | 4.52000 | 0.270 | 0.46700 | 0.449 | 0.330 |

4.0 | ConsensusMean | 0.920 | 0.835 | 3.747 | 3.54000 | 0.002 | 0.47600 | 0.453 | 0.126 |

##### Additional entries for D3

RANK |
TEAM NAME | MAUC | BCA | ADAS MAE | ADAS WES | ADAS CPA | VENTS MAE | VENTS WES | VENTS CPA |
---|---|---|---|---|---|---|---|---|---|

1.0 | ConsensusMean | 0.917 | 0.821 | 4.580 | 4.34000 | 0.120 | 0.72630 | 0.722 | 0.090 |

2.0 | ConsensusMedian | 0.905 | 0.817 | 5.440 | 5.37000 | 0.190 | 0.70720 | 0.648 | 0.100 |

3.0 | BenchmarkMixedEffects | 0.839 | 0.728 | 4.230 | 4.23000 | 0.340 | 1.13300 | 1.133 | 0.500 |

#### Confidence Intervals

Below are confidence intervals (CIs) computed for every submission, based on 50 bootstraps of the test set D4. The first figure (Fig. 1) shows CIs based on forecasts from D2, while the second (Fig. 2) shows CIs for forecasts on D3.

Fig 1. Confidence intervals for forecasts based on the longitudinal D2 prediction set.

Fig 2. Confidence intervals for forecasts based on the cross-sectional D3 prediction set.

#### Meta-analysis

To understand which types of features and algorithms yielded higher performance, we show here associations between predictive performance and feature selection methods, different types of features, methods for data imputation, and methods for forecasting of target variables (diagnosis, ADAS and ventricles). For each type of feature/method and each target variable (clinical diagnosis, ADAS and Ventricles), we show the distribution of estimated coefficients from a general linear model, derived from the approximated inverse hessian matrix at the maximum likelihood estimator. From this analysis we removed outliers, defined as submissions with ADAS MAE higher than 10 and Ventricle MAE higher than 0.15 (%ICV). For all plots, distributions to the right of the gray dashed vertical line are associated with better performance.

The results in Fig. 3 below show trends that indicate what aspects of the methods could be associated with better performance. For feature selection, methods that perform manual selection of features are associated with better predictive performance in ADAS13 and Ventricles. In terms of feature types, including features from many modalities was generally associated with an increase in overall performance, except for FDG (for all target variables). Moreover, augmented features correlate with overall performance improvements especially for ventricle prediction. In terms of data imputation methods, while some differences can be observed, no clear conclusions can be drawn currently. In terms of prediction models, we notice that neural networks are more significantly associated with increased performance in ventricle prediction, while disease progression models are associated with decreased performance in prediction or clinical diagnosis and ventricles. However, given the small number of methods tested (<50) and the large number of degrees of freedom (n=21), these results should be interpreted with care.

Fig 3. Associations between the prediction of clinical diagnosis, ADAS and Ventricle volume and different strategies of (top) feature selection, (upper-middle) types of features, (lower-middle) data imputation strategies and (bottom) prediction methods for the target variables. For each type of feature/method (rows) and each target variable (columns), we show the distribution of estimated coefficients from a general linear model. Positive coefficients, where distributions lie to the right of the dashed vertical line, indicate better performance than baseline (vertical dashed line). For ADAS and Ventricle prediction, we flipped the sign of the coefficients, to consistently show better performance to the right of the vertical line.

#### Demographics of D1-D4 datasets

Summary of TADPOLE datasets D1-D4. Each subject has been allocated to either Control, MCI or AD group based on diagnosis at the first available visit within each dataset. The bottom table contains the number of visits with data available, by modality. For example, in D4 there were a total of 150 visits where an MRI scan was undertaken, which represented a total of 64% of all visits analysed across all subjects in D4.

Measure |
D1 |
D2 |
D3 |
D4 |

Cognitively Normal |
||||

Subjects | 1667 | 896 | 896 | 219 |

Number (%) | 508 (30.5%) | 369 (41.2%) | 299 (33.4%) | 94 (42.9%) |

Visits per subject | 8.3 (4.5) | 8.5 (4.9) | 1.0 (0.0) | 1.0 (0.2) |

Age | 74.3 (5.8) | 73.6 (5.7) | 72.3 (6.2) | 78.4 (7.0) |

Gender (% male) | 48.6% | 47.2% | 43.5% | 47.9% |

MMSE | 29.1 (1.1) | 29.0 (1.2) | 28.9 (1.4) | 29.1 (1.1) |

Converters | 18 (3.5%) | 9 (2.4%) | ||

Mild Cognitive Impairment |
||||

Number (%) | 841 (50.4%) | 458 (51.1%) | 269 (30.0%) | 90 (41.1%) |

Visits per subject | 8.2 (3.7) | 9.1 (3.6) | 1.0 (0.0) | 1.1 (0.3) |

Age | 73.0 (7.5) | 71.6 (7.2) | 71.9 (7.1) | 79.4 (7.0) |

Gender (% male) | 59.3% | 56.3% | 58.0% | 64.4% |

MMSE | 27.6 (1.8) | 28.0 (1.7) | 27.6 (2.2) | 28.1 (2.1) |

Converters | 117 (13.9%) | 37 (8.1%) | 9 (10.0%) | |

Alzheimer’s Disease |
||||

Number (%) | 318 (19.1%) | 69 (7.7%) | 136 (15.2%) | 29 (13.2%) |

Visits per subject | 4.9 (1.6) | 5.2 (2.6) | 1.0 (0.0) | 1.1 (0.3) |

Age | 74.8 (7.7) | 75.1 (8.4) | 72.8 (7.1) | 82.2 (7.6) |

Gender (% male) | 55.3% | 68.1% | 55.9% | 51.7% |

MMSE | 23.3 (2.0) | 23.1 (2.0) | 20.5 (5.9) | 19.4 (7.2) |

Converters | 9 (31.0%) | |||

Number of visits with available data (as % of total visits) |
||||

Cognitive | 8862 (69.9%) | 5218 (68.1%) | 753 (84.0%) | 223 (95.3%) |

MRI | 7884 (62.2%) | 4497 (58.7%) | 224 (25.0%) | 150 (64.1%) |

FDG | 2119 (16.7%) | 1544 (20.2%) | 0 (0.0%) | 0 (0.0%) |

AV45 | 2098 (16.6%) | 1758 (23.0%) | 0 (0.0%) | 0 (0.0%) |

AV1451 | 89 (0.7%) | 89 (1.2%) | 0 (0.0%) | 0 (0.0%) |

DTI | 779 (6.1%) | 636 (8.3%) | 0 (0.0%) | 0 (0.0%) |

CSF | 2347 (18.5%) | 1458 (19.0%) | 0 (0.0%) | 0 (0.0%) |

### Description of Algorithms

##### Summary

We had a total of 33 participating teams, who submitted a total of 58 forecasts from D2, 34 forecasts from D3, and 6 forecasts from custom prediction sets. A total of 8 D2/D3 submissions from 6 teams did not have predictions for all three target variables, so we only computed the performance metrics for the available target variables. Another 3 submissions lacked confidence intervals for either ADAS or ventricle volume, which we imputed using default low-width confidence ranges of 2 for ADAS and 0.002 for Ventricles/ICV.

Table 1 below summarizes the methods used in the submissions in terms of feature selection, handling of missing data, predictive models for clinical diagnosis and ADAS/Ventricles biomarkers, as well as training and prediction times. A detailed description of each submission method can be found here.

Submission | Feature selection | Number of features | Missing data imputation | Diagnosis prediction | ADAS/Vent. Prediction | Training time | Prediction time (one subject) |
---|---|---|---|---|---|---|---|

AlgosForGood | Manual | 16+5* | forward-filling | Aalen model | linear regression | 1 minute | 1 second |

Apocalypse | Manual | 16 | population average | SVM | linear regression | 40 minutes | 3 minutes |

ARAMIS-Pascal | Manual | 20 | population average | Aalen model | - | 16 seconds | 0.02 seconds |

ATRI-Biostat-JMM | automatic | 15 | random forest | random forest | linear mixed effects model | 2 days | 1 second |

ATRI-Biostat-LTJMM | automatic | 15 | random forest | random forest | DPM | 2 days | 1 second |

ATRI-Biostat-MA | automatic | 15 | random forest | random forest | DPM + linear mixed effects model | 2 days | 1 second |

BGU-LSTM | automatic | 67 | none | feed-forward NN | LSTM | 1 day | milliseconds |

BGU-RF/ BGU-RFFIX | automatic | ~67+1340* | none | semi-temporal RF | semi-temporal RF | a few minutes | milliseconds |

BIGS2 | automatic | all | Iterative Soft-Thresholded SVD | RF | linear regression | 2.2 seconds | 0.001 seconds |

Billabong (all) | Manual | 15-16 | linear regression | linear scale | non-parametric SM | 7 hours | 0.13 seconds |

BORREGOSTECMTY | automatic | ~100 + 400* | nearest-neighbour | regression ensemble | ensemble of regression + hazard models | 18 hours | 0.001 seconds |

BravoLab | automatic | 25 | hot deck | LSTM | LSTM | 1 hour | a few seconds |

CBIL | Manual | 21 | linear interpolation | LSTM | LSTM | 1 hour | one minute |

Chen-MCW | Manual | 9 | none | linear regression | DPM | 4 hours | < 1 hour |

CN2L-NeuralNetwork | automatic | all | forward-filling | RNN | RNN | 24 hours | a few seconds |

CN2L-RandomForest | Manual | >200 | forward-filling | RF | RF | 15 minutes | < 1 minute |

CN2L-Average | automatic | all | forward-filling | RNN/RF | RNN/RF | 24 hours | < 1 minute |

CyberBrains | Manual | 5 | population average | linear regression | linear regression | 20 seconds | 20 seconds |

DIKU (all) | semi-automatic | 18 | none | Bayesian classifier/LDA + DPM | DPM | 290 seconds | 0.025 seconds |

DIVE | Manual | 13 | none | KDE+DPM | DPM | 20 minutes | 0.06 seconds |

EMC1 | automatic | 250 | nearest neighbour | DPM + 2D spline + SVM | DPM + 2D spline | 80 minutes | a few seconds |

EMC-EB | automatic | 200-338 | nearest-neighbour | SVM classifier | SVM regressor | 20 seconds | a few seconds |

FortuneTellerFish-Control | Manual | 19 | nearest neighbour | multiclass ECOC SVM | linear mixed effects model | 1 minute | < 1 second |

FortuneTellerFish-SuStaIn | Manual | 19 | nearest neighbour | multiclass ECOC SVM + DPM | linear mixed effects model + DPM | 5 hours | < 1 second |

Frog | automatic | ~70+420* | none | gradient boosting | gradient boosting | 1 hour | - |

GlassFrog-LCMEM-HDR | semi-automatic | all | forward-fill | multi-state model | DPM + regression | 15 minutes | 2 minutes |

GlassFrog-SM | Manual | 7 | linear model | multi-state model | parametric SM | 93 seconds | 0.1 seconds |

GlassFrog-Average | semi-automatic | all | forward-fill/linear | multi-state model | DPM + SM + regression | 15 minutes | 2 minutes |

IBM-OZ-Res | Manual | 10-15 | filled with zero | stochastic gradient boosting | stochastic gradient boosting | 20 minutes | 0.1 seconds |

ITESMCEM | Manual | 48 | mean of previous values | RF | LASSO + Bayesian ridge regression | 20 minutes | 0.3 seconds |

lmaUCL (all) | Manual | 5 | regression | multi-task learning | multi-task learning | 2 hours | milliseconds |

Mayo-BAI-ASU | Manual | 15 | population average | linear mixed effects model | linear mixed effects model | 20 minutes | 1.3 seconds |

Orange | Manual | 17 | none | clinician’s decision tree | clinician’s decision tree | none | 0.2 seconds |

Rocket | manual | 6 | median of diagnostic group | linear mixed effects model | DPM | 5 minutes | 0.3 seconds |

SBIA | Manual | 30-70 | dropped visits with missing data | SVM + density estimator | linear mixed effects model | 1 minute | a few seconds |

SPMC-Plymouth (all) | Automatic | 20 | none | ? | - | ? | 1 minute |

SmallHeads-NeuralNetwork | automatic | 376 | nearest neighbour | deep fully -connected NN | deep fully -connected NN | 40 minutes | 0.06 seconds |

SmallHeads-LinMixedEffects | automatic | ? | nearest neighbour | - | linear mixed effects model | 25 minutes | 0.13 seconds |

Sunshine (all) | semi-automatic | 6 | population average | SVM | linear model | 30 minutes | < 1 minute |

Threedays | Manual | 16 | none | RF | - | 1 minute | 3 seconds |

Tohka-Ciszek-SMNSR | Manual | ~32 | nearest neighbour | - | SMNSR | several hours | a few seconds |

Tohka-Ciszek-RandomForestLin | Manual | ~32 | mean patient value | RF | linear model | a few minutes | a few seconds |

VikingAI (all) | Manual | 10 | none | DPM + ordered logit model | DPM | 10 hours | 8 seconds |

BenchmaskLastVisit | None | 3 | none | constant model | constant model | 7 seconds | milliseconds |

BenchmarkMixedEffects | None | 3 | none | Gaussian model | linear mixed effects model | 30 seconds | 0.003 seconds |

BenchmarkMixedEffectsAPOE | None | 4 | none | Gaussian model | linear mixed effects model | 30 seconds | 0.003 seconds |

BenchmarkSVM | Manual | 6 | mean of previous values | SVM | support vector regressor (SVR) | 20 seconds | 0.001 seconds |

Table 1. Summary of methods used in the TADPOLE submissions. Keywords: SVM – Support Vector Machine, RF – random forest, LSTM – long short-term memory network, NN – neural network, RNN – recurrent neural network, SMNSR - Sparse Multimodal Neighbourhood Search Regression, DPM – disease progression model, KDE – kernel density estimation, LDA – linear discriminant analysis, SM – slope model, ECOC - error-correcting output codes, SVD – singular value decomposition (*) Augmented features

### Participant statistics

#### Locations of participating teams

#### Team categories

#### Prediction methods

Organised by:

Prize sponsors: