Frequently asked questions

Data

How do I access the data?

You need to sign up for an ADNI account, which you can do here.

Once you have an account, you can find the TADPOLE challenge data sets here by following links to "Download", then "Study Data", then "Test Data", then "Data", and finally "Tadpole Challenge Data". Download the zip file, unzip it, and you will find .csv spreadsheets containing the main challenge data sets as well as associated files explaining the contents.

Can I use the TADPOLE data for other studies?

Yes, you are free to use the data. You should however check and adhere to ADNI's rules on acknowledgements of use of their data in publications. We would also appreciate it if you mention in your work that the publication specifically uses the TADPOLE data sets and acknowledge the EuroPOND project for constructing the data set as follows "This work uses the TADPOLE data sets https://tadpole.grand-challenge.org constructed by the EuroPOND consortium http://europond.eu funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 666992".

What is the meaning of the different columns in the spreadsheets?

There are many columns in the TADPOLE data sets TADPOLE_D1_D2.csv and TADPOLE_D3.csv each containing a different piece of information about subjects and visits. The associated data dictionary TADPOLE_D1_D2_Dict.csv gives a brief explanation of the meaning of each column and more information on each measure can be found on the ADNI website. If you are unsure of the meaning of any particular column, please feel free to ask on the challenge forum. Here are some key ones to understand and ones we've had specific questions about:
- Column DX: This contains the clinical diagnosis. The entries specify both the current diagnosis and the baseline so, for example "MCI to Dementia" means that the current diagnosis is Dementia, while the diagnosis at the previous visit was MCI.
- Column DX_bl: This is the baseline diagnosis from the first visit of this subject.
- Column RID: the unique identifying code for a specific subject.
- Column EXAMDATE: the date as which the subject visiting the clinic and the data was acquired.
- Column ADAS13: the composite ADAS13 cognitive test score.
- Column Ventricles: the ventricle volume.

Does a clinical status of "cognitively normal" mean healthy?

In theory, cognitively normal subjects are included in the ADNI study as healthy controls in the same age group as the patients.  However, in practice a reasonable proportion do have some early signs of AD. These are not clinical signs (they show/report no cognitive deficit), but those subjects may show some evidence of amyloid deposition or brain atrophy.

What do values of -4 or NaN mean in the spreadsheets?

Both values indicate that the values are missing - most likely because the subject did not undergo this particular test during this particular visit or the data were somehow corrupt.

I get an error when running the Python/MATLAB scripts downloaded from ADNI. Is this a problem?

No. The scripts are included only to show how the data are generated. They fail now because the ADNI database has grown since they were written. With a bit of adaptation you can run the scripts to get an extended version of the TADPOLE data sets. However, we recommend at least starting by not worrying about these scripts and just using the spreadsheets TADPOLE_D1_D2.csv and TADPOLE_D3.csv as provided.

I'd like to re-generate the TADPOLE datasets. Where can I find the required spreadsheets: ADNIMERGE.csv, ARM.csv, REGISTRY.csv, etc ... ?

The tables can be found on the Loni website -> Study data. Use the search box to search for the names, otherwise use the menu on the left to navigate to the required sections (Imaging -> MRI image analysis, PET image analysis, Enrollment, etc .. ). Place all these spreadsheets in the folder where the Python/MATLAB scripts are and then follow the instructions in the README and the Makefile. If errors are enountered during the running of the scripts, this can be due to several reasons:

1. The ADNI spreadsheets (e.g. ADNIMERGE.csv) changed in the meantime. We try to update the scripts and make them more robust, so keep an eye on the TADPOLE github repo for any updated version of the scripts.
2. A different version of Python/MATLAB was used for running it. This will require participants to make slight changes to the scripts where necessary in order to make them run on their environment. We have also only tested the scripts on Linux and MacOS, but not on Windows. 

Participants are welcome to extend these datasets with extra features, using the scripts we provided as a starting point. See the latest versions of the scripts on the TADPOLE github repo.

Regarding the specific TADPOLE data sets:

1) Is it correct that patients who appear in D2 also appear in D3?
2) Is it correct that all patients in D2 are also included in D1? Similarly, are all patients in D3 included in D1?
3) Are there some patients in D1 who do not appear in D2 or D3?
4) Will all patients included in D3 also be included in the test set, D4? Similarly, will all patients included in D2 be included in D4?
5) Can subject-specific random effects trained on D1 be used for predicting D2/D3?

1. Yes - it is the same list of subjects in both D2 and D3.
2. Yes - D1 is as complete a set of data as we could compile from the entire ADNI history (no point excluding information from this, as people would find it anyway).
3. Yes. D2 and D3 include only “rollover” subjects - those who have agreed to provide future data whereas D1 additionally includes subjects that are no longer part of the study.
4. No. D4 will include only subjects who provide data between the submission deadline in November and the time of evaluation (early 2019 most likely). We do not yet know which subjects will do that so ask for forecasts for all of them.
5. Yes (D2), No (D3). Information in D1 that originates from the subjects in D3 should not be used in any way for training the model used to produce forecasts for D3. This is because the prediction task for D3 aims to mimic "information typically available when selecting a cohort for a clinical trial", i.e., cross-sectional only.

Metrics

Will TADPOLE evaluate the significance of differences among forecasts? How will this affect prize allocation?

We plan to use a bootstrap procedure to evaluate the variance of forecast performance in a similar way to previous challenges, see section 1.4 of the CADDementia Evaluation page for some detail. Although we we use these results to assess and report significance of performance difference among submissions in scientific publications etc., for prize allocation we won't take this into account and will award prizes based on absolute values of performance metrics - the highest performance wins even if it's by a tiny margin.

Will there be enough converters in one year?

Number of MCI participants that remain stable/convert across ADNI-1,-GO,-2:

Number of CN participants that remain stable/convert across ADNI-1,-GO,-2:

The number of MCI and CN participants in ADNI-1, -GO, and -2 that converted at 1, 3 and 5 years is summarised in the table above. The numbers should be sufficient to ensure that the default no-change forecast is not optimal. Later evaluations, e.g. at 3 years and 5 years, will have stronger statistics on conversion.

Why does TADPOLE ask for 50% confidence intervals?

Accurate assessment of confidence in forecasts can be as important as the forecasts themselves. These confidence intervals are the mechanism by which participants quantify their confidence in each forecast.

90% or 95% confidence intervals might be more familiar, but here we ask for 50% confidence intervals. That means you should expect your best guess forecast to fall within the interval 50% of the time. 50% intervals provide a more symmetric evaluation of over- and under-estimation of the confidence than say a 95% interval, as we get similar sample sizes of data points falling inside and outside the confidence interval.

Why does TADPOLE choose ventricle volume as the MRI measurement?

We want to choose a simple and standard MRI marker of neurodegeneration. In consultation with clinicians we decided on ventricle volume, which showed a similar rate of decline to hippocampal volume and whole-brain volume, but was more consistent longitudinally.

What about participants who progress to develop non-AD dementia?

We plan to exclude such subjects from the evaluation.

Submissions

Which forecasts are required to be eligible for the prize?

The minimum entry to be eligible for the prize must gives forecasts from D2 only.

For a full entry (eligible for coauthorship), do I have to submit results from custom processing and prediction sets?

No. You do not have to provide “custom” forecasts for a full entry, but can if you wish. The requirement is that you provide forecasts that use the standard data sets - both D2 and D3.

How will TADPOLE handle missing values in submitted forecasts?

Missing entries will simply be replaced by the input values (copied from D2 or D3) for the missing value with a default low-confidence 50% confidence interval.

Will there be a prize?

Yes! We have sponsorship from the Alzheimer's Association (USA), The Alzheimer's Society (UK), and Alzheimer's Resarch UK and have a total prize fund of 30K GBP.

We plan to divide that fund into 6 separate prizes, although a single team is able to claim multiple prizes. Those prizes are:

1. £5K prize for best forecast of future clinical status.
2. £5K prize for best forecast of future ADAS13 score.
3. £5K prize for best forecast of future ventricle volume.
4. £5K prize for overall best predictive performance.
5. £5K prize for the best forecast from a university student team.
6. £5K prize for the best forecast from a high-school team.
 

See the Details tab of this website for details.

Tools and features

I've downloaded the data, how do I get started?

A good place to look is the example code that we provide, which demonstrates how to read the data into standard programming environments, like matlab or python, extract important parts, construct a simple forecast, and output it in the required format.

Does TADPOLE plan a dynamic leaderboard?

Yes! We are working on this and a prototype is now available - see Leaderboard tab of the website.
Obviously, currently we can only evaluate forecasts using existing data, e.g., the TADPOLE standard training set D1. However, we will define a nominal training and test set as subsets of D1 simply for preliminary testing.

Future challenges

Will you extend the submission deadline or have separate future submissions?

We do not intend to extend the initial submission deadline. We are particularly reluctant to do so, as the longer we wait to close submissions, the fewer ADNI subjects we will be able to use for evaluation, as ADNI3 is actively acquiring data from patients right now - as soon as they acquire data, they publish it on the website, which invalidates it for evaluation.

We may have future submission deadlines. We plan to evaluate the success of the challenge after compiling the first set of results and decide whether to repeat the procedure. We do however plan to award all of our prize fund to submissions to the first deadline and there is no guarantee we will have a similar prize funds for future iterations of the challenge.  We strongly encourage you to enter this round!

What about evaluating performance on non-ADNI data?

This is something we are strongly considering for future iterations of the TADPOLE challenge but for the first submission we will stick with predicting future ADNI data only.

Organised by:  

Prize sponsors: