Project Description

Issue and Context

Download the abstract

Since the early 1980’s, many studies have been initiated worldwide to develop advanced systems for the automated detection and follow-up of Diabetic Retinopathy, the first cause of blindness in people between 25 and 65 years old. These systems, based on automatic image processing, essentially consist in tools for detecting and measuring common lesions (such as microaneurysms, exudates, hemorrhages) and in indexing and automated retrieval techniques applied to image databases.

In view of the numerous studies already available, the major issue is now to assess accurately and objectively the results that have been obtained so far. This problem is far from being solved due to the lack of a large and adequate database accessible to the scientific community. The size of the available databases described in literature is clinically inadequate to allow this kind of assessment.

Two methods, for which several algorithms have been developed by the partners of the MESSIDOR project, have been more particularly studied:

1. The image Analysis-Quantization method which implements segmentation algorithms to detect and quantify elementary lesions like microaneurysms, first non equivocal sign of Diabetic Retinopathy, hemorrhages and exudates whose importance and location are a good marker of the seriousness of the retinal disease.
2. The method for Automated Retrieval (from an annotated image database) of the images closest to the retina image under analysis (request image). The images of the database and the request image are indexed by defining signatures.

The main current issue is to create large databases of retina images and to use them in order to evaluate the various existing algorithms.

Databases

The two main databases will contain color images of the retina, acquired using a retinograph with or without pupil dilation during routine clinical examinations. These examinations will be performed in the four ophthalmology departments involved in the program. To make their diagnosis, ophthalmologists are generally using a central picture and two peripheral pictures of the retina. We will proceed in the same way and we will record the three images in the databases. However, during the MESSIDOR project, only the central image will be annotated.

The images will be saved as uncompressed TIFF format with a 1440 * 960 pixel resolution that is about 4 MB per image.

Training set

This database will be used for testing and improving the available algorithms as well as for validating the methods used to evaluate the algorithms. For each image, the following data will be saved:

• The stage of Diabetic Retinopathy.
• The number of and/or the surface of microaneurysms.
• The degree of exudation: the degree is function of the surface that is occupied by exudates and their locations with respect to the center of vision (Macula).
• The level of hemorrhage, which is defined with respect to the number and/or the surface occupied by hemorrhages.

This database will contain about 300 images. Microaneurysms, exudates and hemorrhages will be marked individually on at least fifty of these images.

Evaluation set

This database will contain about a thousand images. Its purpose will be the evaluation of algorithms. All images in the evaluation set will be annotated in the same way as for the training set. On a hundred of them, microaneurysms, exudates and hemorrhages will be marked individually as in the training set.

Testing implementation

To evaluate a method used for automated detection and image interpretation, it is necessary to compare the result that is obtained by this method with a reference that is considered to be the ground truth. This raises two questions:

1. What is the ground truth or the reference?
2. Which measures should be used to perform the evaluation?

How to obtain references?

To determine the number of microaneurysms, the stage of retinopathy, the degree of exudation and the level of hemorrhage, all images of the training and the evaluation sets will be analysed by all the ophthalmology departments involved in the project and annotated with respect to their conclusions.

For the individual marking of microaneurysms, images will be annotated by two specialists in each ophthalmology department. If there is only a slight difference between departments, we will retain the marking proposed by the department that has provided the images; if there is a significant difference, we will try to reach an agreement between the different departments and/or new rules for selecting microaneurysms will be proposed.

The individual marking of exudates and hemorrhages is not a real problem as there are less ambiguous cases as with microaneurysms. Therefore working towards a consensus between the different departments should be less necessary.

Protocols and metrics

Before adopting a given metrics, we have first to consider the audience to which this evaluation is intended, in other words the people who may be interested in the obtained results. We will first address the medical community. Therefore we must cselect a metrics that is commonly used in the medical field in the evaluation as well as in the presentation of the results. Second, this work is intended to the scientific researchers who develop automatic image processing methods. Then we will have to select the metrics used by these scientists. We propose to compute and describe the results in two different ways: by applying first the performance measurements used by physicians, and second the more detailed statistics commonly used in image processing. The Analysis – Quantization algorithms will be applied to all the images of the evaluation database. We will do the same for the Indexing – Retrieval algorithms for which every image of the evaluation database will serve as request image.

Hence, the algorithms will be evaluated from two points of view:

• Evaluation for the medical community

  It has been decided to classify Diabetic Retinopathy into 6 stages of seriousness, 4 stages of exudation and 3 stages of hemorrhage and to indicate the number of microaneurysms for each image.

  An efficiency indicator will be used, whose implementation remains to be more precisely defined, in order to evaluate performance with respect to medical diagnosis. This efficiency indicator will consist in the percentage of automated diagnoses in agreement with medical diagnoses.

  The Indexing – Retrieval algorithms will be evaluated with respect to the stages of seriousness of Diabetic Retinopathy.

  The Segmentation - Quantization algorithms will be evaluated with respect to the stages of exudation and hemorrhage as well as to the number of microaneurysms.

• Detailed evaluation of algorithms

  In order to evaluate Indexing – Retrieval algorithms, we will use the classical comparison and evaluation criteria applied to retrieval performances (Precision / Recall). These measurements will be done from the annotations concerning the stage of seriousness of Diabetic Retinopathy. And to evaluate Segmentation - Quantization algorithms, we will assess their sensitivity and specificity with respect to the detection of microaneurysms, exudates and hemorrhages that have been marked individually by ophthalmologists.

Expected results

The first expected result is to have a good knowledge of the efficacy, performances, limitations and drawbacks of the algorithms. This should help to:

• Convince ophthalmologists to use automatic methods for diabetic retinopathy evaluation by providing them with quantitative assessments of the efficacy, performance, and potential limitations of available methods.
• Establish a strong collaboration between the Messidor partners to foster the development of practical, supported, software products to be used in tracking-down services, telemedicine, pathology tracking, and as a diagnostic aid in the field of diabetic retinopathy. The successful accomplishment of the Messidor project goals will be a major breakthrough in the field of public health and the treatment of eye disease.

The second expected result is the creation of large databases, which are indispensable to the scientific community which is currently working on retinal images.

Communication and exploitation of results

Communication of scientific results

The scientific results will be communicated through the MESSIDOR internet web site, scientific papers and congress communications.

Exploitation of data and software tools

After the research campaign, the databases will be accessible to the scientific community by signed agreements taking into account any related legal constraints. The value and industrialization of the software tools will be developed according to the rules defined and adopted by all the partners in the exploitation plan which will soon be elaborated. The value of indexing and retrieval algorithms and methods will also be developed in internet teaching applications. This could be another important benefit of MESSIDOR program.