Research Areas
Medical Diagnostic / Prognostic Systems
Medical Imaging

Visual Search for Simple Non-rigid Multi-feature Objects in Histologic images

Key People


Research Areas

Medical Diagnostic/Prognostic Systems
Christos N. Schizas, Constantinos S. Pattichis, Frank Schnorrenberg, and L.T. Middleton, K. Kyriakou, A. Drousiotou, K. Christodoulou (Cyprus Institute of Neurology and Genetics), A. Adamou, M. Vassiliou (Nicosia General Hospital), S. Markidou (St. Savvas Hospital, University of Athens), S. Kollias, A. Stafylopatis (National Technical University of Athens), C.A. Bonsett and J.C. Kincaid (Indiana University School of Medicine).

As medicine becomes more specialised and more complex, and the technology succeeds in offering more possibilities in gathering related medical data, doctors are faced with the challenge of processing vast amounts of data. Artificial Intelligence (AI) was introduced in this field for providing solutions or as an alternative methodology since it is claimed that one of the major research areas in AI is the modelling of human problem solving and decision making. Some of the components of this area that are related to medicine are: The acquisition of medical knowledge; the problem-solving system; a decision making strategy by capturing the abilities of the expert doctors; a means of adding new knowledge and modifying previous knowledge; and a good and friendly user interface with the system.

The breadth of clinical knowledge is an impediment to the development of symbolic knowledge bases that are comprehensive and flexible enough to cope with reality. It has been lately demonstrated that neural networks, genetic algorithms, and fuzzy systems can offer new hope for allowing computers to assist in the challenging and expensive process of medical diagnosis. What turns these new propositions into promising tools are their natural “fault-tolerant” properties, and their capability of finding near-optimum solutions from limited or incomplete data. In this context, new propositions are introduced as tools for building intelligent diagnostic systems. Such systems do not mean to replace the physician from being the decision-maker but, rather, they attempt to enhance ones abilities to reach a correct decision.

Recent advances in computer technology offer to the medical profession specialised tools for gathering medical data, processing power, as well as fast storing and retrieving capabilities. In this research, artificial neural networks (ANN) are introduced as a tool for building an intelligent diagnostic system; the system does not attempt to replace the physician from being the decision-maker but to enhance ones abilities to reach a correct decision. An integrated diagnostic system for assessing certain neuromuscular disorders is used in this study as a convenient vehicle for demonstrating the proposed methodology. The diagnostic system is composed of modules that independently provide numerical data to the system from the clinical examination of a patient, and various laboratory tests previously performed. The examination procedure has been standardised by developing protocols for each specialised area, in co-operation with experts in the particular area. At the conclusion of the clinical examination and laboratory tests, data in the form of a numerical vector represents a medical examination snapshot of the subject. Artificial neural network (ANN) models are developed using the unsupervised self-organising feature maps algorithm. The pictorial representation of the diagnostic models provides the physician with a friendly human-computer interface and a comprehensive tool that can be used for further observations, e.g., for monitoring disease progression.

This line of research has been applied in various problem settings, including the diagnosis of neuromuscular disorders, the assessment of the prognostic factors for breast cancer patients, electromyography, histological assessment, and medical feature extraction and classification.

Medical Imaging - Computer-aided classification of breast cancer nuclei
Constantinos S. Pattichis, Christos N. Schizas, Frank Schnorrenberg, K. Kyriakou (The Cyprus Institute of Neurology and Genetics) and M. Vassiliou (Nicosia General Hospital).

The evaluation of immunocytochemically stained histopathological sections presents a complex problem due to many variations that are inherent in the methodology. In this respect, many aspects of immunocytochemistry remain unresolved, despite the fact that immunocytochemical results may carry important diagnostic, prognostic, and therapeutic information. In this study a modular neural network based approach to the detection and classification of breast cancer nuclei stained for steroid receptors in histopathological sections is described and evaluated. The system, named Biopsy Analysis Support System (BASS), was designed so that it simulates closely the assessment procedures as practised by histopathologists.

The overall system is based on a modular architecture where the detection and classification stages are independent. The system used two different methods for the detection of nuclei: the one approach is based on a feedforward neural network (FNN) which uses a block–based singular value decomposition (SVD) of the image to signal the likelihood of occurrence of nuclei. The other approach consists of a combination of a receptive field filter and a squashing function (RFS), adapting to local image statistics to decide on the presence of nuclei at any particular image location. Both approaches utilise localised operating principles which feature rotational invariance and insensitivity to noise. The classification module of the system is based on a radial basis function neural network.

A total of 57 images captured from 41 biopsy slides containing over 8300 nuclei were individually and independently marked by two experts. A five scale grading system, known as diagnostic index, was used to classify the nuclei staining intensities. The experts’ mutual detection sensitivity (SS) and positive predictive value (PPV) were found to be 79 % and 77 % respectively. The overall joint performance of the FNN and RFS modules was 55 % for SS and 82 % for PPV. A second evaluation by one of the experts carried out on half of the images in the database to assess the accuracy of additional nuclei detected only by the modules resulted in an improvement of SS and PPV values by 1.3 % and 4.7 % for the joint modules, respectively. The classification module correctly classified 76 % of all nuclei in an independent validation set containing 25 images.

This study shows that detection and classification of individual nuclei in histopathological sections can be consistently and reliably performed by a modular neural network based system. Moreover, since the system simulates detection and grading strategies of human experts it will enable the formulation of more efficient standardisation criteria, which will in turn improve the assessment accuracy of immunocytochemically stained histopathological sections.

Visual Search for Simple Non–rigid Multi–feature Objects in Histologic images
Frank Schnorrenberg, Christos N. Schizas

The exploration into principles of visual search and object recognition has been going on for decades. Visual search and object recognition are a part of the basic skills of humans and many animals. They are rapidly and accurately performed. However, despite progress in the neurosciences, psychology, and engineering, the performance of biological visual systems could not be matched, nor sufficiently explained by artificial implementations and models. In particular, tasks that traditionally require human expertise and human visual skills, such as tissue analysis in microscopy, are still essentially conducted manually. The objective of the dissertation is to introduce and test a novel approach to the evaluation of object populations in cluttered scenes. In particular, methods are developed that can aid in the efficient detection, recognition, and classification of simple non–rigid multi–feature objects, such as cell nuclei in biopsy images.

Among the conceptual and computational problems addressed are object detection, scale detection, and object classification. Object detection is investigated in the context of missing or inaccurate performance measures for segmentation. Scale detection is an ability humans perform effortless and which would be desirable for machine vision applications. In an experiment to generate supervisory information regarding images, human experts may be subject to inter-and intra observer variations. As a result object classification may have to be conducted with inconsistent information.

Object detection: In much of the literature, accurate segmentation is the pre-requisite for object recognition and classification. A problem emerges if either there is no useful performance measure for accurate segmentation or accurate segmentation is not required.

Scale detection: Humans have usually no difficulty to indicate the approximate spatial scale of objects in a visual scene after a short glance. In fact, humans use this knowledge to guide their subsequent visual analysis. A computer-based system with this skill may similarly influence subsequent processing stages.

Object classification: In an experiment to generate supervisory information regarding images, human experts may be subject to inter-and intra observer variations. In the absence of an absolute standard the supervisory information will generally be inconsistent and, more importantly, there is no justification to remove the one or the other item from the database. Nevertheless, a learning system may have to be trained with the available information.

The above problems are studied in the context of the evaluation of cell nuclei populations in microscopy images of immunohistochemically stained histopathological tissue sections from breast cancer biopsies. The system was designed in a modular fashion: The object detection module propagates the detection results to the classifier module which in turn classifies each detected object, i.e. cell nucleus, into one of a number of classes. Finally, the diagnostic index is calculated based on the detection and classification results.

The implemented system, called the Biopsy Analysis Support System (BASS), is applied to the evaluation of nuclei populations in microscopy images of immunohistochemically stained histopathological tissue sections from breast cancer biopsies. It features a novel object detection method based on receptive fields and a nonlinearity. The detection method, which has similarities to template matching methods, features a parameterisation which is directly relevant to the medical problem, by giving control over spatial scale and separateness of detected objects. Moreover, the nature of the on-centre-off-surround type receptive fields used makes the system very robust with respect to lighting variations. Thus, calibration and standardisation of the images is less critical as regards nuclei detection. The response characteristics of the receptive field were theoretically analysed and it was found that the parameterisation of the receptive field and the squashing function enable the detection of objects of interest depending on their spatial scale, but almost independent of their staining intensity, which is desirable. A second detection method was implemented utilising singular value decomposition of image blocks and a neural network classifier to detect cell nuclei. Both detection methods were combined in a variety of ways and it was shown that combining the subsystems may lead to a an enhanced performance as compared to the individual subsystems.

BASS classifies nuclei according to a five-scale nuclei classification scheme used in manual assessment. Subsequently, the diagnostic index is computed using a well-established formula from manual semi-quantitative assessment procedures. This module of the system is implemented as a radial basis function network. It classified 76 % of all nuclei correctly in an independent validation set. A feedforward neural network was also applied, but did not perform as well. The classification problem was shown to be difficult due to convoluted class boundaries as visualised by a Sammon plot.

Due to the emphasis on nuclei detection instead of accurate segmentation, any number of nuclei detection and segmentation methods may be integrated in the system to improve the joint performance. One consequence is that BASS can be highly interactive or fully automated since human experts may simply supervise the system and correct detection results. A similar argument with respect to interactivity holds for the classification of nuclei.

A novel scale detection algorithm was implemented to estimate the spatial scale of cell nuclei. The receptive field squashing function subsystem may take advantage of these results by adjusting the receptive field according to the detected scale. This paragraph has to be adjusted when final results are known.

BASS can be used as an independent expert, which may directly be compared on a nuclei detection and classification level or on a diagnostic index level with the performance of other systems or human experts or diagnostic indices derived from the laboratory routine procedure. For example, the ability of BASS to accurately assign the diagnostic index was found to be 69 % compared to that of two experts which was 68  and 77 % respectively. In addition, while other computer-based systems had to introduce new measures since they are based on nuclear area measurements, BASS simply computerizes the manual method. Further improvement is not precluded, however, since additional processing on the detected nuclei and correlation with clinical data is possible. Beyond the analysis of biopsy images, BASS’ utility was also demonstrated for telepathology applications and content-based description of breast cancer biopsy slides.

The above described problems were addressed in a histopathological setting using microscopy images. In particular, nuclear stains were analyzed in images of immunohistochemically stained histopathological sections. The system was designed in a modular fashion: The object detection module propagates the detection results to the classifier module which in turn classifies each detected object, i.e. cell nucleus, into one of a number of classes. The system generates the same detection and classification events a human expert generates when analyzing the image. Therefore, the system can be compared at any level to the performance of human experts. Object detection results originate from one subsystem or a collaborative effort of more subsystems. Two object detection subsystems were implemented and combined in a variety of ways. The one subsystem is based on an iterative solution using a receptive field and a squashing function, while the other subsystem is based on a neural network which was trained on a singular value decomposition of overlapping windows in the image containing the cell nuclei. A novel scale selection algorithm was implemented to estimate the spatial scale of cell nuclei. The receptive field squashing function subsystem may take advantage of these results by adjusting the receptive field according to the detected scale. The classification module utilizes a radial basis function classifier, color features, and a graylevel texture feature. While the algorithms employed lend themselves to user interaction at several levels of detail, manual interaction is not required in the implementation except for selecting the initial parameters.

The proposed solution to the problem of implementing visual search for simple non–rigid multi–feature objects on a machine contributes to a better understanding of efficient scene analysis. In addition, the solution has positive implications for existing tedious, but difficult visual evaluation tasks performed by human experts.


Key People
Adamou A. 
Christodoulou K. (Cyprus Institute of Neurology and Genetics)
Drousiotou A. 
Kyriakou K. 
Markidou S. (St. Savvas Hospital, University of Athens)
Middleton L.T. 
Neocleous C.C. (Higher Technical Institute)
Pattichis C. S. (University of Cyprus)
Schizas C.N. (University of Cyprus)
Stafylopatis A. (National Technical University of Athens)
Vassiliou M. (Nicosia General Hospital)
Bonsett C.A. (Indiana University School of Medicine).
Kincaid J.C. (Indiana University School of Medicine).
Kollias S. (National Technical University of Athens)
Schnorrenberg F.