Automated detection of hyperreflective foci in the outer nuclear layer of the retina

Hyperreflective foci are poorly understood transient elements seen on optical coherence tomography (OCT) of the retina in both healthy and diseased eyes. Systematic studies may benefit from the development of automated tools that can map and track such foci. The outer nuclear layer (ONL) of the retina is an attractive layer in which to study hyperreflective foci as it has no fixed hyperreflective elements in healthy eyes. In this study, we intended to evaluate whether automated image analysis can identify, quantify and visualize hyperreflective foci in the ONL of the retina.


| I N T RODUC T ION
Originally, the term 'hyperreflective foci' was used to define any hyperreflective lesion, dotted or focal in appearance, when imaged using optical coherence tomography (OCT) at any retinal layer (Fragiotta et al., 2021). The presence of hyperreflective foci was first described in diabetic macular oedema as a morphologic sign of lipid extravasation (Bolz et al., 2009). However, in the past decade, hyperreflective foci have been characterized in a considerable amount of different retinal disorders and with the use of various of morphological entities such as migrating retinal pigment epithelium cells, macrophages/microglia in age-related macular degeneration (AMD) (Curcio et al., 2017;Framme et al., 2010;Pang et al., 2015) and degenerated photoreceptor cells (Torm et al., 2020;Uji et al., 2012). Furthermore, a cell-sized migrating intraretinal hyperreflective element has been observed incidentally in a healthy child (Torm et al., 2020).
Although no consensus about the origin of hyperreflective foci exists (Wang et al., 2011), studies suggest that hyperreflective foci may represent aggregates of activated microglia (S. Vujosevic et al. 2013). In contrast, retinal exudates, microaneurysms and haemorrhages are all classified as subtypes of 'hyperreflective elements' and can be distinguished from the hyperreflective foci phenotype based on specific morphological characteristics (Kodjikian et al., 2019).
Hyperreflective foci are not detectable by fundus photography and their occasional presence in healthy subjects suggests that they are distinct from hard exudates. They are currently being investigated as a clinically accessible in vivo diagnostic and prognostic biomarker of potential interest in a broad spectrum of retinal disorders (Abri Aghdam et al., 2015;Altay et al., 2016;Borrelli et al., 2019;Busch K. et al., 2020;Chen et al., 2016;Fragiotta et al., 2018). The manual mapping and counting of hyperreflective foci are a time-consuming process. Therefore, the development and application of tools to effectively visualize and quantify the presence and flux of hyperreflective foci is of great importance and may assist in the clinical diagnosis and in the monitoring of progression and treatment response.
In this study, we determined whether an image analysis pipeline consisting of a convolutional neural network (CNN) classifier may be used for accurate quantification and visualization of hyperreflective foci in the outer nuclear layer of the retina. The avascular outer nuclear layer of the retina was of particular interest in this study because hyperreflective foci imaged on OCT within this region could be assumed to represent true foci, not vessels.

| M AT ER I A L S A N D M ET HOD S
In this exploratory longitudinal study, we investigated six patients presenting with an ophthalmoscopically normal fundus and one patient with reversible fundus abnormalities (see patient characteristics, Table 1). An ethical waiver for the use of anonymized data from each patient was obtained. The study was conducted in agreement with the Tenets of the Declaration of Helsinki and was approved by the local Ethics Committee. Fourteen eyes of seven patients were evaluated.
In every patient, two to four OCT examinations were performed for each eye, over time (see supplemental material). We acquired in total, 2596 B-scans across all time points and patients. All B-scans were examined by one independent examiner who was blinded to the clinical characteristics of the patients (e.g. age, gender, visual function and diagnosis). The examiner manually labelled hyperreflective foci in the outer nuclear layer, defined as the region between the outer plexiform layer and retinal pigment epithelium (RPE) line. Hyperreflective foci were manually annotated according to the following fundamental features: (i) location within the outer retina; (ii) size ≤30 μm; (iii) absence of a shadow cast by the hyperreflective foci and (iv) reflectivity similar to the retinal nerve fibre layer (RNFL). In the latest international consensus guidelines, these morphological characteristics were incorporated as the diagnostic criteria for hyperreflective foci (Kodjikian et al., 2019;Lee & Chung, 2018;Midena et al., 2018). In total, 391 hyperreflective foci were manually annotated among the 189 B scans originating from nine eyes of seven patients.

| Automatic detection of hyperreflective foci
The hyperreflective foci detection method comprises (1) candidate detection, (2) feature extraction and (3) candidate classification (see Figure 1). Candidate detection uses 'blob' detection (Lindeberg, 1996) and detects likely candidates for hyperreflective foci (blobs) in the image. Feature extraction extracts a set of measurements for each of the candidates detected in the first step. The features are radius and the intensity of the detected hyperreflective foci candidate, proximity to a blood vessel in the retina, and the location within the retinal layer where the hyperreflective focus candidate was detected. We applied layer detection (Li et al., 2016;Haeker et al., 2007) to find the retinal layers. Furthermore, image patches centred around hyperreflective foci candidates were extracted. The third part, feature classification, uses CNN for classification of the extracted patches and the features.
The 2548 B-scans for the classification consisted of positive and negative hyperreflective foci candidates found by blob detection. As we report in the results section, there was a large-class imbalance in this data set with 350 positive and 2198 negative candidates. To alleviate the class imbalance, we oversampled the positive examples.
Data were divided into training (80%) and validation (20%) sets. The CNN was trained on the training data. However, the CNN performance on the on-validation data was slightly biased as we did not split specific images or eyes into a test and training set. When we assessed the classification on the validation set, the CNN may have seen a training example originating from the same patient, eye (volume) or even specific B-scan giving rise to unwanted bias.

| OCT acquisition
The development of robust AI algorithms using supervised learning benefits from large and heterogeneous training data sets. Hence, different retinal disorders were examined in this study and data were acquired from different OCT scanning protocols. All patients underwent OCT imaging with Spectralis OCT machines (Heidelberg Engineering, Heidelberg, Germany, software version 6.0.13). Five of the scan protocols comprised a peripapillary ring scan (circular scans with 15 degrees diameter, Automatic Real-Time [ART] 15-25) and a macular volume scan (scan centred on the fovea with 30 × 15 degrees, 19 vertical B-scans, ART, B-scan distance varies from 239-256 μm). In the last two scan protocols, OCT data originate from peripapillary ring scans (circular scans with 15 degrees' diameter, ART set from 15 to 25) and macular volume scans (scans centred on the fovea with 20 × 20 degrees, 25 vertical B-scans, ART 15-50, B-scan distance varied from 234-250 μm). All scans were performed in high-resolution mode with eye T A B L E 1 Demographical and clinical characteristics of the individual patients are presented (i.e. diagnosis, best corrected visual acuity in Snellen decimal notation, visual field tests and treatment) The right eye was normal. Peripheral field loss in the left eye.

Patient characteristics
Amitriptyline 50 mg Azathioprine 50 mg twice daily F I G U R E 1 Visualization of the method pipeline showing candidate foci in the outer nuclear layer that were sufficiently separated from the hyperreflective outer plexiform layer (√) and candidate foci that are embedded within in the ganglion cell layer/ inner plexiform layer (×).
tracking enabled. The OSCAR-IB criteria were used to assess the quality of the retinal OCT scans, and we report results in agreement with the Advised Protocol for OCT Study Terminology and Elements (APOSTEL).

| R E SU LT S
In the manually labelled data set, the blob detector found in total 2548 candidate foci. The algorithm correctly detected 350 out of the 391 manually annotated hyperreflective foci, indicating that the blob detector misses around 11% of hyperreflective foci, identified as true foci by the grader. The CNN classifier confusion matrix can be seen in Table 2. The accuracy of CNN classifier was assessed by splitting the 2548 detections in the manually labelled data set into training (80%) and validation (20%) data sets. On the validation data set, the classifier obtained an accuracy of 96.3%, a sensitivity of 88.4% and a specificity of 97.5%. Corresponding receiver operating characteristic (ROC) curve on the validation set achieved an area under the curve of 0.989 (see ROC curve, Figure 2). Examples of the classification method and heatmap visualizations of hyperreflective foci are shown in Figures 3-5. A box plot showing the distribution of data obtained from feature extraction can be seen in Figure 6.

| DI SC US SION
This longitudinal exploratory study demonstrated that our automated image analysis pipeline performed successfully with an overall high accuracy of the CNN classification (AUC of 0.989) at detecting hyperreflective foci in the outer nuclear layer.
In general, image segmentation is a problem often tackled with deep learning architectures such as a UNet (Ronneberger et al., 2015). However, a UNet would not be able to reliably segment the hyperreflective foci for multiple reasons in this study. Our data comprise OCT images that often possessed only a small number of hyperreflective foci. Consequently, there would not be enough data to train a UNet since a single data point for a UNet is a full image. A large data set with thousands or at least hundreds of data points is necessary when training deep learning-based algorithms. If a data point is a single hyperreflective focus instead, then the lack of data is much less problematic. In this case, we have 391 positive data points along with thousands of negative data points. The segmentation pipeline therefore consists of first locating likely hyperreflective foci with blob analysis and then classifying them with a CNN. A segmentation problem is essentially converted into a classification one. The features we extracted are useful in helping the CNN classify hyperreflective foci. Additionally, they provide useful information about, for example the distribution of foci radii and the retinal layer position of the hyperreflective foci. These features could potentially inform which characteristics of hyperreflective foci are associated with healthy/ unhealthy specimens. A naive UNet segmentation would be unable to extract these features.
The frequent presence of hyperreflective foci with OCT calls for a systematic study of their appearance, location, density, movement, disappearance and associated health characteristics. In this study, we only considered ophthalmoscopically normal eyes or eyes with reversible abnormalities on funduscopic examination. The approach in many prior studies has been to look at severely abnormal conditions such as radiation retinopathy (Frizziero et al., 2016) or neovascular AMD (Mokhtari et al., 2017). In this study, we investigated conditions with no confounding elements of structural abnormalities in the retina which improved the algorithm robustness. This approach is potentially applicable in conditions with scant hyperreflective foci such as multiple sclerosis (Pilotto et al., 2020), where the retina may by relatively normal, and therefore it is easier to train automated image analysis systems.

F I G U R E 3
Examples of the classification method. In patient 1, the CNN classifier detects foci pattern (green circles) from the nonfoci pattern (red circles) in the outer nuclear layer of the retina.
Currently, very few studies concerning the automated or semi-automated detection of retinal hyperreflective foci have been performed (Varga et al., 2019;Yu et al., 2019). These studies are criticized for being unreliable and applicable only from an informative and technical point of view (Midena et al., 2021). Moreover, these studies often examine both the inner and outer retinal layers where it can be challenging to distinguish between vessels and intraretinal hyperreflective elements. To overcome this limitation, only the outer nuclear layer was examined in this study. The outer nuclear layer is considered avascular and therefore every hyperreflective foci seen on an OCT scan can be assumed to represent a true candidate focus and not a vessel in cross-section.
In this study, heatmap visualizations often revealed distinct pattern of hyperreflective foci being located in the fovea or close to the optic nerve head. The use of neural networks for the quantification and visualization of hyperreflective foci could assist in tracking subtle intraretinal changes over time that are difficult to detect for even specialists since the magnitude of the change can be very  small (Lang et al., 2016). Furthermore, the localization and quantification of the actual size of foci structure in the retina is important. One example is shown in Figure 5 with a central serous chorioretinopathy with detachment of the macula but has the added feature of cystoid oedema. The morphological features are illustrated in great details and the hyperreflective elements that are seen after resolution of the detachment are much larger than the hyperreflective foci we see in healthy subjects.
Limitations of this study include the small sample size and the fact that OCT scans were obtained from only one imaging platform type and not compared with other commercially available OCT devices. However, we would expect to benefit from using a heterogeneous OCT material originating from slightly different OCT scanning protocols on the Heidelberg machines. Regarding methodological issues all scans need to be examined for motion artefacts arising from averaging. Furthermore, B scans need to be closely spaced in order to thoroughly evaluate hyperreflective foci movement and errors in scan positioning. Due to the distance between B scans a potential variation in the distribution of hyperreflective foci was not considered detectable in this study.
Longitudinal analysis of hyperreflective foci with high-quality OCT data sets may be crucial in future for understanding the pathophysiologic processes underlying disease progression in various retinal disorders. Further prospective studies are necessary to establish the time intervals and transversal resolution needed to sufficiently and accurate detect and track changes in the density and distribution of hyperreflective foci over time in different retinal disorders. F I G U R E 6 (a) (i) Radius (pixels) shows radius of blobs detected in the pixel unit. One pixel is equal to 2.5 μm. (ii) The layer numbers refer to specific retinal layers and every hyperreflective foci have different anatomical locations corresponding to certain retinal layer numbers. (iii) Max. pixel intensity reflects the brightest pixel contained in blobs detected. (iv) Vessel proximity is a measure of how close a hyperreflective focus is to blood vessels. High values mean close proximity to blood vessels. (b) OCT B-scan with layer numbers marked.

| CONC LUSIONS
In conclusion, this study demonstrates that automated image analysis and machine learning methods can be applied to successfully identify, quantify and visualize the presence and time-resolved dynamics of hyperreflective foci in the outer nuclear layer of the retina using highquality OCT scans.