Repository logo
 
Publication

MDF-Net for abnormality detection by fusing X-rays with clinical data

dc.contributor.authorHsieh, C
dc.contributor.authorNobre, IB
dc.contributor.authorSousa, SC
dc.contributor.authorOuyang, C
dc.contributor.authorBrereton, M
dc.contributor.authorNascimento, JC
dc.contributor.authorJorge, J
dc.contributor.authorMoreira, C
dc.date.accessioned2023-10-08T20:47:56Z
dc.date.available2023-10-08T20:47:56Z
dc.date.issued2023
dc.description.abstractThis study investigates the effects of including patients' clinical information on the performance of deep learning (DL) classifiers for disease location in chest X-ray images. Although current classifiers achieve high performance using chest X-ray images alone, consultations with practicing radiologists indicate that clinical data is highly informative and essential for interpreting medical images and making proper diagnoses. In this work, we propose a novel architecture consisting of two fusion methods that enable the model to simultaneously process patients' clinical data (structured data) and chest X-rays (image data). Since these data modalities are in different dimensional spaces, we propose a spatial arrangement strategy, spatialization, to facilitate the multimodal learning process in a Mask R-CNN model. We performed an extensive experimental evaluation using MIMIC-Eye, a dataset comprising different modalities: MIMIC-CXR (chest X-ray images), MIMIC IV-ED (patients' clinical data), and REFLACX (annotations of disease locations in chest X-rays). Results show that incorporating patients' clinical data in a DL model together with the proposed fusion methods improves the disease localization in chest X-rays by 12% in terms of Average Precision compared to a standard Mask R-CNN using chest X-rays alone. Further ablation studies also emphasize the importance of multimodal DL architectures and the incorporation of patients' clinical data in disease localization. In the interest of fostering scientific reproducibility, the architecture proposed within this investigation has been made publicly accessible( https://github.com/ChihchengHsieh/multimodal-abnormalities-detection ).pt_PT
dc.description.versioninfo:eu-repo/semantics/publishedVersionpt_PT
dc.identifier.citationSci Rep . 2023 Sep 23;13(1):15873.pt_PT
dc.identifier.doi10.1038/s41598-023-41463-0pt_PT
dc.identifier.urihttp://hdl.handle.net/10400.26/47121
dc.language.isoengpt_PT
dc.peerreviewedyespt_PT
dc.subjectRadiografiapt_PT
dc.subjectRadiologistaspt_PT
dc.subjectRaios-Xpt_PT
dc.subjectRadiographypt_PT
dc.subjectRadiologistspt_PT
dc.subjectX-Rayspt_PT
dc.titleMDF-Net for abnormality detection by fusing X-rays with clinical datapt_PT
dc.typejournal article
dspace.entity.typePublication
oaire.citation.issue1pt_PT
oaire.citation.titleScientific Reportspt_PT
oaire.citation.volume13pt_PT
rcaap.rightsopenAccesspt_PT
rcaap.typearticlept_PT

Files

Original bundle
Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
SciRep.pdf
Size:
7.68 MB
Format:
Adobe Portable Document Format
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.85 KB
Format:
Item-specific license agreed upon to submission
Description: