A carregar...
Publicação
Reliability of machine learning models based on data density principles applied to cardiovascular risk assessment
| Nome: | Descrição: | Tamanho: | Formato: | |
|---|---|---|---|---|
| 3.59 MB | Adobe PDF |
Autores
Resumo(s)
Os modelos de machine learning (ML) têm sido cada vez mais utilizados em diversas áreas, alcançando frequentemente desempenhos notáveis. No entanto, em áreas críticas como a da saúde, onde as decisões podem ter consequências significativas, uma preocupação fundamental quanto ao seu uso é a capacidade de confiar nas previsões que estes modelos geram. Assim, ao incorporar medidas de confiabilidade nas previsões, melhora-se a segurança e a transparência, tornando a interação com os sistemas de IA mais segura e compreensível.
A confiabilidade dos modelos de ML pode ser avaliada recorrendo a uma medida de desempenho global, utilizando métricas como a sensibilidade e a especificidade, ou através de uma medida de desempenho ponto a ponto (pointwise), relativa a cada previsão individual. Nesta última abordagem, a confiabilidade pointwise de um modelo de ML pode ser avaliada com base em princípios da densidade, os quais utilizam medidas de distância e características dos dados para comparar uma instância com os exemplos presentes no conjunto de treino.
Este estudo investigou e implementou métricas de confiabilidade pointwise para avaliar previsões de modelos de ML em contexto clínico. Foram implementados três métodos baseados no princípio da densidade, os quais atribuem a cada previsão um nível de confiança no intervalo [0,1]: (1) método baseado na proximidade dos dados, utilizando as distâncias entre pontos de dados; (2) método baseado em clustering, usando o algoritmo Fuzzy C-means; e (3) método baseado no conceito de convex hull, recorrendo às propriedades geométricas deste polígono convexo.
Para testar os métodos, foram usados dois conjuntos de dados. O primeiro é um conjunto de dados sintéticos para a classificação do Índice de Massa Corporal. O segundo é um conjunto de dados reais recolhidos em ambiente clínico, disponibilizado pelo CHUC – Centro Hospitalar e Universitário de Coimbra, o qual inclui informações de 1544 pacientes com síndrome coronária aguda.
Os resultados indicam que os métodos (1) e (3) são eficazes, atribuindo níveis de confiança mais elevados à maioria das previsões corretamente classificadas e níveis de confiança mais baixos às previsões incorretamente classificadas. Por sua vez, o método (2) não conseguiu estabelecer uma relação clara entre as previsões e a confiança, mostrando-se ineficaz para o objetivo pretendido. Convém salientar que estes resultados poderão variar caso sejam usados conjuntos de dados diferentes.
Com base nos resultados obtidos, esta investigação propõe métodos com potencial prático para o suporte à decisão clínica. Estas abordagens podem ajudar a prevenir fatalidades e garantir diagnósticos mais seguros aos profissionais de saúde.
Machine learning (ML) models have been increasingly adopted across various fields, often achieving remarkable performances. However, in critical areas such as healthcare, where decisions can have significant consequences, a fundamental concern is the ability to trust the predictions generated by these models. By incorporating reliability or confidence measures into prediction outcomes, security and transparency are enhanced, thus promoting trust in human-AI interaction. The reliability of a ML model can be assessed either as an overall performance measure, using metrics such as sensitivity and specificity, or as a case-specific evaluation, trough pointwise reliability metrics. In the later approach, pointwise reliability can be evaluated based on density principle, which uses distance measures and data patterns to compare a given instance to the examples in the training set. This study researched and implemented pointwise reliability metrics to evaluate the predictions made by ML models in the clinical context. Three reliability assessment methods based on the density principle were explored for evaluating ML model prediction outcomes, each assigning a reliability level in the range [0,1] to individual predictions: (1) a data proximity based method, which uses distances between data points; (2) a clustering based method, employing the Fuzzy C-Means clustering algorithm; and (3) a convex hull based method, which exploits the convex hull’s geometric properties. These methods were tested on two datasets. The first is a synthetic dataset devised for the classification of the Body Mass Index. The second consists of real-world data collected in a clinical environment, provided by CHUC – Coimbra Hospital and University Centre, comprising information from 1544 acute coronary syndrome patients. The results show methods (1) and (3) as effective, assigning higher reliability levels to most correct predictions, and lower reliability levels to incorrect predictions. However, the clustering method (2) was unable to establish a clear relationship between prediction outcomes and reliability, showing to be ineffective for this specific purpose. It is worth noting that these results may vary when applied to different datasets. Based on the results obtained, this research proposes methods with practical application potential for improving clinical decision support. Such approaches may help prevent fatalities by diseases, and provide healthcare professionals with more reliable diagnosis.
Machine learning (ML) models have been increasingly adopted across various fields, often achieving remarkable performances. However, in critical areas such as healthcare, where decisions can have significant consequences, a fundamental concern is the ability to trust the predictions generated by these models. By incorporating reliability or confidence measures into prediction outcomes, security and transparency are enhanced, thus promoting trust in human-AI interaction. The reliability of a ML model can be assessed either as an overall performance measure, using metrics such as sensitivity and specificity, or as a case-specific evaluation, trough pointwise reliability metrics. In the later approach, pointwise reliability can be evaluated based on density principle, which uses distance measures and data patterns to compare a given instance to the examples in the training set. This study researched and implemented pointwise reliability metrics to evaluate the predictions made by ML models in the clinical context. Three reliability assessment methods based on the density principle were explored for evaluating ML model prediction outcomes, each assigning a reliability level in the range [0,1] to individual predictions: (1) a data proximity based method, which uses distances between data points; (2) a clustering based method, employing the Fuzzy C-Means clustering algorithm; and (3) a convex hull based method, which exploits the convex hull’s geometric properties. These methods were tested on two datasets. The first is a synthetic dataset devised for the classification of the Body Mass Index. The second consists of real-world data collected in a clinical environment, provided by CHUC – Coimbra Hospital and University Centre, comprising information from 1544 acute coronary syndrome patients. The results show methods (1) and (3) as effective, assigning higher reliability levels to most correct predictions, and lower reliability levels to incorrect predictions. However, the clustering method (2) was unable to establish a clear relationship between prediction outcomes and reliability, showing to be ineffective for this specific purpose. It is worth noting that these results may vary when applied to different datasets. Based on the results obtained, this research proposes methods with practical application potential for improving clinical decision support. Such approaches may help prevent fatalities by diseases, and provide healthcare professionals with more reliable diagnosis.
Descrição
Palavras-chave
Confiabilidade pointwise Princípio da densidade Machine Learning Avaliação do risco cardiovascular Pointwise reliability Density principle Cardiovascular risk assessment
Contexto Educativo
Citação
Editora
Licença CC
Sem licença CC