A carregar...
Publicação
Estimating pointwise reliability of machine learning predictions in clinical context
| Nome: | Descrição: | Tamanho: | Formato: | |
|---|---|---|---|---|
| 6.6 MB | Adobe PDF |
Autores
Resumo(s)
À medida que o interesse por machine learning se expande para domínios críticos, como a prática clínica, a necessidade de medidas de confiança nas classificações é vital, dado que as decisões baseadas nesses resultados podem ter consequências significativas. A avaliação tradicional dos modelos de machine learning depende de métricas de desempenho globais, que fornecem informações agregadas, mas não indicam se uma classificação individual é confiável. Esta limitação reforça a importância dos métodos de confiabilidade pontual, que usam medidas para estimar a confiabilidade de classificações individuais de modelos de machine learning.
Este trabalho avalia métodos de confiabilidade pontual, com foco particular em abordagens agnósticas ao modelo e que utilizam o princípio da densidade e ajuste local. Estes princípios avaliam a confiabilidade analisando a semelhança de novas instâncias com os dados de treino e avaliando o desempenho do modelo em determinada região do espaço de dados. Neste trabalho, foram considerados três casos de estudo, abrangendo dois problemas de classificação clínica relacionados com decisões de internamento e mortalidade derivada de problemas cardiovasculares, e uma tarefa de regressão que envolve a previsão do numero de admissões hospitalares. A confiabilidade das classificações dos modelos foi avaliada utilizando abordagens baseadas em clustering, distância e ajuste local. O desempenho das abordagens foi avaliado através da análise da relação entre intervalos de confiabilidade e erros de classificação.
Os resultados demonstraram que os métodos que combinam o princípio de densidade e ajuste local geralmente apresentam melhores resultados do que os que se baseiam num único princípio, alcançando taxas de erro mais baixas para classificações avaliadas como altamente confiáveis. Em tarefas de classificação, o método que combina os princípios de densidade e ajuste local produziu os resultados mais estáveis, apresentando uma base mais sólida para a tomada de decisões críticas. Além da análise de confiabilidade, o caso de estudo com regressão comparou modelos estatísticos tradicionais com abordagens de machine learning na previsão de admissões hospitalares. Os resultados mostram que os modelos de machine learning alcançam uma maior correlação com os valores reais e que, para a análise de confiabilidade, os métodos que combinaram ambos os princípios melhoraram a correlação entre os erros absolutos das previsões e a confiabilidade atribuída.
As interest in machine learning expands into critical domains, such as clinical practice, the need for reliable predictions is critical, as decisions based on these outputs can have significant consequences. Traditional evaluation of machine learning models relies on global performance metrics, which provide aggregate information but does not indicate whether an individual prediction can be trusted. This limitation reinforces the importance of pointwise reliability methods that use measures to estimate the reliability of single predictions. This work assesses pointwise reliability methods, with a particular focus on model agnostic approaches using the density principle and the local fit principle. These principles evaluate reliability by analyzing the similarity of new instances to the training data and by evaluating the performance of the model in local regions of the feature space. Three case studies were considered, covering two clinical classification problems related to treatment decisions and cardiovascular related mortality, as well as a regression task that involves the prediction of hospital admissions. Reliability was assessed using clustering based, distance based, and local fit methods. The performance of the methods was evaluated by analyzing the relationship between reliability intervals and the prediction errors. The results demonstrate that methods combining density and local fit principles generally outperform those relying on a single principle, achieving lower error rates for predictions assigned high reliability. In classification tasks, the method combining density and local fit principles produced the most stable and interpretable results, offering a stronger basis for decision making. In addition to reliability analysis, the regression case study compared traditional statistical models with ML approaches in the forecasting of hospital admission. The findings suggest that the ML models achieved a greater correlation with the actual values and that, for the pointwise results, the methods combining both principles improved the alignment between absolute prediction errors and the assigned reliability scores.
As interest in machine learning expands into critical domains, such as clinical practice, the need for reliable predictions is critical, as decisions based on these outputs can have significant consequences. Traditional evaluation of machine learning models relies on global performance metrics, which provide aggregate information but does not indicate whether an individual prediction can be trusted. This limitation reinforces the importance of pointwise reliability methods that use measures to estimate the reliability of single predictions. This work assesses pointwise reliability methods, with a particular focus on model agnostic approaches using the density principle and the local fit principle. These principles evaluate reliability by analyzing the similarity of new instances to the training data and by evaluating the performance of the model in local regions of the feature space. Three case studies were considered, covering two clinical classification problems related to treatment decisions and cardiovascular related mortality, as well as a regression task that involves the prediction of hospital admissions. Reliability was assessed using clustering based, distance based, and local fit methods. The performance of the methods was evaluated by analyzing the relationship between reliability intervals and the prediction errors. The results demonstrate that methods combining density and local fit principles generally outperform those relying on a single principle, achieving lower error rates for predictions assigned high reliability. In classification tasks, the method combining density and local fit principles produced the most stable and interpretable results, offering a stronger basis for decision making. In addition to reliability analysis, the regression case study compared traditional statistical models with ML approaches in the forecasting of hospital admission. The findings suggest that the ML models achieved a greater correlation with the actual values and that, for the pointwise results, the methods combining both principles improved the alignment between absolute prediction errors and the assigned reliability scores.
Descrição
Palavras-chave
Density principle Local fit principle Machine learning Pointwise reliability
Contexto Educativo
Citação
Editora
Licença CC
Sem licença CC