A carregar...
Publicação
Modernização de soluções de engenharia de dados no setor logístico
| Nome: | Descrição: | Tamanho: | Formato: | |
|---|---|---|---|---|
| 1.05 MB | Adobe PDF |
Autores
Orientador(es)
Resumo(s)
Este relatório descreve o estágio curricular realizado na MixMove entre novembro de 2024 e maio de 2025, no âmbito do Mestrado em Ciência de Dados. O principal foco foi a modernização de soluções de engenharia de dados aplicadas ao setor logístico, com especial ênfase na integração de sistemas heterogéneos, na automação de pipelines ETL e na aplicação de técnicas de data masking, alinhadas com os requisitos da norma ISO 27001. Ao longo do estágio, os principais objetivos passaram pelo desenvolvimento de integrações robustas para clientes como a Volkswagen UK, Enfega, Freja e Rangel; pela modernização dos sistemas de validação de dados; pela automação dos processos de extração, transformação e carregamento de dados; e pela implementação de mecanismos de segurança e monitorização eficazes. O estágio surgiu no contexto de uma infraestrutura tecnológica envelhecida, onde as validações se encontravam dispersas, havia necessidade de trabalhar com dados sensíveis em ambientes de teste, e existiam desafios relacionados com a escalabilidade dos sistemas, que gerem grandes volumes de dados distribuídos por mais de vinte tabelas operacionais. A abordagem metodológica adotada combinou princípios de DataOps com a framework Scrum, através de ciclos iterativos de duas semanas. A recolha de informação baseou-se na análise de documentação técnica, análise de logs de desempenho e numa revisão sistemática da literatura relevante. O tratamento da informação incluiu engenharia reversa de sistemas obsoletos, validações em várias camadas com recurso à biblioteca FluentValidation, e a construção de pipelines ETL automatizados e escaláveis. As fontes de dados abrangeram desde os sistemas operacionais da Volkswagen UK, até ficheiros Excel, Csv e Json, passando por APIs externas como a da Routyn e documentação de sistemas obsoletos. Os resultados alcançados foram bastante positivos, tendo-se verificado uma redução de 85% no tempo de processamento, a eliminação total de erros manuais e uma melhoria significativa na qualidade dos dados processados. A aplicação de técnicas de data masking permitiu assegurar a conformidade com requisitos regulamentares, enquanto a nova arquitetura modular, baseada em microserviços, trouxe ganhos claros em termos de capacidade de manutenção e escalabilidade.
This report presents the curricular internship carried out at MixMove between November 2024 and May 2025, as part of the Master's in Data Science. The main focus was the modernization of data engineering solutions for the logistics sector, with special emphasis on the integration of heterogeneous systems, automation of ETL pipelines, and the implementation of data masking techniques in compliance with ISO 27001 standards. Throughout the internship, the primary goals were to develop robust integrations for clients such as Volkswagen UK, Enfega, Freja, and Rangel; to modernize data validation systems; to automate extraction, transformation, and loading processes; and to implement effective security and monitoring mechanisms. The project was carried out in the context of an outdated technological infrastructure, where validations were scattered, sensitive data had to be handled in test environments, and the systems faced scalability challenges due to high volumes of data spread across more than twenty operational tables. The adopted methodological approach combined DataOps principles with the Scrum framework, through two-week iterative development cycles. Data collection involved technical documentation analysis, performance log inspection, and a systematic literature review. Information processing included reverse engineering of legacy systems, multi-layer validation using the FluentValidation library, and the construction of automated and scalable ETL pipelines. The data sources ranged from Volkswagen UK’s operational systems to Excel, CSV, and JSON files, as well as external APIs such as Routyn and legacy system documentation. The results were highly positive, showing an 85% reduction in processing time, complete elimination of manual errors, and a significant improvement in data quality. The implementation of data masking ensured regulatory compliance, while the new modular microservices-based architecture improved system maintainability and scalability.
This report presents the curricular internship carried out at MixMove between November 2024 and May 2025, as part of the Master's in Data Science. The main focus was the modernization of data engineering solutions for the logistics sector, with special emphasis on the integration of heterogeneous systems, automation of ETL pipelines, and the implementation of data masking techniques in compliance with ISO 27001 standards. Throughout the internship, the primary goals were to develop robust integrations for clients such as Volkswagen UK, Enfega, Freja, and Rangel; to modernize data validation systems; to automate extraction, transformation, and loading processes; and to implement effective security and monitoring mechanisms. The project was carried out in the context of an outdated technological infrastructure, where validations were scattered, sensitive data had to be handled in test environments, and the systems faced scalability challenges due to high volumes of data spread across more than twenty operational tables. The adopted methodological approach combined DataOps principles with the Scrum framework, through two-week iterative development cycles. Data collection involved technical documentation analysis, performance log inspection, and a systematic literature review. Information processing included reverse engineering of legacy systems, multi-layer validation using the FluentValidation library, and the construction of automated and scalable ETL pipelines. The data sources ranged from Volkswagen UK’s operational systems to Excel, CSV, and JSON files, as well as external APIs such as Routyn and legacy system documentation. The results were highly positive, showing an 85% reduction in processing time, complete elimination of manual errors, and a significant improvement in data quality. The implementation of data masking ensured regulatory compliance, while the new modular microservices-based architecture improved system maintainability and scalability.
Descrição
Palavras-chave
Engenharia de Dados DataOps Qualidade de Dados ETL Data Warehouse Data Engineering Data Quality