Implementación de computación en la nube en el laboratorio ADA de la Facultad de Sistemas para analítica de datos : implementación de un lago de datos en los servidores del laboratorio ADA.

Abstract

In this project, the process of designing and implementing a data lake with Hadoop was carried out in a cluster with one master node and four slave nodes using Docker containers. It began with an investigation and analysis of the requirements and objectives of the project, focusing on the importance of the architecture and its adaptation to the resources of the high-performance server of the ADA laboratory. To achieve a successful implementation, a previous experimentation with different configurations and tools was carried out, which allowed understanding the operation of Hadoop and Docker, as well as their interactions. The data lake architecture was designed, including the NameNode, ResourceManager and DataNodes coexisting in a single container with their NodeManagers to improve efficiency and resource utilization. The process of saving data in the data lake was carried out by executing data analysis scripts from the end users of the ADA laboratory, verifying the division into blocks and their replication in the DataNodes to guarantee fault tolerance and high availability. The operation of the internal network, the NameNode, the ResourceManager and the DataNodes was verified by accessing their web interfaces and verifying that all the containers were correctly connected. The project culminated in the successful implementation of a data lake, meeting the objectives and requirements set, and whose architecture was tested and validated by end users belonging to the ADA laboratory.