The aim of this project is to create a data pipeline to ingest, process and analyze data from a remote location. We would use Apache Nifi, a dataflow tool to ingest the data and store it in HDFS. Following that, we would analyze the stored files using Apache Spark and send those files to the Elasticsearch cluster. Furthermore, the entire workflow is containerized to facilitate faster deployments.
Project link: https://datascienceandengineering.com/project-meta/event-data-analysis-using-spark-and-elasticsearch/