Data Engineer (Hadoop Ecosystem)

Overview

Job title : Data Engineer (Hadoop Ecosystem)

Job Location : Gauteng, Johannesburg

Deadline : November 22, 2025

Job Description

We are seeking a skilled Data Engineer to design and develop scalable data pipelines that ingest raw, unstructured JSON data from source systems and transform it into clean, structured datasets within the Hadoop-based data platform. The ideal candidate will play a critical role in enabling data availability, quality, and usability by engineering the movement of data from the Raw Layer to the Published and Functional Layers.

Responsibilities

• Design, build, and maintain robust data pipelines to ingest raw JSON data from source systems into the Hadoop Distributed File System (HDFS).
• Transform and enrich unstructured data into structured formats (e.g., Parquet, ORC) for the Published Layer using tools like PySpark, Hive, or Spark SQL.
• Develop workflows to further process and organize data into Functional Layers optimized for business reporting and analytics.
• Implement data validation, cleansing, schema enforcement, and deduplication as part of the transformation process.
• Collaborate with Data Analysts, BI Developers and Business Users to understand data requirements and ensure datasets are production-ready.
• Optimize ETL / ELT processes for performance and reliability in a large-scale distributed environment.
• Maintain metadata, lineage and documentation for transparency and governance.
• Monitor pipeline performance and implement error handling and alerting mechanisms.

Technical Skills & Experience

Nice to Have

#J-18808-Ljbffr