Data Platform Engineer

Data Platform Engineer - 12 Month Contract

Key Responsibilities

Platform Engineering & Development

Design, implement, and maintain Big Data platforms (e.g., Hadoop, Spark, Kafka) used across the CIB environment.

Build robust batch and real-time data ingestion pipelines using tools like Apache NiFi, Airflow, Spark, and Kafka Streams.

Maintain and enhance enterprise data lakes and warehouse environments using technologies such as Hive, Delta Lake, and Azure Synapse.

Cloud & Hybrid Integration

Architect and deploy data platform solutions on Microsoft Azure (Databricks, Azure Data Lake Storage, Synapse Analytics).

Build hybrid cloud systems to integrate on-premises and cloud-based data infrastructure.

Ensure optimal performance, scalability, and cost-efficiency across cloud workloads.

Data Governance, Compliance & Security

Ensure platform compliance with data governance and privacy regulations (POPIA, GDPR, BCBS239).

Implement robust security controls across infrastructure including encryption, access controls, and audit logging.

Work closely with data stewards and governance teams to integrate metadata management and data cataloging tools.

Automation & DevOps

Develop and maintain CI / CD pipelines for automated testing, deployment, and monitoring of data solutions.

Automate infrastructure provisioning using tools like Terraform and Azure DevOps.

Perform routine system administration, performance tuning, and issue resolution across data platforms.

Monitoring & Support

Implement monitoring solutions (e.g., Prometheus, Grafana, ELK Stack) to ensure system availability and reliability.

Provide L2 / L3 support for production data environments, managing incidents and service requests effectively.

Stakeholder Engagement

Collaborate with cross-functional teams including data scientists, analysts, developers, and compliance teams.

Translate business and analytical requirements into scalable platform solutions.

Participate in Agile sprints and architecture design reviews.

Qualifications & Experience

Minimum Requirements

Bachelors degree in Computer Science, Information Systems, Engineering, or related field.

5+ years of experience in Big Data engineering or platform operations.

Experience in enterprise-grade platforms in banking or financial services.

Technical Skills

Strong proficiency with Hadoop ecosystem : HDFS, Hive, Spark, Kafka.

Expertise in Azure cloud services : Azure Data Factory, Azure Databricks, Azure Data Lake, Synapse Analytics.

Solid programming skills in Python, Scala, Java, and SQL.

Familiarity with Terraform, Git, Jenkins, Docker, and Kubernetes.

Experience with data governance tools such as Apache Atlas or Collibra is advantageous.

Soft Skills

Excellent problem-solving and troubleshooting ability.

Strong communication and collaboration skills.

Ability to work in a fast-paced, high-stakes environment.

Preferred Certifications

Microsoft Certified : Azure Data Engineer Associate

Cloudera Data Platform Certified Developer

Databricks Certified Data Engineer Associate

TOGAF or similar architecture frameworks (advantageous)

#J-18808-Ljbffr