Talent.com
This job offer is not available in your country.
AWS Data Engineer

AWS Data Engineer

wePlacePretoria, South Africa
30+ days ago
Job description

Job Purpose :

Responsible for creating and managing the technological part of data infrastructure in every step of data flow. From configuring data sources to integrating analytical tools all these systems would be architected, built and managed by a general-role Data Engineer.

Minimum education (essential) :

Bachelors degree in Computer Science or Engineering (or similar)

Minimum education (desirable) :

  • Honors degree in Computer Science or Engineering (or similar)
  • AWS Certified Data Engineer; or
  • AWS Certified Solutions Architect; or
  • AWS Certified Data Analyst

Minimum applicable experience (years) :

5+ years working experience

Required nature of experience :

  • Data Engineering development
  • Experience with AWS services used for data warehousing, computing and transformations i.e. AWS Glue (crawlers, jobs, triggers, and catalog), AWS S3, AWS Lambda, AWS Step Functions, AWS Athena and AWS CloudWatch
  • Experience with SQL and NoSQL databases (e.g., PostgreSQL, MySQL, DynamoDB)
  • Experience with SQL for querying and transformation of data
  • Skills and Knowledge (essential) :

  • Strong skills in Python (especially PySpark for AWS Glue)
  • Strong knowledge of data modelling, schema design and database optimization
  • Proficiency with AWS and infrastructure as code
  • Skills and Knowledge (desirable) :

  • Knowledge of SQL, Python, AWS serverless microservices,
  • Deploying and managing ML models in production
  • Version control (Git), unit testing and agile methodologies
  • Data Architecture and Management 20%

  • Design and maintain scalable data architectures using AWS services for example, but not limited to, AWS S3, AWS Glue and AWS Athena.
  • Implement data partitioning and cataloging strategies to enhance data organization and accessibility.
  • Work with schema evolution and versioning to ensure data consistency.
  • Develop and manage metadata repositories and data dictionaries.
  • Assist and support with defining setup and maintenance of data access roles and privileges.
  • Pipeline Development and ETL 30%

  • Design, develop and optimize scalable ETL pipelines using batch and real-time processing frameworks (using AWS Glue and PySpark).
  • Implement data extraction, transformation and loading processes from various structured and unstructured sources.
  • Optimize ETL jobs for performance, cost efficiency and scalability.
  • Develop and integrate APIs to ingest and export data between various source and target systems, ensuring seamless ETL workflows.
  • Enable scalable deployment of ML models by integrating data pipelines with ML workflows.
  • Automation, Monitoring and Optimization 30%

  • Automate data workflows and ensure they are fault tolerant and optimized.
  • Implement logging, monitoring and alerting for data pipelines.
  • Optimize ETL job performance by tuning configurations and analyzing resource usage.
  • Optimize data storage solutions for performance, cost and scalability.
  • Ensure the optimisation of AWS resources for scalability for data ingestion and outputs.
  • Deploy machine learning models into productions using cloud-based services like AWS SageMaker.
  • Security, Compliance and Best Practices 10%

  • Ensure API security, authentication and access control best practices.
  • Implement data encryption, access control and compliance with GDPR, HIPAA, SOC2 etc.
  • Establish data governance policies, including access control and security best practices.
  • Development Team Mentorship and Collaboration 5%

  • Work closely with data scientists, analysts and business teams to understand data needs.
  • Collaborate with backend teams to integrate data pipelines into CI / CD.
  • Assist with developmental leadership to the team through coaching, code reviews and mentorship.
  • Ensure technological alignment with B2C division strategy supporting overarching strategy and vision.
  • Identify and encourage areas for growth and improvement within the team.
  • QMS and Compliance 5%

    Document data processes, transformations and architectural decisions.

    Maintain high standards of software quality within the team by adhering to good processes, practices and habits, including compliance to QMS system, and data and system security requirements.

    Ensure compliance to the established processes and standards for the development lifecycle, including but not limited to data archival.

    Drive compliance to the Quality Management System in line with the Quality Objectives, Quality Manual, and all processes related to the design, development and implementation of software related to medical devices.

    Comply to ISO, CE, FDA (and other) standards and requirements as is applicable to assigned products.

    Safeguard confidential information and data.

    Should you not receive a response from us within one week of your application, your application has unfortunately not been successful.

    Create a job alert for this search

    Data Engineer • Pretoria, South Africa

    Related jobs
    • Promoted
    AWS Platform Engineer – Midrand / Semi-Remote – R700 PH

    AWS Platform Engineer – Midrand / Semi-Remote – R700 PH

    E-MergePretoria, South Africa
    Remote
    We are looking for someone with strong ability to understand and interpret business needs and requirements with an aptitude to move concepts through to proposal and successful implementation.Bachel...Show moreLast updated: 30+ days ago
    • Promoted
    AI and Analytics Engineer – Centurion – R700k to R900k per annum

    AI and Analytics Engineer – Centurion – R700k to R900k per annum

    E-MergePretoria, South Africa
    Were looking for a sharp, business-savvy.Youll work at the intersection of advanced analytics, machine learning, and engineeringenabling better, faster, and future-focused decisions across the busi...Show moreLast updated: 30+ days ago
    • Promoted
    AWS Data Engineer – Centurion / Hybrid – R1.2m PA

    AWS Data Engineer – Centurion / Hybrid – R1.2m PA

    E-MergePretoria, South Africa
    Were a team of curious minds and caffeine-fueled builders on a mission to turn raw data into real-world impact.We believe in pipelines that dont leak, schemas that actually make sense, and dashboar...Show moreLast updated: 30+ days ago
    • Promoted
    Azure Data Engineer

    Azure Data Engineer

    Dina Gates Recruitment (Pty) LtdKempton Park, South Africa
    We are seeking a skilled Azure Data Engineer with a minimum of 2 years of experience to join our dynamic data team.The ideal candidate will have a strong background in SQL, Microsoft Azure, data wa...Show moreLast updated: 22 days ago
    Data Engineer- Ab Initio

    Data Engineer- Ab Initio

    Fourier IT Innovation (Pty) LtdCenturion, Gauteng, South Africa
    Quick Apply
    South African company with over 21 years of experience delivering exceptional professionalism, deep technical knowledge, and a passion for innovation. Our team thrives on pushing boundaries—both as ...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    Dis-Chem LifeWorkFromHome, Gauteng, South Africa
    Purpose of the Role : The Data Engineer is the architect and builder of Dis-Chem Life’s data foundation, creating the infrastructure that turns raw information into a strategic asset.This role goes ...Show moreLast updated: 3 days ago
    • Promoted
    Lead Data Engineer

    Lead Data Engineer

    Absa Group LimitedRandburg, ZA
    Job title : Lead Data Engineer.Job Location : Gauteng, Randburg.Leveraging a thorough understanding of the business data requirements & own data subject matter expertise (ahead of trend), set data ...Show moreLast updated: 30+ days ago
    • Promoted
    SQL Data Engineer

    SQL Data Engineer

    Datonomy SolutionsSandton, South Africa
    We are looking for a capable and detail-oriented.You will play a key role in ensuring the reliability, performance, and quality of data pipelines, with a strong emphasis on compliance and metadata ...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer (Contractor)

    Data Engineer (Contractor)

    The Legends AgencyWorkFromHome, Gauteng, South Africa
    Our client is a specialist data and technology consultancy, delivering innovative solutions that empower financial services businesses to make smarter, faster decisions. With a strong focus on advan...Show moreLast updated: 1 day ago
    • Promoted
    Data Engineer

    Data Engineer

    Network ITCenturion, Gauteng, South Africa
    Our client is seeking a skilled Data Engineer to join our growing team.If you are passionate about the technical aspects of the data landscape and thrive on building optimized processes that delive...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    Recruitment Dynamix (Pty) LtdPretoria, South Africa
    Leading, well establishied business has an interestin and stimulating opportunity for an experienced Data Engineer.Help stakeholders understand the data through exploration.Build and maintain secur...Show moreLast updated: 25 days ago
    • Promoted
    Data Streaming Platform Engineer – Midrand / Semi-Remote – R650 PH

    Data Streaming Platform Engineer – Midrand / Semi-Remote – R650 PH

    E-MergePretoria, South Africa
    Remote
    Our client is currently seeking a.Data Streaming Platform Engineer.Youll work with cutting-edge technologies to power connected vehicle systems, predictive maintenance, manufacturing analytics, and...Show moreLast updated: 30+ days ago
    • Promoted
    Cloud Network Engineer

    Cloud Network Engineer

    Baraka IT Solutions (Pty) LtdPretoria, South Africa
    Cloud Network Engineer : This role will be centered around the following functions : a) Statistical data processing (Data Mining) b) Artificial Intelligence (AI), including generative AI c) Predictiv...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer - 2053

    Data Engineer - 2053

    JorDan HRPretoria, Gauteng, South Africa
    Data Engineer with Degree and Certified AWS Cloud Practitioner or similar and knowledge of data formats such as Parquet AVRO JSON XML CSV etc. REST APIs required for a reputable global company based...Show moreLast updated: 3 days ago
    • Promoted
    Senior Data Engineer

    Senior Data Engineer

    OfferZenPretoria, South Africa
    Senior Data Engineer Are you a talented senior data engineer looking for your Dream Developer Job? OfferZen, the developer jobs marketplace, reverses the job search process.We connect developers wi...Show moreLast updated: 2 days ago
    • Promoted
    Data Engineer

    Data Engineer

    Oostelike Personeel KonsultanteCenturion, South Africa
    Legal requirements : Must be a South African Citizen with a clean criminal background and ITC (no bad debt) - checks will be done on shortlisted applicants. Appointment : Permanent, full-time position...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    AFGRI Group HoldingsCenturion, Gauteng, South Africa
    This roles responsibility is to design develop and maintain data-based solutions including ensuring that the operationalization of data pipelines and data stores are high-performing efficient organ...Show moreLast updated: 30+ days ago
    • Promoted
    Data Engineer

    Data Engineer

    BramblesWorkFromHome, South Africa
    CHEP helps move more goods to more people, in more places than any other organization on earth via our 347 million pallets, crates and containers. We employ approximately 13,000 people and operate i...Show moreLast updated: 3 days ago