Lead Data Engineer
- Full-time
We are seeking a highly capable Data Engineer with strong experience in scalable data platforms, large-scale scraping, LLM integrations, and agentic AI systems. This role is ideal for a hands‑on builder who enjoys solving complex data challenges and developing intelligent automation solutions that operate reliably in production environments.
You will play a key role in designing and building modern data pipelines, AI-powered workflows, and autonomous systems that can interact with tools, APIs, databases, and enterprise data sources.
Responsibilities
- Design and build scalable batch and near-real-time data pipelines.
- Experience specifically with Claude Code.
- Ingest and process data from APIs, databases, files, event streams, and web sources.
- Develop and maintain large-scale scraping and data acquisition systems.
- Clean, validate, transform, and model data for analytics and AI applications.
- Build LLM-powered workflows and agentic AI solutions using:
- Structured outputs
- Retrieval-augmented generation (RAG)
- API and database integrations
- Support modern data lake, warehouse, and lakehouse architectures.
- Implement monitoring, observability, schema validation, and data quality controls.
- Collaborate with engineering teams to deliver secure, scalable, and maintainable production systems.
Qualifications and experience
- Strong experience with Python and/or Node.js/TypeScript.
- Advanced SQL and relational database skills.
- Experience with AWS or similar cloud platforms.
- Hands‑on experience with data lakes, warehouses, or lakehouse technologies.
- Practical experience using LLM platforms such as OpenAI, Anthropic, Gemini, Bedrock, or similar.
- Experience building agentic workflows involving:
- Tool use
- Retrieval systems
- Memory
- Experience with scraping frameworks such as Playwright, Puppeteer, Scrapy, or Selenium.
- Strong understanding of APIs, containers, CI/CD, testing, and secure engineering practices.
The reference number for this position is NG61354 which is a contract remote role offering a salary of up to R650 per hour salary negotiable based on experience.
#J-18808-Ljbffr