Senior Data Engineer
Job Summary
As a Senior Data Engineer, you will design, build, and optimize high-performance data pipelines and platforms powering analytics, dashboards, and AI models across the enterprise. Your mission is to deliver accessible, reliable, and production-ready data to support Data Scientists and Analysts. You will champion automation, scalability, and best practices to accelerate the company’s data and AI maturity.
Responsibilities
- Design and maintain scalable end-to-end data pipelines.
- Optimize data infrastructure (Lakehouse/Warehouse) performance.
- Collaborate on MLOps lifecycle and AI model-serving pipelines.
- Ensure data quality, governance, and strict security compliance.
Required Skills
- 5+ years in Data Engineering
- Expertise in SQL, Python, Spark, and Airflow
- Deep AWS stack proficiency (Redshift, Glue, S3, etc.)
- Experience with Databricks and MLOps tools
Job Details
Responsibilities
1. Data Pipeline Engineering & Automation
- Design, build, and maintain scalable, end-to-end pipelines for data ingestion, transformation, and delivery.
- Automate ETL/ELT workflows (Airflow, Glue, Step Functions, Prefect) to eliminate manual intervention.
- Implement validation, version control, and rollback mechanisms.
- Build self-healing, auto-scaling pipelines ensuring operational resilience.
2. Data Infrastructure & Performance Optimization
- Develop and optimize lakehouse and warehouse architectures using Databricks, Snowflake, Redshift, S3, EMR, Glue, and Lake Formation.
- Apply best practices in data partitioning, indexing, and caching to improve query speed and manage costs.
- Integrate monitoring, alerting, and logging (CloudWatch, Prometheus, Grafana) for proactive issue resolution.
- Collaborate with the Data Architect to ensure scalability, efficiency, and alignment with enterprise standards.
3. AI & Analytics Enablement
- Build data foundations for forecasting, segmentation, retention, and KPI decomposition models.
- Partner with Data Scientists to develop model-serving pipelines with automated retraining and versioning.
- Create reusable feature stores, model registries, and tracking frameworks supporting the MLOps lifecycle.
- Enable AI-assisted analytics through natural language query, LLM integration, and automated insights.
4. Data Quality, Governance & Documentation
- Maintain detailed documentation of pipelines, lineage, and metadata.
- Enforce access control, encryption, and compliance with PDPA, GDPR, and internal governance.
- Develop automated quality checks, anomaly detection, and audit trails.
Requirements
- Min 5 years in data engineering, pipeline design, or infrastructure operations.
- Proven experience managing large-scale (multi-terabyte) datasets with high uptime.
- Expert in SQL, Python, and frameworks such as Spark, Hadoop, dbt, and Airflow.
- Strong knowledge of AWS stack (Redshift, Glue, S3, EMR, Athena, Lambda, Lake Formation).
- Familiar with Databricks, Snowflake, and MLOps tools (SageMaker, MLflow, Vertex AI).
- Certification in AWS Certified Data Engineer will be an added advantage.