Senior Data Engineer

• Design and build scalable cloud-native data platforms from greenfield to production • Design and implement near-real-time ingestion pipelines using event-driven patterns • Define and enforce data platform standards including Data Lake/Lakehouse principles, medallion architecture, and data contracts • Refactor, optimize, and modernize Spark and PySpark scripts for performance and maintainability • Introduce best practices for code quality, testing, and CI/CD across data pipelines • Drive adoption of AI tooling and agentic workflows within the data engineering team • Ensure data quality, observability, and reliability across pipelines and platforms • Design and deliver self-service tooling and microservices that abstract platform complexity • Collaborate directly with the Customer’s scientific and engineering teams • Monitor platform performance, reliability, and cloud infrastructure costs • 5+ years of experience in data engineering • 3+ years of production experience with Databricks and Delta Lake • Strong expertise in medallion architecture and event-driven asynchronous pipeline design • Advanced proficiency in Python, PySpark, and SQL • Deep hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization • Strong AWS expertise including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway • Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles • Practical experience with Git workflows, pull request reviews, and automated testing • Experience configuring secure CI pipelines using OIDC or service principals • Practical MLflow experience including experiment tracking, model registration, and reproducibility practices • Strong understanding of data quality, lineage, provenance, schema contracts, and versioning • Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX • Experience with monitoring, observability, and cloud cost optimization • Upper-Intermediate English level or higher

WILL BE A PLUS

• Experience packaging workloads in Docker containers and running batch jobs on AWS • Familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy • Experience building feature sets and train/test splits that avoid data leakage • Experience with regulated or high-security environments including data residency requirements • Experience disabling or restricting AI features in secured environments

PERSONAL PROFILE

• Strong analytical mindset • Ownership and accountability • Ability to communicate complex technical concepts clearly • Proactive approach to problem-solving • Attention to scalability and maintainability • Collaborative and team-oriented attitude