Senior /Staff Data & Cloud Platform Engineer
Explicitly requires AI-assisted (vibe) coding skills and use of AI tools to automate workflows and improve efficiency.
About the Role
Senior/Staff Data & Cloud Platform Engineer responsible for designing, building, and operating scalable data ingestion and processing platforms across cloud and on-prem environments. The role focuses on ETL/ELT pipelines, workflow orchestration, infrastructure automation, security, and cross-functional delivery to support TB-scale manufacturing and analytics workloads.
Job Description
Role
Senior/Staff Data & Cloud Platform Engineer focused on building scalable, secure data platforms for ingestion, processing, storage, and analytics across AWS, GCP, and on-prem environments. The role emphasizes pipeline engineering, workflow orchestration, schema and metadata management, infrastructure automation, and operational reliability for large-scale manufacturing and engineering data.
Key Responsibilities
- Build scalable ingestion pipelines for structured, semi-structured, and engineering data sources.
- Develop batch, scheduled, and event-driven workflows using Apache Airflow and cloud-native services.
- Implement ETL/ELT frameworks, reusable pipeline patterns, incremental ingestion, and data quality checks.
- Implement schema registry/versioning, schema evolution, and compatibility controls.
- Design delta detection, CDC-style processing, watermarking, and incremental refresh mechanisms.
- Extract and normalize manufacturing identifiers (e.g., Lot, Die, Wafer, Test, Flow, Product, Step).
- Set up secure cloud landing zones for AWS, GCP, and on-prem validation and analytics pipelines.
- Automate infrastructure provisioning with Terraform and CI/CD, and manage platform operations for containerized workloads (Kubernetes, Docker).
- Implement IAM, RBAC, hybrid connectivity, secrets management, and cost optimization for TB-scale data.
- Monitor, troubleshoot, and optimize platform reliability, performance, and cost.
- Partner cross-functionally with IT, AI/Data Science, Product Engineering, and manufacturing teams.
Requirements
- Bachelor’s degree in Computer Science, Data Engineering, Cloud Engineering, Information Systems, or related technical field.
- 5+ years of experience in data engineering, cloud/platform engineering, DevOps, or infrastructure engineering.
- Strong hands-on experience with Python and SQL.
- Practical experience with GCP services (Dataflow, Pub/Sub, Cloud Functions, BigQuery, Cloud Storage, IAM) and AWS services (Glue, Lambda, S3, Redshift, IAM).
- Experience with Terraform, Kubernetes, Docker, CI/CD, cloud networking, and operational automation.
- Experience implementing schema/metadata management, incremental processing (CDC/delta), and data quality validation.
- Knowledge of cloud security, IAM/RBAC, hybrid connectivity, and cost management for large-scale data.
- Proven ability to leverage AI-assisted (vibe) coding techniques and AI tools to automate workflows and improve efficiency.
Preferred
- Domain exposure to semiconductor, manufacturing, validation, reliability, test, yield, or RCA data environments.
- Familiarity with AI/analytics platform foundations, data governance, lineage, and governed data products.
- Certifications in AWS, GCP, Kubernetes, Terraform, or data engineering are a plus.
Tools & Technologies (explicitly referenced)
Apache Airflow, Python, SQL, GCP (Dataflow, Pub/Sub, Cloud Functions, BigQuery, Cloud Storage, IAM), AWS (Glue, Lambda, S3, Redshift, IAM), Terraform, Kubernetes, Docker, CI/CD.