Senior ML Data Engineer (P508)

The Relevancy Sciences Team is responsible for creating relevant and personalized customer experiences for Kroger's E-commerce platform, which ranks among the top 10 ecommerce companies in the US. We generate trillions of recommendations at scale and deliver them to millions of Kroger customers daily. Our team maintains a comprehensive portfolio of machine learning solutions for search & product recommendations. We are seeking a talented and experienced Senior ML Data Engineer to join our data science team, with specialized expertise in building search and recommender systems.

You will architect, build, and operate the critical data infrastructure that powers our machine learning models, spanning from feature engineering to training data generation. This role serves as the bridge between ML requirements and production data systems, with ownership of feature stores, training/evaluation pipelines, and ML-specific data operations. You will enable data scientists to iterate rapidly while ensuring production-grade reliability and scalability.

1. Feature Store Operations & Governance (40%)

• Own the feature request lifecycle from intake through deployment, driving reusability and maintaining a searchable feature catalog
• Design and build scalable feature pipelines that compute features from diverse sources (BigQuery, Azure Data Lake) and write to Feature Store infrastructure (Vertex AI Feature Store + BigQuery)
• Build streaming feature engineering pipelines using Apache Beam/Dataflow for real time feature computation and low-latency model serving with sub-second data freshness
• Ensure point-in-time correctness and online/offline feature consistency to prevent data leakage
• Implement drift detection, data quality monitoring, and alerting mechanisms
• Develop self-service tools and templates that enable teams to independently create features

1. Training & Evaluation Data Pipelines (30%)

• Build automated pipelines that generate ML-ready training datasets by combining features with labeled target variables
• Implement point-in-time correctness logic and sophisticated sampling strategies to ensure balanced, representative datasets
• Maintain comprehensive dataset versioning for full traceability across model versions
• Generate detailed evaluation reports with performance metrics segmented by business dimensions
• Support operations across both Azure and Vertex AI environments during platform migration

1. ML Data Operations & Reliability (20%)

• Serve as Tier 2/3 on-call responder for feature data quality incidents, diagnosing and resolving pipeline failures and performance issues
• Maintain comprehensive lineage tracking and metadata management for full data traceability
• Support regulatory compliance through proper data governance and documentation

1. Standards, Education & Collaboration (10%)

• Establish and enforce feature naming conventions, data quality thresholds, and point-in-time correctness patterns
• Conduct workshops on feature engineering best practices and provide expert guidance on feature design
• Partner with Data Scientists, ML Engineers, Data Engineering, and MLOps teams to optimize infrastructure and align with technical strategy

Required Qualifications:

• 3+ years of hands-on experience building and maintaining ML data pipelines in production environments with demonstrated expertise in scaling and reliability
• Expert-level SQL skills and advanced Python programming capabilities with experience in data processing frameworks and ML libraries
• Proven experience with cloud data platforms, with strong preference for GCP ecosystem including BigQuery, Dataflow, Vertex AI Feature Store, and associated ML services
• Deep understanding of end-to-end ML workflows including training data preparation, model evaluation methodologies, and serving infrastructure requirements
• Production operations mindset with experience in monitoring, alerting, on-call responsibilities, and meeting SLA commitments

Strongly Preferred Qualifications:

• Hands-on experience with Feature Store platforms such as Vertex AI Feature Store, Feast, Tecton, or similar enterprise solutions
• Deep knowledge of point-in-time correctness principles, temporal joins, and time-series data modeling best practices
• Multi-cloud experience with both Azure and GCP platforms, including data migration and hybrid cloud architectures
• Strong familiarity with core ML concepts including feature engineering, label creation, train/test/validation splits, and data leakage prevention
• Background spanning both analytics engineering and ML-specific data engineering with understanding of the unique requirements of each domain

Success Indicators:

• Improved Data Science Productivity:Data Scientists spend significantly less time on data preparation and infrastructure concerns, enabling more focus on model development and experimentation
• Increased Feature Reuse:Measurable increase in feature reuse across multiple models and teams, reducing redundant development effort and improving consistency
• Reliable Automation:Training and evaluation data generation processes operate reliably with minimal manual intervention and high uptime
• Efficient Incident Response:Data quality incidents are triaged quickly with clear escalation paths and rapid resolution times
• Accelerated ML Iteration:Overall ML model development and iteration velocity improves measurably across all teams using the platform

#LI-SSS