Senior Data Platform Engineer II (Databricks)

Primary Duties:

• Develop and implement scalable and performant solutions.
• Partner, as a peer, with Engineering Managers, Product Managers, and stakeholders throughout Aledade to develop and execute technical roadmaps using Agile processes.
• Mentor and coach more junior engineers including thorough pull request reviews for other developers and be receptive to critical feedback on your own work.

Minimum Qualifications:

• BS/BTech (or higher) in Computer Science, Engineering or a related field or equivalent experience.
• 6+ years experience as an engineer building and optimizing highly scalable distributed data systems (e.g., Databricks, Spark, or Snowflake).
• 3+ years of experience working with SQL and data modeling on large multi-table data sets.
• 3+ years of experience acting as a trusted technical decision-maker in a team setting, solving for short-term and long-term business value.
• 3+ years of experience coaching other engineers.

Preferred KSA’s:

Platform & Infrastructure (The "Databricks/Cloud" Core)
• Databricks & Lakehouse Architecture: Deep expertise in managing Databricks workspaces, including Unity Catalog for data governance, lineage, and fine-grained access control.
• Infrastructure as Code (IaC): Advanced proficiency with Terraform (or similar) to automate the provisioning and scaling of Databricks clusters, cloud resources (AWS preferred), and networking.
• Snowflake Proficiency (Nice-to-Have): Experience managing Snowflake environments, specifically focusing on warehouse cost optimization, security integration, and secure data sharing.
• Modern Database Internals: In-depth knowledge of distributed systems, including partitioning, liquid clustering/Z-Ordering, sharding, and high-availability strategies for petabyte-scale data.
Performance, Reliability & DevOps
• Observability & Optimization: Proven track record in performance monitoring and query tuning for distributed workloads to ensure system reliability and cost-efficiency.
• Data Engineering Lifecycle: Experience designing and optimizing high-throughput ETL/ELT pipelines and ingestion systems (batch and streaming) using Spark.
• Deployment & Orchestration: Experience building robust CI/CD pipelines for data infrastructure and deploying services using containerization (Docker, Kubernetes).


Security, Compliance & Domain Knowledge
• Sensitive Data Handling: Expertise in building systems that handle protected information, with specific experience in HIPAA and SOX compliance frameworks.
• Healthcare Data Expertise: Experience navigating health-tech data complexities, such as Electronic Health Records (EHR), clinical data formats (HL7/FHIR), and claims data.

Physical Requirements:

• Sitting for prolonged periods of time. Extensive use of computers and keyboard. Occasional walking and lifting may be required.