Lead end-to-end data engineering on GCP: design scalable data platforms and architectures (lake/lakehouse/warehouse), build and optimize BigQuery/Spark pipelines, ensure data quality/security, enable ML workflows with Data Science, mentor engineers, and own delivery and performance.
We are seeking a Senior Data Engineer (Lead) to drive and own end-to-end data engineering initiatives. This role will lead all data engineering efforts, working closely with Data Science and Analytics teams to design scalable data platforms, enable advanced analytics, and support machine learning use cases.
The ideal candidate will bring deep expertise in cloud data engineering (GCP), strong data modeling capabilities, and proven experience in leading enterprise-grade data solutions.
Responsibilities- Lead and manage end-to-end Data Engineering delivery across projects and initiatives
- Act as the primary technical owner for data pipelines, architecture, and platform design
- Mentor and guide a team of data engineers, ensuring best practices and coding standards
- Design, build, and optimize scalable data pipelines on GCP
- Define and implement modern data architectures (data lake, lakehouse, warehouse)
- Ensure high performance, reliability, and data quality across pipelines
- Partner closely with Data Science teams to enable ML/AI workflows
- Translate business and modeling requirements into optimized data structures
- Support feature engineering, model training, and deployment pipelines
- Design logical and physical data models for analytics and ML use cases
- Implement dimensional modeling (Star/Snowflake schemas) and data vault where applicable
- Optimize datasets for performance, scalability, and usability
- Build and manage solutions using GCP services such as: BigQuery, Cloud Composer (Airflow), Cloud Storage, Dataproc
- Ensure security, governance, and cost optimization on GCP
Required Qualification and Skills:
- 8+ years of experience in Data Engineering, with leadership experience
- Strong expertise in GCP ecosystem and services
- Proficiency in SQL, Python, and/or Scala
- Hands-on experience with ETL frameworks and distributed processing
- Solid experience in Dimensional modeling, Data warehousing concepts, Data structures for ML/analytics
- Experience with Apache Spark and Real-time and batch processing frameworks
- Experience working with cross-functional teams (Data Science, Analytics, Business)
- Proven ability to lead, mentor, and drive delivery
- Strong ownership mindset with leadership capabilities
- Excellent problem-solving and architectural thinking
- Ability to operate in a fast-paced, collaborative environment
Preferred Qualifications:
- Experience in ML data pipelines / feature stores
- Knowledge of data governance, lineage, and quality frameworks
- Exposure to healthcare/payor domain (nice to have)
- Certifications in GCP (Professional Data Engineer)
Similar Jobs
Cloud • Greentech • Social Impact • Software • Consulting
Lead execution of social media, PR, and internal communications. Manage community engagement, social listening, content publishing, KPI tracking, byline and award submissions, employee advocacy, and executive updates. Collaborate with cross-functional teams and PR agency, maintain brand consistency, and introduce AI-driven communications efficiencies.
Top Skills:
Ai-Driven Communications ToolsSocial Listening ToolsSocial Media Automation And Scheduling ToolsSocial Media Management Tools
Fintech
Design, develop, and deliver scalable .NET back-end and front-end solutions within an Agile, DevOps-oriented team. Lead technical design, code reviews, testing-first practices, CI/CD pipelines, containerized deployments, and ensure non-functional requirements (security, performance, scalability, maintainability). Mentor peers and collaborate across product, design, and architecture to produce high-quality software.
Top Skills:
.Net.Net Core.Net Framework 4.8ActivebatchAzureAzure App ServiceAzure Container InstancesAzure DevopsC#C# 10CqrsDockerDomain Driven DesignGitKubernetesMs Sql ServerOci ContainersOpenshiftPowershellRestSoapSonarqubeSsisT-SqlVbVisual Studio Code
Fintech
Administer and support Inspira's hybrid cloud and on-prem infrastructure including Azure, VMware, networking, security, backups/DR, AD, SQL Server, containers/Istio and Helm. Implement IaC and CI/CD, monitor cost and performance, resolve infrastructure incidents, maintain runbooks, cross-train staff, and support integrations with development and client service teams.
Top Skills:
Azure Api ManagementAzure BackupAzure FunctionsAzure Notification HubAzure Service BusAzure Site RecoveryAzure VnetsBashCi/CdCisco MerakiContainersExchangeHelmInfrastructure As CodeIntuneIstio Service MeshJavaScriptJSONLogic AppsMicrosoft 365Microsoft Active DirectoryAzurePowershellPrivate EndpointsPythonRestRoute ServersRoute TablesSharepointSQL ServerTeamsVeeamVirtual MachinesVMwareVmware Site RecoveryVpn Gateway
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
