Designs, builds, and maintains scalable data pipelines on Databricks, focusing on data quality, integrity, and governance, while collaborating with teams to deliver data solutions.
Description
At CDW, we make it happen, together. Trust, connection, and commitment are at the heart of how we work together to deliver for our customers. It's why we're coworkers, not just employees. Coworkers who genuinely believe in supporting our customers and one another. We collectively forge our path forward with a level of commitment that speaks to who we are and where we're headed. We're proud to share our story and Make Amazing Happen at CDW. The Senior Data Engineer designs, builds, and maintains scalable data pipelines and Lakehouse solutions on the Databricks platform to support enterprise data and AI initiatives. This role partners closely with architects, analytics, and data science teams to deliver high-quality, reliable data products. The engineer operates with significant autonomy and contributes deep technical expertise across the full data lifecycle.
* What you'll do:
* Build and maintain scalable data pipelines, ETL and ELT processes, and data models within the Databricks platform.
* Design, develop, and deploy data and AI solutions using Databricks, Spark, Delta Lake, and related technologies.
* Develop batch and streaming pipelines using tools such as Databricks Workflows and Azure Data Factory.
* Design logical data flow diagrams and normalized schemas, implementing Lakehouse patterns such as the Medallion Architecture (Bronze, Silver, Gold layers).
* Ensure data quality, integrity, security, and governance throughout the data lifecycle, including use of Unity Catalog.
* Optimize Spark jobs and data transformations through effective partitioning, caching, and join strategies.
* Monitor pipeline execution, identify failures, and troubleshoot complex data processing issues.
* Collaborate with data architects, analysts, data scientists, and business stakeholders to understand requirements and deliver solutions.
* Support documentation of data processes, standards, and data flows.
What we expect of you:
* 5 Years of experience designing, developing, and deploying data solutions on the Databricks platform.
* Proficiency in Python, including PySpark, and SQL.
* Hands-on experience with Spark, Delta Lake, and Lakehouse architectures.
* Experience implementing data quality, governance, and security practices across data pipelines.
* Familiarity with machine learning concepts, tools, and libraries such as TensorFlow, PyTorch, Scikit-learn, and MLflow is a plus.
* Experience configuring and integrating external AI models and working with AI governance and monitoring tools is a plus.
* Experience with asynchronous programming patterns in Python for building scalable data or AI workloads is a plus.
* Strong problem-solving, collaboration, and communication skills.
* CDW is committed to being an AI-fluent organization
* We're looking for people who bring curiosity, a learner's mindset, and a willingness to engage with ever-evolving technology and tools. We value adopting AI as a partner, openness to experimentation, and a shared interest in learning together on AI. Our goal is to create a culture where AI enhances- not replaces- human creativity and decision-making. You don't need to be an expert today; what matters is your readiness to explore, adapt, and grow with us as we integrate AI responsibly and effectively into our work.Additionally, CDW is committed to fostering an equitable, transparent, and respectful hiring process for all applicants. During our application process, our goal is to understand your experience, strengths, skills, and qualifications. As an AI forward company, we see AI not just as a tool, but as a catalyst for new ways of thinking, creating, and communicating. We encourage candidates to embrace an AI mindset, one that's curious, adaptive, and ready to explore what's possible. We welcome thoughtful use of AI to expand your perspective and elevate how you share your story, while ensuring your application remains rooted in your own background, judgment, and voice.
* About Us
* CDW is a Fortune 500 technology solutions provider that helps businesses, government, education, and healthcare organizations achieve what's possible through technology. What makes CDW different isn't just what we do- it's how we do it. At CDW we act as one- building trust, speaking candidly, and working together to achieve more. We play to win- focusing on what matters most and delivering for our customers. And we think forward- staying curious, moving fast, and continuously learning. We believe meaningful work happens when people feel supported, heard, and empowered to contribute. That's why we think of ourselves as coworkers, not just employees- working together to solve complex challenges and deliver real impact for our customers and communities. As a full-stack, full-lifecycle technology partner, CDW brings deep expertise, strong relationships, and broad industry knowledge to help turn ideas into outcomes. When you join CDW, you become part of a collaborative environment where your work matters, your growth is supported, and your contributions help shape what's next. Together, we deliver the full promise of what technology can do.
* Together, we Make Amazing Happen.
* CDW is an equal opportunity employer. All qualified applicants will receive consideration for employment without regards to race, color, religion, sex, sexual orientation, gender identity, national origin, disability status, protected veteran status or any other basis prohibited by state and local law.
At CDW, we make it happen, together. Trust, connection, and commitment are at the heart of how we work together to deliver for our customers. It's why we're coworkers, not just employees. Coworkers who genuinely believe in supporting our customers and one another. We collectively forge our path forward with a level of commitment that speaks to who we are and where we're headed. We're proud to share our story and Make Amazing Happen at CDW. The Senior Data Engineer designs, builds, and maintains scalable data pipelines and Lakehouse solutions on the Databricks platform to support enterprise data and AI initiatives. This role partners closely with architects, analytics, and data science teams to deliver high-quality, reliable data products. The engineer operates with significant autonomy and contributes deep technical expertise across the full data lifecycle.
* What you'll do:
* Build and maintain scalable data pipelines, ETL and ELT processes, and data models within the Databricks platform.
* Design, develop, and deploy data and AI solutions using Databricks, Spark, Delta Lake, and related technologies.
* Develop batch and streaming pipelines using tools such as Databricks Workflows and Azure Data Factory.
* Design logical data flow diagrams and normalized schemas, implementing Lakehouse patterns such as the Medallion Architecture (Bronze, Silver, Gold layers).
* Ensure data quality, integrity, security, and governance throughout the data lifecycle, including use of Unity Catalog.
* Optimize Spark jobs and data transformations through effective partitioning, caching, and join strategies.
* Monitor pipeline execution, identify failures, and troubleshoot complex data processing issues.
* Collaborate with data architects, analysts, data scientists, and business stakeholders to understand requirements and deliver solutions.
* Support documentation of data processes, standards, and data flows.
What we expect of you:
* 5 Years of experience designing, developing, and deploying data solutions on the Databricks platform.
* Proficiency in Python, including PySpark, and SQL.
* Hands-on experience with Spark, Delta Lake, and Lakehouse architectures.
* Experience implementing data quality, governance, and security practices across data pipelines.
* Familiarity with machine learning concepts, tools, and libraries such as TensorFlow, PyTorch, Scikit-learn, and MLflow is a plus.
* Experience configuring and integrating external AI models and working with AI governance and monitoring tools is a plus.
* Experience with asynchronous programming patterns in Python for building scalable data or AI workloads is a plus.
* Strong problem-solving, collaboration, and communication skills.
* CDW is committed to being an AI-fluent organization
* We're looking for people who bring curiosity, a learner's mindset, and a willingness to engage with ever-evolving technology and tools. We value adopting AI as a partner, openness to experimentation, and a shared interest in learning together on AI. Our goal is to create a culture where AI enhances- not replaces- human creativity and decision-making. You don't need to be an expert today; what matters is your readiness to explore, adapt, and grow with us as we integrate AI responsibly and effectively into our work.Additionally, CDW is committed to fostering an equitable, transparent, and respectful hiring process for all applicants. During our application process, our goal is to understand your experience, strengths, skills, and qualifications. As an AI forward company, we see AI not just as a tool, but as a catalyst for new ways of thinking, creating, and communicating. We encourage candidates to embrace an AI mindset, one that's curious, adaptive, and ready to explore what's possible. We welcome thoughtful use of AI to expand your perspective and elevate how you share your story, while ensuring your application remains rooted in your own background, judgment, and voice.
* About Us
* CDW is a Fortune 500 technology solutions provider that helps businesses, government, education, and healthcare organizations achieve what's possible through technology. What makes CDW different isn't just what we do- it's how we do it. At CDW we act as one- building trust, speaking candidly, and working together to achieve more. We play to win- focusing on what matters most and delivering for our customers. And we think forward- staying curious, moving fast, and continuously learning. We believe meaningful work happens when people feel supported, heard, and empowered to contribute. That's why we think of ourselves as coworkers, not just employees- working together to solve complex challenges and deliver real impact for our customers and communities. As a full-stack, full-lifecycle technology partner, CDW brings deep expertise, strong relationships, and broad industry knowledge to help turn ideas into outcomes. When you join CDW, you become part of a collaborative environment where your work matters, your growth is supported, and your contributions help shape what's next. Together, we deliver the full promise of what technology can do.
* Together, we Make Amazing Happen.
* CDW is an equal opportunity employer. All qualified applicants will receive consideration for employment without regards to race, color, religion, sex, sexual orientation, gender identity, national origin, disability status, protected veteran status or any other basis prohibited by state and local law.
CDW Chicago, Illinois, USA Office
625 W Adams St., Chicago, IL, United States, 60661
Similar Jobs at CDW
Information Technology
Manage project scopes and timelines, coordinate Agile sprints, coach teams, facilitate communications, resolve conflicts, and ensure deliverable quality.
Top Skills:
AgileItil 4Safe AgileScrum
Information Technology
The Senior Solution Architect will design and develop solutions using the Microsoft Azure data stack, collaborating with sales teams, and supporting pre-sales activities while mentoring peers.
Top Skills:
Azure Data StackAzure SqlAzureMicrosoft FabricSynapse
Information Technology
The Senior Accountant will apply accounting principles to analyze vendor-funded incentives, ensure compliance with US GAAP, and support cross-functional business decisions while preparing financial reports and resolving discrepancies.
Top Skills:
Asc 705-20ExcelUs Gaap
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
