Build and maintain Databricks-based batch and streaming data pipelines for an enterprise AI platform. Responsibilities include lakehouse modeling, governance, data quality, unstructured document ingestion, chunking, embeddings, vector search, and LLM evaluation datasets. The role deploys cloud infrastructure on Azure or GCP, applies testing and CI/CD practices, and partners with AI and platform engineers to deliver secure, reliable data for retrieval-augmented generation systems.
Who we are:
At Focused, we move quickly to deliver quality software that achieves client outcomes and meets their customer's needs. We strategically partner with our clients to leverage our expertise in design and software, while our clients bring their own domain expertise. We work with a variety of clients from different industries, collaborating as we get new products to market, modernizing legacy systems, or helping teams learn the skills they need to be successful.
Our values:
- Listen first • We are experts in product practices but lifelong learners in the domain of our customers. We research, collaborate, and understand.
- Learn why • We ask questions and talk to users to understand problem spaces, objectives, and goals, which allows us to deeply invest and drive towards the outcomes of our clients.
- Love your craft • We love diving into a variety of domains and solving problems. We take pride in delivering value, in communicating progress, and guiding our clients to success.
We're looking for a Data Engineer to build the data foundation behind an enterprise AI platform for a global enterprise client. Agents and LLM applications are only as good as the data they can reach, and you'll own getting it there: clean, governed, and ready for retrieval. Most of your work will be in Databricks, with a strong lean toward AI use cases.
Key Responsibilities:
Data Pipelines & Platform
- Design and build batch and streaming pipelines in Databricks (Spark, Delta Lake, Workflows)
- Model data in a lakehouse architecture that serves both analytics and AI workloads
- Implement governance and access controls with Unity Catalog
- Monitor data quality and pipeline reliability
Data for AI
- Build ingestion and chunking pipelines for unstructured data (documents, PDFs, knowledge bases) that feed RAG systems
- Generate, store, and refresh embeddings; operate vector search (Databricks Vector Search or similar)
- Build datasets for LLM evaluation and help teams measure retrieval quality
- Partner with AI and platform engineers so agents get the right data with the right permissions
Cloud & Engineering Practices
- Deploy data infrastructure on Azure and/or GCP using IaC
- Bring software engineering habits to data work: version control, testing, CI/CD
Required Qualifications:
- 5+ years in data engineering
- Strong hands-on experience with Databricks, Spark, and Delta Lake
- Proficiency in Python and SQL
- Experience building production data pipelines in a cloud environment (Azure or GCP preferred)
- Experience with data modeling and data quality practices
Preferred Qualifications (Exceptional Candidates):
- Experience building data pipelines for RAG, embeddings, or vector search
- Experience with Unity Catalog and Mosaic AI
- Experience with streaming (Structured Streaming, Kafka, Event Hubs)
- Consulting or client-facing experience
What Sets Exceptional Candidates Apart:
- You've improved an AI system's answers by fixing the data underneath it
- You've handled messy unstructured data at scale
- Databricks certifications or open-source contributions
What to know before you apply:
- This role will require being in the Chicago office 3 days per week and up to 20% travel within the United States.
- Focused is unable to sponsor or take over sponsorship of the employment Visa process at this time.
- The Chicago base salary range for this role is $145,000 - $175,000.
Focused Chicago, Illinois, USA Office
We are in the recently renovated Old Post Office Building in downtown Chicago! The office has world class amenities including an 18,000 sq ft gym, indoor bocce and ping pong and a 3.5 acre rooftop with basketball and pickle ball.
Similar Jobs
AdTech • eCommerce • Food • Marketing Tech • Retail
Designs and governs scalable data and AI platforms, including batch and streaming pipelines, automation, infrastructure as code, security, observability, and cloud services. Builds reusable developer tooling and golden paths, improves reliability through monitoring and SLOs, and guides architecture decisions across complex programs. The role also coaches data teams, documents standards, and supports critical incidents.
Top Skills:
AnsibleAzureCi/CdCloudFormationContainer OrchestrationDatabricksDatadogInfrastructure As CodeTerraform
Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Architect and own marketing data infrastructure, including Databricks pipelines, Gold Layer tables, semantic models, and ETL/ELT workflows. Build trusted data products for analytics, automation, and AI agents; translate ambiguous business needs into scalable solutions; lead cross-team initiatives; ensure reliability and data quality; mentor engineers and analysts; and develop, evaluate, and monitor production LLM and agent systems using Python and SQL.
Top Skills:
Ai AgentsBigQueryClaude CodeCursorDatabricksDbtEtl/EltLlmsPythonSalesforceSnowflakeSQLStreamlit
Artificial Intelligence • Professional Services • Business Intelligence • Consulting • Cybersecurity • Generative AI
Leads global teams developing production AI, GenAI, RAG, agentic workflow, and MLOps solutions for health systems and health plans. Translates clinical, financial, and operational challenges into HIPAA-compliant use cases involving clinical decision support, population health, revenue cycle, research, and optimization. Oversees healthcare data science engagements, model governance, executive stakeholder communications, client delivery, and team development.
Top Skills:
Anthropic ClaudeAws BedrockAws SagemakerAzure Health Data ServicesAzure Machine LearningAzure Openai ServiceChromaCi/CdDatabricksGitGoogle Vertex AiKerasLangchainLlamaindexMlopsNoSQLPandasPineconePythonRagScikit-LearnSemantic KernelSnowflakeSQLTransformers
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
