Lead the development of data pipelines and transformations in Azure Databricks, converting Scala programs to PySpark while leveraging various Azure technologies.
Job Description :
Location - US (remote) . East coast preferable
Primary Skill :
Technical Lead -Data Engneering
- Azure databricks- Build Data Pipelines using ADB-Leveraging Python and Spark
- Build Data Transformation in ADB
- Convert Scala Pgms to Py Spark Pgms
- Understand the SCALA program in DataBricks
- ADF
- Secondary Skills:
- Azure data lake gen 2
- Synapse analytics (dedicated SQL pool)
- Exp in delta Lake
- Pyspark
Nice to Have : Tidal Scheduler , Exp in Healthcare Domain
Hexaware Technologies Chicago, Illinois, USA Office
Chicago, United States, 0
Similar Jobs
Information Technology • Software • Analytics • Business Intelligence
Leads teams developing and maintaining complex cloud-based big data platforms and pipelines. Owns architecture, delivery strategy, data modeling, governance, observability, reliability, security, and cost optimization. Translates business requirements into technical plans, oversees coding and production support, manages risks and client communication, establishes quality and CI/CD practices, and mentors engineers while providing cross-team architectural governance.
Top Skills:
AlertingCi/CdCloud ComputingData CatalogingData GovernanceData LineageData ModelingEvent-Driven PipelinesGCPGitKappa ArchitectureLambda ArchitectureLoggingMicroservicesMonitoringObservabilityPythonScrumSQL
Information Technology • Consulting
Leads hands-on big data processing, data engineering, and analytics workflows using Azure Databricks, Python, and SQL. Develops insights and reporting with Power BI, Tableau, and Excel. Analyzes complex datasets, communicates findings to business stakeholders, manages competing priorities, meets deadlines, and promotes continuous improvement.
Top Skills:
Azure DatabricksExcelPower BIPythonSQLTableau
Information Technology • Consulting
Design, build, and optimize end-to-end ETL/ELT pipelines using Python and PySpark for large structured and unstructured datasets. Ensure data quality through validation and reconciliation, troubleshoot performance issues, implement CI/CD and code review practices, and collaborate with IT and business stakeholders to translate requirements into scalable data solutions.
Top Skills:
AWSAzureBitbucketCi/CdData WarehousingDistributed ComputingEtl/EltGCPGitHadoopMicrosoft Sql ServerPysparkPythonSparkSQL
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
