Lead the development of data pipelines and transformations in Azure Databricks, converting Scala programs to PySpark while leveraging various Azure technologies.
Job Description :
Location - US (remote) . East coast preferable
Primary Skill :
Technical Lead -Data Engneering
- Azure databricks- Build Data Pipelines using ADB-Leveraging Python and Spark
- Build Data Transformation in ADB
- Convert Scala Pgms to Py Spark Pgms
- Understand the SCALA program in DataBricks
- ADF
- Secondary Skills:
- Azure data lake gen 2
- Synapse analytics (dedicated SQL pool)
- Exp in delta Lake
- Pyspark
Nice to Have : Tidal Scheduler , Exp in Healthcare Domain
Hexaware Technologies Chicago, Illinois, USA Office
Chicago, United States, 0
Similar Jobs
Information Technology • Consulting
Lead design and implementation of data ingestion, data lake, and ETL solutions using Big Data technologies. Build performant cloud data pipelines, integrate multiple data sources, optimize query performance, enable analytics and data science teams, and lead offshore engineering teams.
Top Skills:
Apache FlumeSparkApache StormAPIsAWSAzureBig DataCassandraData LakeDatabricksETLGCPHbaseJavaKafkaMongoDBNoSQLPysparkPythonRabbitMQScalaSpark StreamingSQLWeb ServicesXML
Information Technology • Consulting
Lead development of enterprise data and analytics systems using Snowflake and Python. Build and optimize data pipelines on AWS (Lambda, Step Functions, ECS), write advanced SQL, and apply domain knowledge of asset management and financial markets.
Top Skills:
Amazon EcsAws LambdaAws Step FunctionsPythonSnowflakeSQL
Information Technology • Consulting
Lead data engineer responsible for designing, building, and operating production data pipelines using Python, Airflow, Snowflake, DBT, and AWS. Automate ETL, implement CI/CD, testing and DBT validations, troubleshoot pipelines, and work client-facing to deliver reliable data solutions.
Top Skills:
Ai IdeApache AirflowAws EcsAws IamAws S3Ci/CdCsvDbtGitJinjaJSONPythonSnowflakeSQLXML
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
