Design, build, and optimize scalable cloud-based data pipelines using PySpark and Databricks. Develop and maintain ETL/ELT workflows, model data for warehousing in AWS Redshift, and tune performance. Troubleshoot production issues and work with Python and SQL to process large-scale distributed data. Prior domain experience in e-commerce, airlines, transportation, or logistics is desirable.
Data Engineer – PySpark, Databricks & AWS Redshift
Location: Chicago, IL (locals Only)
Employment Type: W2 Contract
Job Description
We are seeking an experienced Data Engineer with strong hands-on expertise in PySpark, Databricks, and AWS Redshift to join our data engineering team. The ideal candidate will have a solid background in designing, building, and optimizing scalable data pipelines and data processing solutions in cloud-based environments.
Requirements
Required Qualifications
Atleast 7 plus years of Strong professional experience as a Data Engineer working with large-scale data platforms.
Extensive hands-on experience with PySpark.
Strong experience with Databricks, including developing and optimizing production data pipelines.
Strong hands-on experience with AWS Redshift.
Proficiency in Python and SQL.
Experience building and maintaining ETL/ELT pipelines and data processing workflows.
Strong understanding of data warehousing, data modeling, and distributed data processing concepts.
Experience working with cloud-based data platforms, preferably AWS.
Strong troubleshooting, performance-tuning, and problem-solving skills.
Prior experience within e-commerce, airlines, transportation, or logistics environments is highly desirable.
Similar Jobs
Artificial Intelligence • Professional Services • Business Intelligence • Consulting • Cybersecurity • Generative AI
Design and build data infrastructure, pipelines, and transformations using Azure Data Factory, Azure Databricks, and Snowflake. Develop data architecture and modeling, maintain data lakes/warehouses, ensure data quality and security, optimize performance, and collaborate with clients to deliver tailored data solutions and governance.
Top Skills:
Azure Data FactoryAzureMicrosoft Azure DatabricksSnowflake
Artificial Intelligence • Professional Services • Business Intelligence • Consulting • Cybersecurity • Generative AI
Lead design and implementation of scalable data infrastructure, pipelines, and transformations using cloud platforms. Build and optimize data warehouses (Snowflake, Databricks), orchestrate workflows (Airflow, Hadoop), ensure data quality and security, and coach teams to deliver analytics solutions that drive tax practice insights and client outcomes.
Top Skills:
Apache AirflowAWSAzure Data FactoryDatabricksHadoopSnowflake
Artificial Intelligence • Professional Services • Business Intelligence • Consulting • Cybersecurity • Generative AI
Lead design and implementation of data infrastructure, pipelines, and warehouses using Azure Data Factory, Databricks, and Snowflake. Manage data modeling, quality, security, and analytics frameworks. Lead client accounts, mentor teams, and drive innovation to deliver data-driven business insights.
Top Skills:
Azure Data FactoryMicrosoft Azure DatabricksSnowflake
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
