The Big Data Lead will implement ETL pipelines, ensure data integrity, troubleshoot PySpark applications, and integrate with existing frameworks while leading a team.
Responsibilities: • Experience with big data processing and distributed computing systems like Spark. • Implement ETL pipelines and data transformation processes. • Ensure data quality and integrity in all data processing workflows. • Troubleshoot and resolve issues related to PySpark applications and workflows. • Understand source, dependencies and data flow from converted PySpark code. • Strong programming skills in Python and SQL. • Experience with big data technologies like Hadoop, Hive, and Kafka. • Understanding of data warehousing concepts and relational databases like SQL. • Demonstrate and document code lineage. • Integrate PySpark code with frameworks such as Ingestion Framework, DataLens, etc., • Ensure compliance with data security, privacy regulations, and organizational standards. • Knowledge of CI/CD pipelines and DevOps practices. • Strong problem-solving and analytical skills. • Excellent communication and leadership abilities. Qualifications: • 4+ years of experience in big data development, Hadoop , Hive & Spark framework. • Good to have experience in SAS. • Strong Python, PySpark Development and SQL knowledge. • Certification in big data or cloud technologies is preferred.
Similar Jobs
Information Technology • Consulting
Lead the development of data pipelines and transformations in Azure Databricks, converting Scala programs to PySpark while leveraging various Azure technologies.
Top Skills:
AdfAzure Data Lake Gen 2Azure DatabricksDelta LakePysparkPythonSparkSynapse Analytics
Information Technology • Consulting
The Big Data Lead will manage database development, ETL/ELT processes, and data warehousing, optimizing performance and ensuring data pipelines work reliably.
Top Skills:
AWSAws GlueAws S3AzureAzure BlobAzure Data FactoryAzure DevopsGitJenkinsOracleSnowflakeSQLTalendTeamcity
Information Technology • Consulting
This role focuses on optimizing large-scale financial modeling applications by implementing MLOps practices and maintaining end-to-end pipelines on AWS.
Top Skills:
AWSCi/CdMachine LearningMlopsSoftware Engineering
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
