Design, build, and maintain AWS-based data pipelines and ETL processes using Glue, Spark/PySpark and EMR. Integrate multiple data sources into data lakes and Redshift, implement DevOps CI/CD practices, and work with Kafka, Hudi, and various AWS storage and analytics services.
This is a remote position.
- 3+ Years of Development / Data Engineering Experience
- 2+ Years Developing with AWS Cloud solutions
- 2+ Years Developing in Spark/PySpark
- 2+ Years Developing AWS Glue ETL
- 2 + Years AWS Storage models (S3 and DynamoDB)
Some hands-on developing with on-prem ETL tooling (Ab Initio preferred, Informatica)
Requirements
- Bachelor of degree in computer or any related field.
- Minimum of 12+ years’ experience in the said role.
- Strong experience in AWS GLUE, EMR, Hudi, experience extracting data from multiple sources & loading data into Data Lakes, AWS Redshift
- Experience working with AWS Elastic search, RDS, PostgreSQL preferred
- Experience w/ AWS services such as Lambda, EMR, SNS/SQS, Event Bridge, Lake Formation & Athena
- Experience w/ integrating applications/systems (data producers) with Enterprise Kafka topics (Confluent Kafka integration w/ AWS S3, Redshift)
- Experience in Requirements Analysis, Data Analysis, Application Design, Application Development & Integration Testing
- Working knowledge of on-prem Extraction, Transformation, Cleansing and Loading methodology and principles
- Experience implementing and adding to DevOps principles (GitLab, maintaining CI/CD pipelines)
- Experience in Java svcs is nice to have
N/A:- Please kindly note that this role is for USC and GC Visa holders only.
Interested and qualified candidates should send their resume to [email protected] using the job title as the subject of the mail.
Similar Jobs
Information Technology • Software
Own and scale AWS OpenSearch production infrastructure and PostgreSQL-to-OpenSearch data pipelines. Design indices and mappings, optimize queries and bulk indexing, monitor cluster health, implement ingestion reliability, perform reindexing/backfills, build reconciliation and recovery processes, and collaborate with backend, DB, and product teams on searchable schemas and runbooks.
Top Skills:
Amazon S3Amazon SqsAws DmsAws EcsAws EksAws LambdaAws Opensearch ServiceClickhouseCloudwatchElasticsearchGoIamJavaOpensearchOpensearch IngestionOpensearch Query DslPostgresPythonTerraformTinybird
Information Technology • Software
Own and scale AWS OpenSearch infrastructure and PostgreSQL-to-OpenSearch pipelines. Design indices, mappings, analyzers, and retention policies. Optimize queries and bulk indexing, monitor cluster health, implement retries/idempotency, backfills, reconciliation, and disaster recovery. Build dashboards, alerts, runbooks, and participate in incident response while collaborating with backend, DB, infra, and product teams.
Top Skills:
Aws DmsAws LambdaAws Opensearch ServiceClickhouseCloudwatchEcsEksElasticsearchGoIamJavaJvmOpensearchOpensearch IngestionOpensearch Query DslPostgresPythonS3SqsTerraformTinybird
Gaming • Software
Design, build, and maintain scalable AWS-based data pipelines and storage (S3, Redshift, RDS, DynamoDB). Implement IaC (CloudFormation/CDK), ETL optimization, monitoring, security (IAM, OWASP/CIS best practices), and CI/CD/code pipeline controls. Collaborate with product teams, document solutions, and provide training to support analytics and reliable data infrastructure.
Top Skills:
Amazon DynamodbAmazon KinesisAmazon RdsAmazon RedshiftAmazon S3AWSAws CdkAws CloudformationAws Data PipelineAws GlueAws IamCis Aws Foundations BenchmarkCweETLInfrastructure-As-CodeMonitoring And LoggingOwaspStatic Code Analysis
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
