Design, build, and maintain Azure-native data pipelines and lakehouse layers. Ingest from APIs, IoT, ERP and files; develop Databricks PySpark jobs and Synapse workloads; ensure data quality, security, CI/CD, and governance while collaborating with analytics consumers.
About the project:
You will be embedded in the Data Platform team of a global mining organization, helping design and deliver a modern Azure-native data infrastructure. The team ingests data from dozens of operational sources — including REST and SOAP APIs, IoT sensors, ERP systems, and flat-file feeds — transforming it into curated, analytics-ready datasets that drive decisions across finance, operations, and supply chain.
This is a hands-on engineering role. You will spend most of your time building and maintaining robust pipelines, optimizing data flows within Azure Synapse and Databricks, and ensuring the platform remains reliable, scalable, and secure.
Key responsibilities:
- Gather and analyze business requirements to identify and prioritize integration opportunities.
- Design, build, and maintain scalable data pipelines using Azure Data Factory to orchestrate ingestion from diverse source systems.
- Develop and optimize data transformation logic and analytical workloads in Azure Synapse Analytics (dedicated and serverless SQL pools, Synapse Pipelines).
- Build and maintain Databricks notebooks and jobs (PySpark / Spark SQL) for large-scale data processing and feature engineering.
- Implement robust API extraction patterns — REST and SOAP — handling pagination, authentication (OAuth 2.0, API keys), rate limiting, and error recovery.
- Design and maintain the lakehouse architecture: raw, curated, and serving layers using Azure Data Lake Storage Gen2.
- Ensure data quality through validation frameworks, monitoring pipelines, alerting, and lineage documentation.
- Collaborate with Analytics Engineers and data consumers to understand downstream requirements and model data accordingly.
- Apply best practices for CI/CD, infrastructure-as-code (Azure DevOps, Terraform / Bicep), and environment management.
- Participate in data governance activities including metadata management, access control, and documentation.
Requirements:
- 4+ years of experience in data engineering or a closely related software engineering role.
- Strong analytical and communication skills
- Strong hands-on experience with Azure Data Factory — authoring pipelines, triggers, linked services, and data flows.
- Solid experience with Azure Synapse Analytics — SQL pools, Synapse Pipelines, and integration with ADLS Gen2.
- Proven experience with Databricks — PySpark, Delta Lake, job clusters, and Unity Catalog is a plus.
- Strong API extraction skills: designing reliable ingestion from REST and SOAP endpoints, handling complex authentication and error scenarios.
- Proficiency in SQL (T-SQL or Spark SQL) and Python for data manipulation and pipeline scripting.
- Familiarity with data modelling concepts: star/snowflake schemas, data vault, or medallion architecture.
- Experience with Azure DevOps for source control (Git), CI/CD pipelines, and automated deployments.
- Understanding of data security, role-based access control (RBAC), and encryption in Azure.
- Strong problem-solving mindset and ability to work autonomously in a distributed team environment.
Nice to have:
- Experience with Azure Purview for data cataloguing and governance.
- Knowledge of streaming data patterns (Azure Event Hubs, Kafka, Spark Streaming).
- Familiarity with dbt or similar SQL-based transformation frameworks.
- Background in mining, heavy industry, or operational technology (OT) data environments.
- Power Platform awareness (not required, but beneficial for understanding downstream consumers).
We offer*:
- Flexible working format - remote, office-based or flexible
- A competitive salary and good compensation package
- Personalized career growth
- Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
- Active tech communities with regular knowledge sharing
- Education reimbursement
- Memorable anniversary presents
- Corporate events and team buildings
- Other location-specific benefits
*not applicable for freelancers
Similar Jobs
Information Technology
The Senior Data Engineer will develop end-to-end technical data solutions, ensuring performance and security while integrating data across cloud platforms like Snowflake and AWS, and optimizing SQL queries.
Top Skills:
AWSAws Data Migration ServicesAzureAzure DatafactoryDatabricksFivetranGCPInformaticaJavaKafkaNifiPythonScalaSnowflakeSparkSQL
Information Technology
Design and maintain end-to-end data pipelines using Microsoft Fabric and Azure Synapse, ensuring data quality and performance optimization while collaborating with cross-functional teams.
Top Skills:
Azure Synapse AnalyticsMicrosoft FabricPysparkPythonSparkSQL
Information Technology
The role involves designing and maintaining data pipelines, developing ETL/ELT processes, building Power BI dashboards, and ensuring data quality and governance.
Top Skills:
AWSAzureEtl/EltGitPower BIPythonSQL
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
