Design, develop, and optimize scalable batch and real-time data pipelines on GCP. Implement Medallion Architecture, build Python and PySpark transformations, optimize BigQuery and SQL workloads, deploy Dataflow pipelines, and orchestrate workflows with Cloud Composer and Airflow. Manage GCS storage, CI/CD, version control, IAM, data governance, security, reliability, scalability, and cost efficiency across enterprise data platforms.
This is a remote position.
KANINI is seeking a highly skilled Senior Data Engineer with deep expertise in Google Cloud Platform (GCP) and modern data architecture. The ideal candidate will have hands-on experience designing scalable data pipelines, implementing Medallion Architecture, and building robust enterprise-grade data solutions.
This role requires strong technical proficiency in BigQuery, PySpark, Dataflow, and Airflow, along with a solid understanding of cloud data governance, performance optimization, and CI/CD practices.
Key Responsibilities
- Design, develop, and maintain scalable batch and real-time data pipelines on GCP
- Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data processing
- Build high-performance data transformations using Python and PySpark
- Develop and optimize complex SQL queries for analytical workloads
- Work extensively with BigQuery for large-scale data processing and performance tuning
- Develop and deploy pipelines using Cloud Dataflow
- Orchestrate workflows using Cloud Composer (Apache Airflow)
- Manage data storage and lifecycle using Google Cloud Storage (GCS)
- Implement version control and CI/CD pipelines using Git-based tools
- Ensure data security, governance, and access control using GCP IAM
- Optimize data solutions for performance, scalability, reliability, and cost-efficiency
Required Skills & Experience
- Strong hands-on experience with Google Cloud Platform (GCP)
- Expertise in BigQuery (partitioning, clustering, query optimization)
- Proven experience implementing Medallion Data Architecture
- Strong programming skills in Python and PySpark
- Hands-on exposure on Java
- Advanced proficiency in SQL (complex joins, window functions, performance tuning)
- Hands-on experience with Cloud Dataflow
- Experience with Cloud Composer (Airflow) for orchestration
- Experience working with Google Cloud Storage (GCS)
- Knowledge of version control systems (Git) and CI/CD practices
- Strong understanding of GCP IAM and cloud security best practices
Preferred Qualifications
- Experience working with large-scale enterprise data platforms
- Knowledge of data warehousing and data lake concepts
- Familiarity with real-time streaming frameworks
- Experience in data governance and data quality frameworks
- Exposure to
Similar Jobs
Financial Services
Provides security architecture and enablement across SaaS platforms, APIs, custom applications, data integrations, cloud environments, and AI workflows. Partners with product, engineering, data, architecture, identity, infrastructure, enterprise, and procurement teams to conduct threat reviews, develop secure-by-design patterns, assess vendors, implement guardrails, and manage risks. Translates security requirements into practical solutions, supports enterprise data strategy and M&A integration, and mentors teams as a trusted security advisor.
Top Skills:
APIsArtificial IntelligenceCloud ComputingData LineageData Loss Prevention (Dlp)EncryptionIdentity And Access ManagementSaaS
Cloud • Fintech • Food • Information Technology • Software • Hospitality
Leads Toast’s merchant liens, levies, garnishments, UCC, tax lien, and bankruptcy compliance function. Builds policies, governance, workflows, systems, KPIs, and a specialized oversight team while insourcing operations from a third-party partner. Manages complex escalations involving customers, creditors, attorneys, Legal, Risk, Finance, Product, and Engineering, ensuring compliant withholding, remittance, releases, and handling of high-risk cases.
Artificial Intelligence • eCommerce • Fintech • Payments • Retail • Software • Analytics
Provide Tier 2 technical support for Android-based POS software and hardware, troubleshooting payments, inventory, reporting, networking, and device issues. Investigate logs and system behavior, resolve escalated customer cases, and collaborate with Product and Engineering on bugs and fixes. Maintain troubleshooting documentation, assist with releases and hardware compatibility testing, and participate in on-call coverage for critical incidents.
Top Skills:
AndroidBarcode ScannersCash DrawersIp ConfigurationPayment DevicesPos SystemsReceipt PrintersRemote Access ToolsRoutersTouch TerminalsWifi
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
