Design, implement, and maintain PySpark applications to automate large-scale financial data reconciliations. Build transformation and matching algorithms, integrate with rules engines, analyze data gaps, and collaborate with analysts and architects to ensure data quality and system resilience.
Job Title: PySpark Data Reconciliation Engineer
Summary:
We're seeking a skilled PySpark Data Reconciliation Engineer to join our team and drive the development of robust data reconciliation solutions within our financial systems. You will be responsible for designing, implementing, and maintaining PySpark-based applications to perform complex data reconciliations, identify and resolve discrepancies, and automate data matching processes. The ideal candidate possesses strong PySpark development skills, experience with data reconciliation techniques, and the ability to integrate with diverse data sources and rules engines.
Key Responsibilities:
Data Reconciliation Development:
- Design, develop, and test PySpark-based applications to automate data reconciliation processes across various financial data sources, including relational databases, NoSQL databases, batch files, and real-time data streams.
- Implement efficient data transformation, matching algorithms (deterministic and heuristic) using PySpark and relevant big data frameworks.
- Develop robust error handling and exception management mechanisms to ensure data integrity and system resilience within Spark jobs.
Data Analysis and Matching:
- Collaborate with business analysts and data architects to understand data requirements and matching criteria.
- Analyze and interpret data structures, formats, and relationships to implement effective data matching algorithms using PySpark.
- Work with distributed datasets in Spark, ensuring optimal performance for large-scale data reconciliation.
Rules Engine Integration:
- Integrate PySpark applications with rules engines (e.g., Drools) or equivalent to implement and execute complex data matching rules.
- Develop PySpark code to interact with the rules engine, manage rule execution, and handle rule-based decision-making.
Problem Solving and Gap Analysis:
- Collaborate with cross-functional teams to identify and analyze data gaps and inconsistencies between systems.
- Design and develop PySpark-based solutions to address data integration challenges and ensure data quality.
- Contribute to the development of data governance and quality frameworks within the organization.
Qualifications and Skills:
- Bachelor's degree in Computer Science or a related field.
- 5+ years of hands-on experience in big data development, preferably with exposure to data-intensive applications.
- Strong understanding of data reconciliation principles, techniques, and best practices.
- Proficiency in PySpark, Apache Spark, and related big data technologies for data processing and integration.
- Experience with rules engine integration and development
- Strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions.
- Excellent communication and collaboration skills to work effectively with business analysts, data architects, and other team members.
- Familiarity with data streaming platforms (e.g., Kafka, Kinesis) and big data technologies (e.g., Hadoop, Hive, HBase) is a plus.
Photon Chicago, Illinois, USA Office
11 East Adams Street Suite 1100, Chicago, IL, United States, 60603
Similar Jobs
Edtech • Social Impact
Own new-logo acquisition for higher education: generate pipeline, run full sales cycles with CS faculty and administrators, close campus partnerships, manage HubSpot pipeline and forecasts, represent CodePath at conferences, and meet quarterly new-partner and enrollment targets. Expect regular travel to events and campus meetings.
Top Skills:
Hubspot
Edtech • Social Impact
Embed with teams to audit workflows, build and productionize LLM/agent automations (using Claude and integrations), create reusable assets and guardrails, coach teams to self-sufficiency, and measure outcomes to prioritize future engagements.
Top Skills:
AsanaClaudeClaude CodeGoogle WorkspaceHubspotLlm ApisLow-Code/No-Code Agent FrameworksMcp IntegrationsNotionSlackZendesk
Artificial Intelligence • Professional Services • Business Intelligence • Consulting • Cybersecurity • Generative AI
Lead and manage internal investigations, develop and implement compliance policies, advise on regulatory requirements, analyze operational risks, communicate findings to stakeholders, coach and lead teams, and support compliance program implementation and training to strengthen internal controls and ethical standards.
What you need to know about the Chicago Tech Scene
With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.
Key Facts About Chicago Tech
- Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
- Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
- Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
- Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory
