Candid Logo

Candid

Senior Data Pipeline Engineer

Posted 9 Days Ago
Remote
Hiring Remotely in United States
100K-130K Annually
Senior level
Remote
Hiring Remotely in United States
100K-130K Annually
Senior level
Develops, architects, and maintains production ETL/ELT and C#/.NET data pipelines across cloud data platforms. Responsibilities include supporting Iceberg, DynamoDB, AWS Lambda, and Trino/Starburst systems; onboarding data sources; tuning performance; monitoring data quality; resolving incidents; coordinating schemas; documenting processes; mentoring engineers; and supporting releases, deployments, architecture, and roadmap decisions.
The summary above was generated by AI

Position Summary

Candid is a nonprofit that provides the most comprehensive data and insights about the social sector. We get you the information you need to do good. Candid currently has an opportunity for a Senior Data Pipeline. The Senior Data Pipeline Engineer is a senior technical role responsible for the day-to-day development, and delivery of Candid's data pipelines. This role blends deep SQL and ETL engineering expertise with hands-on experience across modern cloud data platform technologies to design, build, and tune pipelines that move data reliably from source systems into Candid's data warehouse and lakehouse.  

Position: Senior Data Pipeline Engineer

Reporting to: Data Operations Manager

Supervises: N/A

Schedule: 35-hour work week, Monday through Friday

Compensation: $100,000 - $130,000 (this range is for the NYC area and will be adjusted for other localities; additionally, factors like skills and experience will be considered).

Location: Remote. In-person attendance is expected twice per year during our annual, weeklong all-staff summits. Additional in-person meeting participation is expected at least once per quarter for senior leaders and at least once per month for the executive team. Staff not located in the NYC area are expected to travel for these meetings. 

Benefits: Health insurance (medical, dental, vision), retirement contribution with additional option for a match, paid life insurance and AD&D, paid leave time (PTO, compassionate leave, volunteer, holiday, parental), short-term and long-term disability, pre-tax transit, flexible spending accounts, supplemental insurance, summer hours, and Public Service Loan Forgiveness (PSLF) program eligible employer.

Applicants for this position must be legally authorized to work in the United States without employer sponsorship, now or in the future.


Responsibilities  

  • Pipeline development: Architect, develop, and maintain ETL/ELT and C#/.NET pipeline services (SSIS, Airflow, AWS Lambda, enhancer and message-processing services) that move data reliably from source systems into Candid's warehouse and lakehouse.
  • Cloud platform & performance: Support Candid's evolving cloud data platform, including Iceberg ingestion and transformation pipelines, DynamoDB, and query performance tuning via Trino/Starburst.
  • Data quality & incident resolution: Monitor data quality end-to-end and investigate production pipeline issues at the root cause, documenting findings and driving fixes through to reprocessing.
  • Source onboarding & schema coordination: Scope and onboard new external and regulatory data sources end-to-end, and contribute to schema and architecture decisions, coordinating changes across ingestion and transformation layers.
  • Documentation, mentorship & process: Maintain SOP and technical documentation, mentor and review the work of Associate-level engineers, and participate in the team's release and deployment process.
  • Partnership & support: Partner with the Data Operations Manager on architecture and roadmap decisions, and provide escalation-level support for mission-critical pipeline issues, including after-hours as needed.

Requirements  

  • 5+ years of experience in database engineering, data engineering, or ETL development, with demonstrated ownership of production data pipelines.
  • Expert-level Transact-SQL skills (stored procedures, views, indexing) and hands-on C#/.NET development experience.
  • Proven experience architecting ETL processes with SSIS; experience with orchestration platforms such as Apache Airflow preferred.
  • Experience with AWS and open table formats (Apache Iceberg, Parquet); AWS Lambda, DynamoDB, and Trino/Starburst a plus.
  • Strong data quality and root-cause troubleshooting skills, with experience resolving production data incidents.
  • Familiarity with source control, CI/CD, and release/deployment processes across Dev/QA/Prod environments.
  • Ability to mentor and review the work of other engineers; excellent written and verbal communication.
  • Good business judgment, independent prioritization, and the ability to work well in a distributed team.
  • Willingness to perform other duties and special projects as needed/requested.
  • Sensitivity and respect for racial, gender, sexual orientation, and cultural differences.
  • Commitment to Candid's values: driven, direct, accessible, curious, and inclusive. 

About Candid 
Candid’s mission is to get you the information you need to do good.

The world’s problems are only growing, and change can’t wait. Nonprofits are needed now more than ever, but all too often their work goes without adequate support.

Candid makes it easier and faster for nonprofits and funders to connect in pursuit of solutions to change the world. Candid is where nonprofits find grants, donors find nonprofits that inspire them, and all can gain insights about the work being done for good.

Candid is a qualifying nonprofit organization as defined by the Public Service Loan Forgiveness Program. As such, Candid employees may claim their employment time on their PSLF application. We offer a competitive salary and excellent benefits. Due to the high volume of applicants we typically receive, we regret that we can only contact candidates we would like to interview.

For more information on positions available at Candid, please visit our website: Work with us

Candid is an equal opportunity employer. Candid provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.

This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.
Notice: We are aware of emails/communications impersonating Candid’s recruitment staff. All legitimate recruiting communications come from a @candid.org email address. If you’ve received any communications claiming to be Candid that does not come from a @candid.org email address, please forward to [email protected]. We appreciate your interest in Candid.


Equal Opportunity Employer/Protected Veterans/Individuals with Disabilities
This employer is required to notify all applicants of their rights pursuant to federal employment laws. For further information, please review the Know Your Rights notice from the Department of Labor.

Similar Jobs

29 Minutes Ago
Remote or Hybrid
124K-380K Annually
Expert/Leader
124K-380K Annually
Expert/Leader
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Leads underwriting strategy, portfolio performance, capacity deployment, and profitable growth for a complex commercial property business. Oversees large and layered property risks, catastrophe exposure, pricing, broker relationships, and market strategy. Partners with actuarial, claims, risk engineering, product, distribution, and finance teams. Develops underwriting talent, succession plans, and organizational capabilities while serving as a senior authority on complex property underwriting decisions.
61K-126K Annually
Senior level
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Manages, investigates, evaluates, and resolves attorney-represented and litigated auto and homeowners bodily injury claims. Assesses coverage, liability, damages, reserves, settlement authority, suspicious claims, and subrogation opportunities. Communicates with policyholders, witnesses, claimants, counsel, and management; prepares trial reports and attends legal proceedings. Oversees outside and in-house counsel practices and billing, maintains claim files, and provides required reports.
29 Minutes Ago
Remote or Hybrid
67K-126K Annually
Mid level
67K-126K Annually
Mid level
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Manages medical and disability claim caseloads using nursing expertise to coordinate treatment, negotiate return-to-work plans, facilitate maximum medical improvement, and support claim resolution. Communicates with injured employees, providers, attorneys, employers, and claims teams; documents case strategies and nursing activities; coordinates utilization, peer, vocational, and medical director resources; and manages complex or catastrophic cases.
Top Skills: Internet ExplorerMS Office

What you need to know about the Chicago Tech Scene

With vibrant neighborhoods, great food and more affordable housing than either coast, Chicago might be the most liveable major tech hub. It is the birthplace of modern commodities and futures trading, a national hub for logistics and commerce, and home to the American Medical Association and the American Bar Association. This diverse blend of industry influences has helped Chicago emerge as a major player in verticals like fintech, biotechnology, legal tech, e-commerce and logistics technology. It’s also a major hiring center for tech companies on both coasts.

Key Facts About Chicago Tech

  • Number of Tech Workers: 245,800; 5.2% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: McDonald’s, John Deere, Boeing, Morningstar
  • Key Industries: Artificial intelligence, biotechnology, fintech, software, logistics technology
  • Funding Landscape: $2.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Pritzker Group Venture Capital, Arch Venture Partners, MATH Venture Partners, Jump Capital, Hyde Park Venture Partners
  • Research Centers and Universities: Northwestern University, University of Chicago, University of Illinois Urbana-Champaign, Illinois Institute of Technology, Argonne National Laboratory, Fermi National Accelerator Laboratory

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account