Principal Devleopment Engineer

We’re building a world of health around every individual — shaping a more connected, convenient and compassionate health experience. At CVS Health®, you’ll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger – helping to simplify health care one person, one family and one community at a time.

Position Summary

As a Principal Software Development Engineer you will be the senior technical anchor across AI Platform within HCD. You will own the platform decisions every agent depends on, set the engineering standards the whole portfolio is built against, drive the hardest architectural problems across the Agent Development Lifecycle, and be a force multiplier for a team of talented engineers who are shipping real work at high velocity.

Role and Responsibilities

• Own the LLM gateway strategy end-to-end — including model access governance, latency benchmarking across routing layers

• Lead Model Context Protocol (MCP) farm design for authenticated, scalable tool integrations across all agentic applications

• Define and maintain reference architectures, template repositories, SDKs, and the AI QuickStart that set the paved road for agent delivery across HCD and OGAI

• Drive the Skills and Plugin Marketplace strategy, ensuring a platform-agnostic reusable component model that works across teams and technology stacks

• Own and continuously evolve the ADLC framework — the team's standard for taking agents from use case discovery through infrastructure planning, evaluation design, development, testing, and production deployment

• Enforce engineering quality gates across all agent types: RAG knowledge assistants, BrowserUse and RPA automation, conversational chatbots, voice agents, and multi-step workflow agents

• Ensure eval framework design happens before development begins — with defined test datasets, hallucination rate targets, adversarial test suites, and business outcome metrics for every production agent

• Own structured rollback procedures, canary deployment patterns, and production readiness standards that prevent instability from propagating through the portfolio

• Provide principal-level technical leadership across the active agent portfolio.

• Own the hardest cross-cutting technical challenges in the portfolio — sensitive data handling at the agent layer,  integration patterns, BrowserUse automation reliability, RAG retrieval quality at scale, and latency under real practical scenarios.

• Unblock delivery across multiple simultaneous work streams by resolving architecture dependencies, provisioning blockers, and cross-team coordination issues before they stall sprints

• Drive the deployment automation strategy so agents consistently move from dev to production through a standardized, auditable pipeline

• Own the observability mandate — driving adoption across all production agents and defining eval standards that reflect real business outcomes, not just infrastructure uptime

• Build and maintain CI and CD pipelines with integrated LLM evaluations, prompt versioning, hallucination and failure rate tracking, and regression test suites tied to production datasets

• Define and enforce the AI SDLC: style guides, working agreements, code review standards, engineering handbook practices, and the agent quality run-book the full team works against

• Own guardrails infrastructure including policy enforcement, adversarial test patterns, and the AI quality pipeline that catches low-quality outputs before they reach production

• Maintain the AI spend tracking system — cost per team, cost per agent, cost per model — with automated reporting for senior leadership

• Own the AI metrics dashboard delivering adoption trends, deployment frequency, token consumption, and per-agent business KPIs to executive stakeholders

• Provide compliance and security design leadership for agents operating on sensitive data, including agent identity and managed identity strategy at the platform level

Required Qualifications

• 9+ years of professional software engineering experience

• Deep expertise in Python with strong cloud infrastructure experience, preferably Azure, and hands-on familiarity with API gateway patterns and LLM routing

• Demonstrated ability to influence architecture and engineering standards across multiple teams and work streams without direct management authority

Preferred Qualification:

• 3+ years in a principal, staff, or equivalent senior individual contributor role

• Hands-on experience with AI observability tooling such as Arize, LangSmith, or Phoenix and the ability to design eval pipelines that measure what actually matters for business outcomes

• Proven track record designing and operating production AI systems — LLM APIs, agent frameworks, RAG architectures, or RPA automation — at meaningful scale

• Strong working knowledge of security and compliance requirements for AI systems handling sensitive data, including PHI and PII handling in a regulated environment

Education:

• Bachelor's degree preferred/specialized training/relevant professional qualification.

Pay Range

The typical pay range for this role is:

This pay range represents the base hourly rate or base annual full-time salary for all positions in the job grade within which this position falls.  The actual base salary offer will depend on a variety of factors including experience, education, geography and other relevant factors.  This position is eligible for a CVS Health bonus, commission or short-term incentive program in addition to the base pay range listed above.  This position also includes an award target in the company’s equity award program. 
 

Our people fuel our future. Our teams reflect the customers, patients, members and communities we serve and we are committed to fostering a workplace where every colleague feels valued and that they belong.

Great benefits for great people

We take pride in offering a comprehensive and competitive mix of pay and benefits that reflects our commitment to our colleagues and their families.

Additional details about available benefits are provided during the application process and on Benefits Moments.

Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state and local laws.