Company Description
Job Description
Love cutting-edge tech? We do too.
At Epsilon, we do more than collect and store data. We help some of the world's biggest brands discover real opportunities inside the data types, delimiters and decimals. At Epsilon, we can analyze anonymized, privacy-safe data at internet scale and handle 300B+ of online interactions a day and have $3.8 trillion in multichannel purchases in our database.
We are looking for a Senior Data Engineer with extensive experience in the big data ecosystem to work in our Data Pipeline team. As part of the Epsilon DMS Data Organization, the Data Pipeline team's responsibilities include but is not limited to maintaining streaming and batched data pipelines collecting hundreds of billions of data rows daily, near real time aggregation of a variety of data sets (using Spark Structured Streaming) and maintaining system of record raw and aggregate data sets (HDFS and HIVE). Additionally, as part of the Data Pipeline team work closely with Real Time Bidding (RTB), data warehousing, ETL and decision science teams in building, maintaining and optimizing solutions.
The candidate must be proficient in Scala or Java, with the ability to be a key contributor in building and maintaining Spark jobs and AirFlow DAGs. This role will be hands on in code development and data architecture to optimize the platform for growth and maximum efficiency. The person in this role will need to be able to work both independently and as part of a team to meet required specifications of solution delivery.
Responsibilities

• You will design and code data pipelines connecting Real Time Bidding platform, HDFS, Elastic, MPP and Postgres databases utilizing Flume, Kafka, Spark, Cassandra/Scylla and other technologies as needed.
• Maintain and extend Spark framework used by DMS Data Organization to support various aspects of the business.
• Build and maintain AirFlow DAGs for job management.
• Work closely with infrastructure teams in capacity planning, hardware procurement and build outs.
• Build and maintain metrics collection to help with identifying production issues, optimizing job performance and alerting on error conditions (Pager Duty).
• Develop test cases to demonstrate new code meets functional requirements.
• Ideal candidate can lead in one or more areas of: design, code development, data modeling, cross team communication, application maintenance and identifying opportunities for improving code quality or performance through refactoring and/or incorporating new technologies.

Required qualifications

• Bachelor's Degree in Computer Science or equivalent degree is required.
• 4+ years of experience developing in Java and/or Scala
• Spark experience a big plus
• Strong experience in SQL, bash and Python
• Experience with Hadoop Stack (HDFS, Hive, YARN, HBase)
• Experience with Kafka and the Kafka producer and consumer APIs (Kafka Connect, K-streams and ksql a plus)
• Experience with Postgres and Cassandra/Scylla a big plus
• Docker and Kubernetes experience a big plus
• ELK stack experience a plus
• Experience with scheduling applications with complex interdependencies
• Good experience in working with geographically and culturally diverse teams
• Excellent written and verbal communication skills
• Excellent analytical and problem-solving skills
• Ability to diagnose and troubleshoot problems quickly

Qualifications
Additional Information
When you're one of us, you get to run with the best. For decades, we've been helping marketers from the world's top brands personalize experiences for millions of people with our cutting-edge technology, solutions and services. Epsilon's best-in-class identity gives brands a clear, privacy-safe view of their customers, which they can use across our suite of digital media, messaging and loyalty solutions. We process 400+ billion consumer actions each day and hold many patents of proprietary technology, including real-time modeling languages and consumer privacy advancements. Thanks to the work of every employee, Epsilon has been consistently recognized as industry-leading by Forrester, Adweek and the MRC. Positioned at the core of Publicis Groupe, Epsilon is a global company with more than 8,000 employees around the world. Check out a few of these resources to learn more about what makes Epsilon so EPIC:

• Culture: https://www.epsilon.com/us/about-us/our-culture-epsilon
• DE&I: https://www.epsilon.com/us/about-us/diversity-equity-inclusion
• CSR: https://www.epsilon.com/us/about-us/corporate-social-responsibility
• Life at Epsilon: https://www.epsilon.com/us/about-us/epic-blog

Great People Deserve Great Benefits
We know that we have some of the brightest and most talented associates in the world, and we believe in rewarding them accordingly. If you work here, expect competitive pay, comprehensive health coverage, and endless opportunities to advance your career.
Epsilon is an Equal Opportunity Employer. Epsilon's policy is not to discriminate against any applicant or employee based on actual or perceived race, age, sex or gender (including pregnancy), marital status, national origin, ancestry, citizenship status, mental or physical disability, religion, creed, color, sexual orientation, gender identity or expression (including transgender status), veteran status, genetic information, or any other characteristic protected by applicable federal, state or local law. Epsilon also prohibits harassment of applicants and employees based on any of these protected categories.
Epsilon will provide accommodations to applicants needing accommodations to complete the application process.
#LI-AM1
REF134146K