Principal Cloud Data Engineer
Job Description
At Discover, be part of a culture where diversity, teamwork and collaboration reign. Join a company that is just as employee-focused as it is on its customers and is consistently awarded for both. We’re all about people, and our employees are why Discover is a great place to work. Be the reason we help millions of consumers build a brighter financial future and achieve yours along the way with a rewarding career.
Discover Financial Services is seeking a Principal Cloud Data engineer to join our MAPS Program team as we migrate our General Ledger functionality to the Cloud. Successful candidates will partner with our business partners to understand their data needs, and build data pipelines in the AWS Cloud. Additionally you will provide engineering leadership to create and enhance data solutions enabling seamless integration and flow of data across our data ecosystem. Working alongside our Data Ops organization, you also will provide lead level technical consulting to peer data engineers during design and development for highly complex and critical data projects.
Responsible for using knowledge to apply a hands-on approach with next generation technologies to contribute to the team that delivers the latest data-driven platforms & next generation analytic technologies. This role is for a data ETL lead on a large-scale (100+ files / 50+ applications), multi-year application migration platform for Finance. Experience with AWS/S3 is optimal.
Responsibilities
- Provides senior-level technical consulting to peer-data engineers during design and development for highly complex and critical data projects.
- Provides engineering leadership to create and enhance data solutions enabling seamless integration and flow of data across the data ecosystem.
- Designs and develops data ingestion frameworks leveraging Open-Source tools such as NiFi, Sqoop, Hive, Java, Pig, Python, as well as data processing/transformation frameworks leveraging Open-Source tools.
- Provides support for deployed data applications and analytical models.
- Designs and develops real-time processing solutions using Open-Source tools.
- Be part of MAPS Program team translating business and technology requirements into our ETL/ELT architecture using tools such as Abinitio or Informatica.
- Collaborate with cross functional teams such as AWS cloud & platform engineering, App Dev, Data Ops, DBAs and business teams.
- Develop batch data ingestion solutions leveraging technologies for AWS.
- Develop data driven solutions utilizing current and next generation technologies to meet evolving business needs.
- Ability to quickly identify an opportunity and recommend possible technical solutions.
- Provide support for deployed data applications by being a trusted advisor to Data Scientists and other data consumers by identifying data problems and guiding issue resolution with partner Data Engineers and source data providers.
- Provide subject matter expertise in the analysis, preparation of specifications and plans for the development of data processes.
- Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.
Minimum Qualifications
At a minimum, here’s what we need from you:
- Bachelor’s Degree in Computer Science or related field
- 6+ years of experience in Data Platform Administration/Engineering
Preferred Qualifications
If we had our say, we’d also look for:
- 8+ years of experience in Data Platform Administration/Engineering
- Hands on experience with AWS based solutions such as RDS, Lambda, Dynamodb, Redshift, Snowflake, EC2, Chef Cookbooks and S3
- Experience with one of the ETL tools (Abinitio, Informatica or Data Stage) and tools/frameworks within the Big Data ecosystem
- Experienced in Agile methodologies.
- Proficiency in one of the scripting languages such as Shell/Python/Scala/Java.
- Good understanding of Big Data technology trends, with knowledge of technologies such as Kinesis, Kafka, Spark, Hive, pySpark.
- Experience in version control systems such as Git, GitLab, etc.
- Ability to work in a fast-paced, rapidly changing environment.
- 2+ years of experience using Cloud technologies
- Prior experience with Banking or Financial domain
- AWS Cloud Services certification
#LI-KE
The same way we treat our employees is how we treat all applicants – with respect. Discover Financial Services is an equal opportunity employer (EEO is the law). We thrive on diversity & inclusion. You will be treated fairly throughout our recruiting process and without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status in consideration for a career at Discover.