Lead Data Engineer
Job Description
At Discover, be part of a culture where diversity, teamwork and collaboration reign. Join a company that is just as employee-focused as it is on its customers and is consistently awarded for both. We’re all about people, and our employees are why Discover is a great place to work. Be the reason we help millions of consumers build a brighter financial future and achieve yours along the way with a rewarding career.
Responsible for designing and developing data-driven platforms and next-generation analytic technologies.
On top of that, as a lead data engineer, you will provide engineering leadership to create and enhance data quality control engineering solution. You will be on the cutting edge of finding and integrating new technology and tools for data centric projects. For your first imitative, you will focus on implementing Ab Initio DQE (Data Quality Environment) platform to enable our next gen Data Quality platform for ETL processing in AWS platform. You should have experience in Ab Initio GDE programming, Express-IT business rule development, MDH (Metadata Hub) and ACE (application configuration environment) template development. You will also be helping build customized code for data DQ rule injection across multiple platforms
Responsibilities
- Designs and develops complex and critical data projects.
- Designs and develops real-time processing solutions.
- Creates and enhances data solutions that enable seamless integration and flow of data across the data ecosystem.
- Designs and develops data ingestion frameworks, leveraging open-source tools and data-processing frameworks.
- Develops, implements, and supports application codes and analytical models.
- Partners with the business and technology team to complete business initiatives in data quality control space. Develop data driven solutions utilizing current and next generation technologies to meet evolving business needs.
- Ability to quickly identify an opportunity and recommend possible technical solution in data qualify control domain.
- Follow Agile development approach for custom data pipeline development (Cloud and on prem).
- Utilize multiple development languages/tools such as Ab Initio, Python, SPARK, Java, CI/CD pipeline, customized API calls to build prototypes and evaluate results for effectiveness and feasibility.
- Work heavily within the AWS ecosystem, using AWS services.
- Operationalize open source data-analytic tools for enterprise use.
- Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, Classification, etc.
- Help couch and management junior team members.
Minimum Qualifications
At a minimum, here’s what we need from you:
- Bachelor’s Degree in Computer Science, or related field
- 4+ years of experience in Data Platform Administration/Engineering
Preferred Qualifications
If we had our say, we’d also look for:
- 6+ years of experience in Data Platform Administration/Engineering
- Excellent written and verbal communication, presentation and professional speaking skills.
- Master’s degree in Computer Science, Information Technology, Operations Research, or related engineering field
- Development experience with programming languages such as Java, python and spark
- Development experience with developing REST API
- Strong in Unix/Shell scripting
- 3+ years of experience of being a lead engineer amongst a team of equal or junior level developers.
- Deep level understanding and implementation experience across AWS Data Services such as Glue, Kinesis, SQS, Redshift, VPC, IAM, EC2, RDS, SNS, CloudWatch, CloudTrail, Step Functions, Lambda, EMR
- Ab Initio DQE (Data Quality Environment) architecture
- Ab Initio Express IT business rule development
- Ab Initio ACE (Application configuration environment) template development
- Ab Initio GDE ETL development and MDH #LI- BG1
Discover Financial Services is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, protected veteran status, among other things, or as a qualified individual with a disability.