Lead Data Engineer
Discover. A more rewarding way to work.At Discover Financial Services, you’ll find yourself in the company of some of the industry’s smartest and most reliable professionals. And at a company that rewards dedication, values innovation and supports growth.Thrive in an environment that promotes teamwork and shared success. Build on a foundation of mutual respect. Join the company that understands rewarding careers like no other, with this exceptional opportunity: Job Description:
As a Lead Data Engineer you will provide engineering leadership to create and enhance data solutions enabling seamless delivery of data across our enterprise. You will be on the cutting edge of finding and integrating new technology and tools for data centric projects. Additionally you will provide lead level technical consulting to peer data engineers during design and development of highly complex and critical data projects. Some of these projects will include designing and developing data ingestion and processing/transformation frameworks leveraging open source tools such as NiFi, EMR, Java, Scala, Spark APIs, AWS Glue, etc. Additionally you will work on real time processing solutions using tools such as Spark Streaming, MQ, Kafka, and AWS Kinesis. You will deploy application code using CI/CD tools and techniques.For your first initative, you will focus on implementing Ab Initio DQE (Data Quality Environment) platform to enable our next gen Data Quality platform for ETL processing. You should have experience in Ab Initio GDE programming, Express-IT business rule development and ACE(application configuration environment) template development.
- Develop data driven solutions utilizing current and next generation technologies to meet evolving business needs.
- Ability to quickly identify an opportunity and recommend possible technical solutions.
- Utilize multiple development languages/tools such as Python, SPARK, HBase, Hive, Java to build prototypes and evaluate results for effectiveness and feasibility.
- Operationalize open source data-analytic tools for enterprise use.
- Utilize tools available to you across AWS Services
- Develop real-time data ingestion and stream-analytic solutions leveraging technologies such as Kafka, Apache Spark, NIFI, Python, HBase and Hadoop.
- Custom Data pipeline development (Cloud and locally hosted)
- Work heavily within AWS and Hadoop ecosystems
- Provide support for deployed data applications and analytical models by being a trusted advisor to Data Scientists and other data consumers by identifying data problems and guiding issue resolution with partner Data Engineers and source data providers.
- Provide subject matter expertise in the analysis, preparation of specifications and plans for the development of data processes.
- Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.
- Bachelors degree in Computer Science, Engineering, or related field
- Ab Initio DQE (Data Quality Environment) architecture
- Ab Initio Express IT business rule development
- Ab Initio ACE (Application configuration environment) template development
- Ab Initio GDE ETL development
- Deep level understanding and implementation experience across AWS Data Services such as Glue, Kinesis, SQS, Redshift
- Snowflake MPP Database
- Deep level understanding and efficiency in the Hadoop technology stack,
- Proficient in Spark application coding using Scala
- Proficient in Java development
- Expert level knowledge in SQL, and relational databases
- Strong in Unix / shell scripting
- Expert in creating very efficient HiveQL and SparkQL queries and can educate peers on the topic
- 5+ years of experience of being a lead engineer amongst a team of equal or junior level developers
- Excellent written and verbal communication, presentation and professional speaking skills
- Collaborative individual who excels in working within a team and with business partners identify, develop and deliver innovative data solutions
- Ability to demonstrate leadership to managers, and peer level staff.
- Strong ability to build and leverage external relationships.
- Decision making abilities while gathering information and then put your decisions into action.
- Passionate learner who enjoys education through class room training and self-discovery on a variety of emerging technologies
- Promote a risk-aware culture, ensure efficient and effective risk and compliance management practices by adhering to required standards and processes.
#LI-KE We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, national origin, religion, sexual orientation, gender identity, status as a veteran, and basis of disability or any other federal, state or local protected class.