Principal Cloud Data Engineer/Architect at Discover
Discover. A more rewarding way to work.
At Discover Financial Services, you’ll find yourself in the company of some of the industry’s smartest and most reliable professionals. And at a company that rewards dedication, values innovation and supports growth.
Thrive in an environment that promotes teamwork and shared success. Build on a foundation of mutual respect. Join the company that understands rewarding careers like no other, with this exceptional opportunity:
As a Principal Cloud Data Engineer & Architect you will provide engineering leadership to create and enhance data solutions enabling seamless integration and flow of data across our data ecosystem which includes AWS. Additionally you will provide senior level technical consulting to peer data engineers during design and development of highly complex and critical data projects. You will work in technologies such as Spark(python/scala), Kafka, EMR, MPP Databases (Snowflake & Redshift), NoSQL DBs, and other various AWS services (Lambda, Glue, Step Functions..). Additionally, you will work on real time processing solutions using tools such as Kafka, Kinesis and Spark streaming and will deploy application code using CI/CD tools and techniques.
You will be a part of an exciting cutting edge team consisting of high performing engineers knowledgeable across a wide range of technologies. Your teams mission will be to create frameworks, patterns and tools that enable outside data engineering teams to build and manage data pipelines with ease and elegance.
- Develop data driven solutions utilizing current and next generation technologies to meet evolving business and data engineering needs.
- Ability to quickly identify an opportunity and recommend possible technical solutions.
- Utilize multiple development languages such as Python, Scala, Java, w/ various libraries and packages such as SPARK, etc..
- Build prototypes and evaluate results for effectiveness and feasibility.
- Work heavily within the AWS ecosystem, using AWS services
- Operationalize open source data-analytic tools for enterprise use.
- Develop real-time data ingestion and stream-analytic solutions leveraging technologies such as Kafka, Spark Streaming, Kinesis, and EMR.
- Develop patterns, frameworks and tools for API and batch based data pipelines
- Utilize data stores/databases such as Snowflake, Redshift, Graph Databases and other NoSQL DBs
- Custom Data pipeline development (Cloud and locally hosted)
- Provide subject matter expertise in the analysis, preparation of specifications and plans for the development of data processes.
- Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.
- Promote a risk-aware culture, ensure efficient and effective risk and compliance management practices by adhering to required standards and processes.
- AWS data services (Lambda,Glue, EMR, Kinesis, Step Functions, Data Pipeline)
- Deep understanding of the Hadoop technology stack
- Deep expert level knowledge of either Python, Java or Scala
- Data pipeline / ETL development experience
- Spark application coding in Scala / Python (pySpark)
- Deep knowledge and very strong in SQL, and Relational Databases
- Strong in Unix / Shell scripting
- Knowledge of ETL tools such as Ab Initio or Informatica a big plus
- Bachelors degree in Computer Science, related field, or equivalent work experience
- 7+ years of experience of being a lead engineer and able to coach/provide guidance to peer and junior engineers.
- Excellent written and verbal communication, presentation and professional speaking skills
- Collaborative individual who excels in working within a team and with business partners to identify, develop and deliver innovative data solutions
- Influencing skills/ability. Must be able to work with effectively with different levels of management and all business areas.
- Ability to demonstrate leadership to managers, and peer level staff.
- Ability to build and leverage external relationships.
- Decision making abilities while gathering information and then put your decisions into action.
- Passionate learner who enjoys education through class room training and self-discovery on a variety of emerging technologies
We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, national origin, religion, sexual orientation, gender identity, status as a veteran, and basis of disability or any other federal, state or local protected class.