LEAD DATA ENGINEER
JOB PURPOSE:
As a Lead Software Engineer you will lead the design, development, testing and deployment of data pipelines, software systems and other applications.
JOB DUTIES:
Responsibilities of the incumbents are across functional lines with individuals assigned in new program development and/or maintenance of existing distributed computing systems in cloud. Be responsible for overseeing and mentoring junior developers and handle code reviews.
Competent to perform all programming, project management, and development assignments without close supervision; normally assigned the more complex aspects of systems work.
Acts as team leader for development and support groups; provides programming and application/technical leadership, guidance, and assistance to other team members.
Works directly on complex application/technical problem identification and resolution, including responding to off-shift and weekend support calls.
Works independently on complex systems or infrastructure components that may be used by one or more applications or systems.
Drives application development focused around delivering business valuable features
Mentor and assist software engineers, providing technical assistance and direction as needed
Maintains high standards of software quality within the team by establishing good practices and habits
Identifies and encourage areas for growth and improvement within the team
Guide the team to develop a structured application/interface code, new program documentation, operations documentation and user guides in a casual, flexible environment
Communicate with end users and internal customers to help direct development, debugging, and testing of application software for accuracy, integrity, interoperability, and completeness
Lead development of new functionality and applications on cross-functional Agile project teams
Performs integrated testing and customer acceptance testing of components that requires careful planning and execution to ensure timely, quality results.
Employee is also responsible for performing other job duties as assigned by Caterpillar management from time to time.
Qualifications
Basic qualifications:
· Position requires a four-year degree from an accredited college or university.
· 7 years or more of software development experience or at least 5 years of experience with master’s degree in computer science or related field.
· Demonstrated experience leading software engineering projects.
· 5 years or more of experience in designing and developing software applications in Java, Python Scala or at least three years of experience in Java or Scala with master’s degree in computer science or related field.
· 5 years or more of developing backend data pipelines in AWS using Python (PySpark) and Scala.
· 5 years or more of experience analyzing data using various data science libraries
· 3 years or more experience in the framework’s like Scalatra, Akka, Django
Top candidates will also have:
· Proven experience in many of the following,
o Designing, developing, deploying and maintaining software at scale.
o Developing software applications using relational and Nosql databases.
o Developing data pipelines (batch, near real-time, streaming) using AWS Technology stack.
o Deep understanding of distributed data pipelines using AWS managed services.
o Experience with data warehousing applications like Snowflake, Redshift
o Application architectural patterns, such as MVC, Microservices, Event-driven, etc.
o Deploying software using CI/CD tools such as Jenkins, GoCD, Azure Devops etc.
o Deploying and maintaining software using public clouds such as AWS or Azure.
o Working within an Agile framework (ideally Scrum)
· Strong understanding and/or experience in many of the following,
o Batch or stream processing systems such as Apache Spark, Flink, Akka, Storm
o Data pipelines using python, Spark for MPP
o Message brokers such as Kafka, Rabbitmq, AWS SQS, AWS SNS, Apache ActiveMQ, Kinesis.
o Experience designing well-defined Restful APIs
o Experience writing API proxies on platforms such as Apigee Edge, AWS API Gateway or Azure API Gateway
o Hands one experience with API tools such as Swagger, Postman and Assertible
o Test driven development and behavior driven development.
o Hands on experience with testing tools such as Selenium and Cucumber and their integration into CI/CD pipelines.
o Datastores such as MongoDB, Cassandra, Redis, Elasticsearch, MySQL, Oracle.
o Debugging and maintaining software in Linux or Unix platforms.
· Understanding of front-end state management libraries (redux)
· Must demonstrate solid knowledge of computer science fundamentals like data structures and algorithms.
· Ability to work under pressure and within time constraints
· Passion for technology and an eagerness to contribute to a team-oriented environment
· Demonstrated leadership on small to medium-scale projects impacting strategic priorities
· Bachelor’s degree in Computer science or Electrical engineering or related field is required