Big Data Developer
The ideal candidate will be able to:
Utilize Big Data Technologies such as Hadoop and Spark, Kafka and In-memory applications using Java, SQL, Pig, Hive, Impala, Scala, based on the concepts of Lamda architecture
Analyze multiple sources of structured and unstructured data to propose and design data architecture solutions for scalability, high availability, fault tolerance, and elasticity.
Understand business taxonomy and provide data analysis, design, modeling, and implementation leadership to project teams, database development group(s), IT, and the enterprise and ensure that analytical solutions are aligned with our architectural direction.
Develop conceptual, logical and physical design for various data types and large volumes.
Architect, design and implement high performance, large volume data integration processes, database, storage, and other back-end services
Implement security, encryption best practices for big data environments.
Create and maintain data flow diagrams, entity relationship diagrams, system diagrams, business and logical systems integration as it pertains to Analytics/Business Intelligence and big data systems.
Find and implement solutions to integrate event-based, semi-structured and 3rd party data sources (e.g. E-commerce transaction data) with enterprise relational models and strategies for transactional and analytical systems to effectively share data.
Partner with our Enterprise Architecture team to ensure that the defined architectures align with other architecture domains (e.g. infrastructure, EDW, security, etc).
- Solid experience with NoSQL and MPP/columnar database technologies
- Demonstrated ability to architect and model mission critical BI, OLAP, OLTP, ETL, NoSQL, MPP, In-memory and batch Hadoop solutions leveraging multiple data technologies (Oracle, Teradata, Informatica; Hive, HBase; Impala, Solr, Spark)
- Operational expertise in the use of SQL/HQL/NoSQL
- Demonstrated experience translating business and technical requirements into comprehensive data reporting strategies and analytic solutions.
- Extensive background and expertise in developing and managing data technologies, technical operations, reusable data services, and related tools and technologies.
- Excellent communication and interpersonal skills as well as leadership and management abilities
- 5+ years of IT experience; 4 focused in areas of Big Data architecture and operation.
- 3+ years architecting big data solutions with a major RDBMS platform (Oracle, Teradata)
- 3+ Experience with Hadoop ecosystem frameworks such as: MapReduce, Hive, Yarn, HBase, Pig Latin, Storm, Kafka, Flume HCatalog, Spark, Oozie, Zookeeper, Java and others
- 2 years functioning as a Technical Leader/Manager role in a Big Data environment.
- Working knowledge of setting up, configuring and optimization of Hadoop clusters
- Experience with one or more of Hadoop-capable BI tools such as Datameer, Platfora, Splunk, and similar
- Experience with cloud and platform services, including self-provisioning, dynamic resource allocation and usage metering
- Ability to lead large scale, long-term data projects and work effectively leading cross-functional teams
- Experience executing or managing data operations is a plus