Previous Job
Previous
Senior Staff Engineer, Big Data (657240)
Ref No.: 18-11376
Location: Santa Clara, California
Start Date / End Date: 09/24/2018 to 03/22/2019
Description:
We are looking for an experienced big data engineer to design and build our data pipeline and ETL processes, which are scalable, highly performant and able to handle diverse datasets including IoT, leveraging the latest big data technologies such as Hadoop/HDFS, Kafka, and HBase etc.
Responsibilities
Build data pipelines and ETL from heterogeneous sources including IoT using Kafka, Flume, Sqoop, Spark Streaming and custom code etc. Your solutions will support both real-time or near real-time applications
Partner with product owners, data analyst and data scientists, to better understand their requirements, find bottlenecks and come up with the best resolution, etc.
Expand and grow data platform capabilities to solve new data problems and challenges
You will work closely with development and QA teams both local and oversea to design ingestion pipelines
You will also need to mentor other team members
Desired Skills & Experience
6+ years of software development experience with minimum 2-year hands-on experience in Hadoop ecosystem and other Big Data technologies (Kafka, Cassandra, MongoDB and Elastic search)
Expert level software development experience using Java and/or Python
Hands-on experience with building big data ingestion pipelines using Hadoop, HBase, Kafka, Elastic Search etc
Knowledge and hands-on experience with relational databases (Greenplum and Oracle MySQL/Postgres etc) and database schema design
Experience with Unix/Linux and shell scripting
Knowledge of professional software engineering practices & best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes and testing
Ability to take a project from scoping requirements through actual launch of the project
BS/MS Degree in Computer Science or related field
Excellent inter-personal and teamwork skills, having experiences working with oversea teams is a plus