Data Engineer

Insight Global

San Francisco



The ideal candidate has a strong software development foundation, experience building a real time data pipeline, and proficiency using Spark with Python or Spark with Scala.

In this role: 

*You will join the early stage of projects and participate in defining the platform for the next generation products 

*You will build API and processing pipeline for multi-tenant, high-volume, and real-time system on Spark and Hadoop platform

*You will transform complex analytical models into scalable, production-ready solutions

*You will continuously integrate and ship code into the cloud environment

You will develop applications from ground up using a modern technology stack such as Scala, Spark, Postgres, Angular 2, Play, and NoSQL

*You will participate in defining the architecture of modern tools and runtime for operating your code such as AWS, Docker, Puppet, and Terraform 

*You will work directly with Product Owners to deliver products in a collaborative and agile environment 

Who you are: 

*You are passionate to be part of cutting edge projects and are motivated by delivering world-class products with great architecture on an aggressive schedule 

*You are not intimidated by challenges; thrives even under pressure; is passionate about your craft; and hyper focused on delivering exceptional results 

*You love to learn new technologies and mentor junior engineers to raise the bar on your team 

*You use, or are passionate about learning, functional programming, AWS, Docker, Spark, Alluxio, Scala, Akka, Kafka, Cassandra, ElasticSearch, Hadoop, Yarn, and Mesos – and anything else that will help you solve big, messy problems 

Job Requirements

*Bachelors or higher in Computer Science, Mathematics, or other relevant field 

*Expert level knowledge of JVM and OOP 

*2+ years of experience in Scala or other functional languages 

*Experience in Hadoop stack: HDFS, Spark, NoSQL 

*Experience in designing API and documenting through Open API/Swagger 

*Experience in Unix/Linux including basic commands and scripting 

*Strong knowledge of source control systems and best practices (Git, Merging, Branching, Pull Requests, etc.) 

*Strong knowledge and understanding of algorithm optimization and distributed systems 

*Solid understanding of continuous integration, deployment and monitoring 

*Excellent communication skills


Optional Requirements

*Designing mission-critical system using AWS, Docker, microservices architecture 

*Experience with Cats or Scalaz and know how to apply them in the messy world of FP-OOP

*Experience working with Play Scala or Akka HTTP

*Experience working with Shapeless and can explain generic programming

*Knowledge of Statistics 

*Experience in Fintech