Previous Job
PySpark (Python)/Scala Spark Developer
Ref No.: 18-04704
Location: New York City, New York
KAYGEN is an emerging leader in providing top talent for technology based staffing services. We specialize in providing high-volume contingent staffing, direct hire staffing and project based solutions to companies worldwide ranging from startups to Fortune 500 and Managed Service Providers (MSP) across a wide variety of industries

Job Description.
  1. PySpark (Python)/Scala Spark:
    1. Performing ETL jobs in Batch Modes.
    2. Performing ETL using Real-Time Spark streaming.
    3. Python/Scala programming (intermediate level)
    4. Hands on experience in Spark version 1.6 and >2.
    5. Working with different file formats: Hive, Parquet, CSV, JSON, ORC, Avro etc. Compression techniques.
    6. Integrating PySpark with different data sources, example: oracle, postgres, mysql, MS sqlserver etc.
    7. SparkSQL, DataFrames & Datasets.
    8. Performance Tuning techniques.
  2. Good to have:
  3. Basic Client techniques in spark. (optional for Data Engineering)
  4. Working with Hive, No Sql Databases like Hbase, Cassandra etc
At KAYGEN, we are always looking for dynamic, talented and experienced individuals. We invite you to join our team of talented IT professionals, consulting at client locations across the globe. Our culture is team-orientated; we strive to stand by our core values of respect, honesty and integrity. Our team of experienced staffing experts will work with you to find you the best opportunity. For more information please visit us at