Previous Job
Ref No.: 19-11067
Location: New York, New York
The client is looking for Airflow Developer (This is an Open-source Workflow management platform). A potential candidate, who knows and has done a lot of work on Apache Airflow, clustering, structure management, mapping and using front end tools specially React.
Expected to do:
  • Minimum 2-3 years of experience on Apache Airflow in recent project.
  • Develop guidelines for Airflow cluster and DAG's. Performance tuning of the DAG's and task implementation.
  • Develop DAG - data pipeline to on-board and change management of datasets.
  • Experience with Apache Airflow, including installing, configuring and monitoring Airflow cluster.
  • Implement CI/CD for Airflow DAG deployment and test automation frameworks.
  • Build DAG to orchestrate augmented pipeline implementing models in pipeline, running reconciliation, DQ, etc.  
  • understanding of airflow rest services and integration of airflow of platform eco-system
  • Orchestrating the Airflow / workflow in hybrid cloud environment is big plus
  • Experience working in an Agile or iterative approach to delivery preferred.
Should be hands on with:
  • Python & Pandas, with demonstrated experience in ETL.
  • Ability to maintain clean and secure data environments.
  • Working in a hybrid cloud environment.
  • Develop and contribute across the stack (Javascript/React, Java back end)
  • Experience in setting and Schedulers: celery, Dask, Mesos
  • Messaging queue : RabbitMQ or Redis
  • Developing hooks, templates, Alerts and scheduling for pipelines in Airflow