Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Python and Apache Spark.
• Build high-performance data workflows with pushdown optimization, streaming, replication, partitioning, and clustering techniques.
• Evaluate and integrate modern data platforms and tools into our enterprise architecture.
• Engineer and manage feature pipelines to support real-time fraud detection systems.
• Design data models and processing strategies that align with distributed system principles, ensuring scalability, consistency, and performance across large-scale environments.
• Develop solutions that are production-ready, maintainable, and built with observability and operational excellence in mind.
• Apply clean code practices, SOLID principles, and architecture patterns to deliver robust and extensible systems.
• Participate in code reviews, testing, and deployment activities.
• Contribute to architectural decisions and continuous improvement initiatives.

Required Skills

• 3+ years of professional experience in Python development with a strong focus on data engineering.
• Hands-on expertise with Apache Spark and Big Data processing.
• Strong SQL skills and experience with both relational and distributed data systems.
• Solid understanding of software engineering principles, clean code, and design patterns.
• Experience in system design and architecture for scalable, data-intensive applications.
• Ability to evaluate new technologies and recommend solutions aligned with enterprise needs.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration abilities.

Beneficial Skills (Nice to have)

• Exposure to cloud platforms such as Snowflake, Databricks, or similar.
• Familiarity with Kafka, Redis, Airflow, Jenkins, and Git.
• Understanding of observability practices and tools like Open Telemetry, Grafana, Loki, or Tempo, with a mindset for embedding observability into system design.
• Awareness of Kubernetes, Helm, GitOps, and containerization concepts.
• Background in real-time systems, or financial services domains.
• Good sense of humor