Job Description:

• Design and implement scalable real-time data processing pipelines using Apache Kafka, Kafka Streams (kstream), and Apache Flink.
• Collaborate with data engineers, software developers, and data scientists to build robust streaming solutions that meet business needs.
• Optimize and monitor streaming jobs for performance, reliability, and scalability.
• Ensure data quality, consistency, and governance in real-time pipelines.
• Develop metrics and monitoring dashboards to ensure observability across streaming systems.
• Work on integration of real-time data pipelines with downstream data stores (e.g., PostgreSQL, Elasticsearch, S3, DWHs).
• Participate in architecture discussions and help define best practices for stream processing.

Required Skills:

• 3+ years of experience in real-time data processing and streaming analytics.
• Strong hands-on experience with Apache Kafka, particularly Kafka Streams (kstream).
• Solid knowledge and experience with Apache Flink for stateful stream processing.
• Proficient in Java or Scala (Python a plus).
• Experience designing event-driven architectures and working with distributed systems.
• Familiarity with containerization and orchestration (e.g., Docker, Kubernetes).
• Understanding of exactly-once semantics, state management, and checkpointing in stream processing.