ETL Dev:

• Design and develop scalable ETL pipelines using AWS services such as:
  • AWS Glue for serverless data integration
  • AWS Lambda for lightweight transformations
  • Amazon S3 for data lake storage
  • Amazon Redshift or RDS for data warehousing
• Integrate data from diverse sources including APIs, databases, and flat files into AWS-based data platforms.
• Implement data transformation logic using PySpark, Python, or SQL within AWS Glue or Lambda.
• Monitor, schedule, and orchestrate ETL workflows using AWS Step Functions, Glue Workflows, or Apache Airflow on Amazon MWAA.
• Ensure data quality, consistency, and lineage using AWS Glue Data Catalog and AWS Lake Formation.
• Optimize ETL performance and cost-efficiency through partitioning, parallelism, and resource tuning.
• Implement security best practices including encryption, IAM roles, and VPC configurations.
• Collaborate with data engineers, analysts, and DevOps teams to support analytics and reporting needs.
• Document ETL processes, data flows, and architecture using tools like AWS Architecture Diagrams or Confluence.