|
Data Engineer
Role Name: Data Engineer
Location: Hanover, NJ Contract Role JOB DESCRIPTION: 1 Cloud Data Architecture Development Design and implement scalable data architectures using AWS services S3 Redshift Glue Athena EMR DynamoDB Build optimize and maintain ETLELT pipelines for structured and unstructured data Develop real time and batch ingestion pipelines using AWS Glue Lambda Kinesis or Kafka 2 Data Processing Transformation Utilize Spark AWS EMRGlue Python or Scala to build transformation workflows Implement data quality checks validation rules and automated error handling mechanisms Ensure data is optimized for analytics BI and ML use cases 3 Data Warehouse Lakehouse Management Build and maintain data lakes on Amazon S3 and data warehouses on Redshift Snowflake Create and optimize database schemas partitioning compression and performance tuning Manage metadata cataloging and data lineage via AWS Glue Data Catalog or similar tools 4 DevOps Automation Implement CICD pipelines for data workflows using CodePipeline Code Build GitHub Actions or Jenkins Use Infrastructure as Code IaC tools like CloudFormation or Terraform to automate provisioning Monitor pipelines and infrastructure using CloudWatch CloudTrail and AWS Config 5 Security Compliance Governance Apply AWS security best practices including IAM roles KMS encryption VPC networking and Secrets Manager Maintain compliance with organizational data governance and regulatory standards Ensure data privacy retention policies and audit requirements are met 6 Collaboration Stakeholder Management Partner with data scientists analysts and business teams to understand data requirements Provide technical guidance on AWS data capabilities and architectural best practices Troubleshoot pipeline failures performance bottlenecks and data quality issues" | ||||||