AWS Data Engineer

Delta Computer Consulting • West Carson, California, United States • 11h ago

MUST be local to Torrance, CA, or Orange County, CA, or Los Angeles
MUST have a minimum of 10 years of experience

Job Description:

Develop and Maintain Data Integration Solutions
Design and implement data integration workflows using AWS Glue/EMR, Lambda, Redshift
Demonstrate proficiency in Pyspark, Apache Spark and Python for data processing large datasets
Ensure data is accurately and efficiently extracted, transformed, and loaded into target systems
Ensure Data Quality and Integrity
Validate and cleanse data to maintain high data quality
Ensure data quality and integrity by implementing monitoring, validation, and error handling mechanisms within data pipelines
Optimize Data Integration Processes
Enhance the performance, optimization of data workflows to meet SLAs, scalability of data integration processes and cost-efficiency on AWS cloud infrastructure
Identify and resolve performance bottlenecks, fine-tuning queries, and optimizing data processing to enhance Redshift's performance
Regularly review and refine integration processes to improve efficiency
Support Business Intelligence and Analytics
Translate business requirements to technical specifications and coded data pipelines
Ensure timely availability of integrated data for business intelligence and analytics
Collaborate with data analysts and business stakeholders to meet their data requirements
Maintain Documentation and Compliance
Document all data integration processes, workflows, and technical & system specifications
Ensure compliance with data governance policies, industry standards, and regulatory requirements

What will this person be working on:

The IT Data Integration Engineer / Developer is tasked with designing, developing, and managing data integration processes to ensure seamless data flow and accessibility across the organization. This role is pivotal in integrating data from diverse sources, transforming it to meet business requirements, and loading it into target systems such as data warehouses or data lakes. The aim is to support the organization's data-driven decision-making by providing high-quality, consistent, and accessible data.

Required Skills:

Bachelor's degree in computer science, information technology, or a related field. A master's degree can be advantageous
7-10+ years of experience in data engineering, database design, ETL processes
5+ in programming languages such as PySpark, Python
5+ years of experience with AWS tools and technologies (S3, EMR, Glue, Athena, RedShift, Postgres, RDS, Lambda, PySpark)
3+ years of experience of working with databases/ data marts/data warehouses
Proven experience in ETL development, system integration, and CI/CD implementation
Experience in complex database objects to move the changed data across multiple environments
Solid understanding of data security, privacy, and compliance
Excellent problem-solving and communication skills
Display good communication skills to collaborate with multi-functional teams effectively
Participate in agile development processes including sprint planning stand-ups and retrospectives
Provide technical guidance and mentorship to junior developers
Attention to detail and a commitment to data quality
Continuous learning mindset to keep up with evolving technologies and best practices in data engineering
3+ years of big 4 consulting experience is preferred
A stable work history with large enterprise organizations