Job Title: Data Engineering Lead
Location: Hyderabad
Experience: 6-8 Years
Company Overview:
UpGrad is a leader in the online education space, providing high-quality learning experiences to
individuals and businesses worldwide. We are looking for a Data Engineering Lead to drive the
development of data solutions that will shape the future of our data infrastructure and analytics
capabilities.
Role Overview:
As the Data Engineering Lead, you will play a pivotal role in managing and building scalable
data architectures. You will be responsible for the development and maintenance of data
pipelines, data integration processes, and ensure the smooth functioning of data platforms. The
ideal candidate will have hands-on experience with Apache Spark (Scala), Apache Airflow, AWS
Redshift, AWS Glue, EMR, and data modeling.
Key Responsibilities:
● Lead the design, development, and optimization of scalable data pipelines to support
various data-driven initiatives.
● Implement efficient Data Pipelines to integrate and transform data from multiple sources
into AWS Redshift.
● Architect data solutions using Apache Spark (Scala) and manage workflows with Apache
Airflow.
● Architect real-time data streaming solutions using Kafka or Kinesis to ensure efficient
and scalable data flow.
● Utilize AWS services such as Redshift, Glue, EMR to build robust and scalable data
platforms.
● Oversee the configuration and management of Bitbucket for version control and Jira for
project tracking and issue management.
● Collaborate with cross-functional teams, including product teams, Data analytics and
Data science, to deliver data solutions that support business needs.
● Ensure data quality, consistency, and availability across all systems.
● Mentor junior data engineers, fostering technical growth within the team.
● Take ownership of Data modeling, ensuring data is structured to optimize both
operational and analytical use cases.
● Stay updated with the latest advancements in big data technologies and continuously
improve the team’s practices.
Required Skills and Experience:
● 6-8 years of experience in data engineering or related fields.
● Proficiency with Apache Spark (Scala) for large-scale data processing, handling datasets
up to Terabytes in size.
● Experience with real-time data processing tools like Kafka or Kinesis.
● Expertise in Apache Airflow for workflow orchestration and scheduling.
● Strong experience with AWS Redshift, Glue, and EMR for cloud-based data processing.
● Solid understanding of Data Pipeline and data integration processes.
● Hands-on experience with SQL and Data modeling best practices.
● Excellent problem-solving abilities and a strategic approach to technical challenges.
● Strong leadership and communication skills to manage projects and collaborate with
stakeholders.
Preferred Qualifications:
B-tech, M-tech or MCA
Good to have experience in education technology or a similar fast-paced industry.
Familiarity with other AWS services like Step Functions, Lambda, S3, and Athena.
Knowledge of Python for automation or additional scripting tasks.