Designation: Lead Data Engineer
Location: Bangalore/Mohali/Gurugram
Experience: 10-14 years
Job Description
- Mentor a team of developers in building and maintaining high-quality, scalable, and secure data pipelines and analytics applications on the Databricks platform
- Work closely with data engineers, business stakeholders to understand their needs and design and develop solutions that meet those needs
- Design, develop, and test data pipelines and analytics applications using Databricks and other relevant technologies
- Responsibilities:
- Technical Leadership:
- Lead and mentor a team of data engineers, providing guidance and support on technical aspects and best practices.
- Define and implement the data architecture for the organization, ensuring scalability, performance, and security.
- Evaluate and adopt new technologies and tools within the Databricks ecosystem.
- Data Pipeline Development:
- Design and develop complex data pipelines for ingesting, processing, and transforming large datasets.
- Ensure data quality and consistency throughout the data pipeline.
- Optimize data pipelines for performance and resource efficiency.
- Collaboration and Communication:
- Collaborate with cross-functional teams (business stakeholders, downstream user teams) to understand data needs and translate them into technical implementation.
- Communicate effectively with stakeholders about data architecture, pipeline design, and operational challenges.
- Document data pipelines and procedures for maintainability and knowledge sharing.
- Problem-Solving and Innovation:
- Identify and troubleshoot issues within the data pipelines and infrastructure.
- Proactively propose improvements and optimizations to the data platform.
- Stay up-to-date on the latest trends and advancements in big data technologies.
Qualifications
- 8 + years of experience as a data engineer, with at least 2 years in a lead role.
- Proven experience working with the Databricks platform.
- Strong knowledge of big data technologies and frameworks (Spark, Kafka, etc.).
- Experience with cloud platforms (AWS, Azure, GCP) is a plus.
- Proficiency in programming languages like Python, Scala, or Java.
- Excellent communication and collaboration skills.
- Ability to work independently and as part of a team
Mandatory Skills
Data Lake, Data Pipeline, spark, data warehousing, Databricks certifications.