Responsibilities:
As part of our team, your role will involve establishing a clear vision for data engineering practices, and harmoniously aligning it with the data architecture. Collaboration with product managers is essential to comprehend business requirements and identify opportunities for data leverage. This position also entails the responsibility of creating, designing, and developing complex data processing pipelines. A cooperative relationship with the data intelligence team is necessary for designing scalable implementations and the production of data models. The role involves writing clean, iterative code, and utilizing various continuous delivery practices to deploy, support, and operate data pipelines. The selection of suitable data modeling techniques, optimization and design of physical data models, and understanding of the trade-offs between various data modeling techniques, form an integral part of this role.
Required Skills:
· You should be familiar with AWS and Azure Cloud.
· You must have extensive knowledge of Snowflake, SnowPro Core certification is a must-have.
· Proven experience with DBT.
· You should have configured and deployed Airflow and integrated various operators in airflow (especially DBT & Snowflake).
· You should be able to design build, and release pipelines, and understand of Azure DevOps Ecosystem.
· You must have an excellent understanding of Python (especially PySpark) and be able to write metadata-driven programs.
· Familiar with Data Vault (Raw, Business) also concepts like Point In Time, and Semantic Layer.
· You should be resilient in ambiguous situations and can clearly articulate the problem in a business-friendly way.
· You should believe in documenting processes managing the artifacts and evolving them over time.
Good to have skills:
- If you have experience with data visualization techniques and can communicate insights to the audience. Experience with Terraform and Hashicorp Vault is highly desirable. Knowledge of docker and Streamlit is a big plus.