Position: Azure Data Engineer Lead and Developer
Location : Chennai(Hybrid-3 days a week)/ If Non Local, need quarterly once, sit with team for a week or two
Duration: Full Time
Positions: 7
Notice Period: 15-40 days (Non- Negotiable)
**Should have good Comms skills- Its client facing role**
1st preference will be given to local candidates
Candidates from Chennai who can visit office 3 days a week.
If Non Local, need quarterly once, sit with team for a week or two (Accommodation and travel will be taken care of by MIT)
ADF Lead 1 and 2 - 8 to 12 years ADF, ADLS, Pyspark, Synapse (Optional)
Dev 1,2,3,4,5 - 5 to 8 years ADF, ADLS, Pyspark, Synapse (Optional)
Job Summary:
We are seeking a talented and experienced Azure Data Engineer to join our team. As an Azure Data Engineer, you will be responsible for designing, implementing, and managing data solutions on the Microsoft Azure cloud platform. You will play a critical role in ensuring that our data infrastructure is robust, scalable, and capable of supporting our data analytics and business intelligence needs.
Responsibilities:
Data Modelling: Design, develop and maintain data models, schemas, and data warehouses in Azure SQL Data Warehouse or other relevant data warehousing solutions to support analytical and reporting needs. Ensure data is organized efficiently for optimal query performance.
Data Transformation & ETL: Implement data transformation workflows and processes, including data cleansing, enrichment, and aggregation, using Azure Data Factory, Azure Databricks, or other ETL relevant tools to move data between systems for analytics and reporting.
Data Ingestion and Integration: Design, develop, and maintain data pipelines for ingesting data from various sources, both structured and unstructured, into Azure data platforms ensuring data quality and reliability.
Data Storage and Management: Design, implement and manage data storage solutions on Azure, including Azure SQL Database, Azure Data Lake Storage, Cosmos DB, and Azure Blob Storage, while optimizing for performance, scalability, and cost-efficiency.
Data Security: Implement data security measures, including encryption, access control, and compliance with data privacy regulations like GDPR, HIPAA, etc.
Monitoring: Monitor data pipelines and storage systems for performance and reliability, proactively identifying and resolving issues. Optimize data solutions for cost-effectiveness.
Performance Optimization: Optimize data pipelines and queries for performance and cost efficiency, monitoring and troubleshooting as needed to meet service-level agreements (SLAs).
Collaboration: Collaborate with cross-functional teams, including data scientists, analysts, and application developers, to understand their data requirements and provide them with access to clean and well-organized data that meet business requirements.
Documentation: Maintain comprehensive documentation for data pipelines, data models, configurations, processes, and storage solutions for knowledge sharing and future reference.
Continuous Learning: Stay up to date with Azure data services and emerging trends in data engineering and cloud technologies.
Qualifications:
•Bachelor’s degree in Computer Science, Information Technology, or a related field (or equivalent work experience).
•Proven experience as a Data Engineer with a focus on Azure data services.
•Strong proficiency in Azure services such as Azure Data Factory, Azure Databricks, Azure SQL Database, Azure Data Lake Storage, Azure Blob Storage, etc.
•Proficiency in ETL (Extract, Transform, Load) processes and data integration techniques.
•Experience with data modelling and data warehousing concepts.
•Knowledge of data governance and security best practices.
•Programming skills in languages such as Python, SQL, or Scala.
•Familiarity with version control systems (e.g., Git) and DevOps practices.
•Strong problem-solving and troubleshooting skills.
•Excellent communication and teamwork skills.
Preferred Qualifications:
•Azure certifications such as Azure Data Engineer Associate or Azure Solutions Architect.
Experience with DevOps practices and tools for automation and continuous integration and deployment (CI/CD)
•Experience with big data technologies like Apache Spark, Hadoop, or Hive.
•Familiarity with data visualization tools like Power BI or Tableau.