ETL Lead

We are seeking a highly skilled and experienced lead for development and management of ETL processes. The ideal candidate will have 10-12 years of experience in data engineering and ETL development . Hands on experience on Azure Data Factory (ADF), Synapse Pipelines, Databricks, Azure Data Lake, SSIS , SQL , PySpark, Synapse Spark Pools , SQL , ETL processes (SSIS), Azure Data Lake Storage Gen2 (ADLS), Delta Lake, Azure Synapse Analytics (Dedicated SQL Pool, Serverless SQL),Data Vault . This role requires a strong technical background, hands-on expertise in ETL development, leadership in mentoring and grooming a team, and the ability to drive business discussions and translate complex business logic into effective data solutions. Key Responsibilities:

• Hands-on ETL Development & Implementation:
• Lead and participate in the development, design, and implementation of ETL workflows using Azure Data Factory (ADF), Synapse Pipelines, Databricks, Azure Data Lake, SSIS , SQL , PySpark, Synapse Spark Pools
• Working knowledge on Azure Data Lake Storage Gen2 (ADLS), Delta Lake, Azure Synapse Analytics (Dedicated SQL Pool, Serverless SQL), Data Vault
• Hands-on development of complex ETL pipelines, data transformation logic, and integration processes to ensure efficient data flow and high-volume data processing.
• Implement strategies for efficient data extraction, transformation, and loading from source to destination systems, ensuring robust and scalable solutions.
• Oversee the setup, configuration, and deployment of ETL systems to ensure optimal performance.
• Monitor ETL workflows, troubleshoot issues, and optimize data processing to handle high-volume data efficiently.
• Continuously review and refine ETL pipelines to enhance performance, scalability, and fault tolerance.
• Excellent knowledge in creating stored procedure, views, function etc.

• Work closely with business stakeholders to identify complex business logic and translate it into ETL processes, ensuring the accurate and effective transformation of data according to business requirements.
• Design and implement business rules, data validation logic, and complex transformations within ETL workflows to ensure data integrity and alignment with business objectives.

• Ensure adherence to best practices for ETL design, coding standards, and data governance.
• Oversee code reviews and testing of ETL solutions to maintain high-quality standards.
• Document ETL designs, business logic, and workflow processes to ensure knowledge sharing and maintainability of systems.

• Lead, mentor, and manage a team of data engineers, providing guidance on best practices for ETL development, troubleshooting, and data pipeline optimization.
• Foster an environment of continuous learning and development, helping team members solution and prioritize their work
• Facilitate team collaboration and ensure effective communication to achieve successful project outcomes.
• Collaborate with cross-functional teams, including data architects, analysts, and business teams, to design data solutions that drive business decisions.
• Lead discussions to gather business requirements, clarify objectives, and ensure data solutions are aligned with business needs.
• Lead the estimation of work effort and resources for various data engineering projects, ensuring realistic timelines and project deliverables.
• Participate in sprint planning and tracking progress, ensuring that the team meets deadlines and expectations set by the business.
• Assess project risks, identify dependencies, and develop mitigation strategies to ensure timely delivery.

• 10-12 years of experience in data engineering and ETL development. Hands on experience on Azure Data Factory (ADF), Synapse Pipelines, Databricks, Azure Data Lake, SSIS , SQL , PySpark, Synapse Spark Pools , SQL , ETL processes (SSIS), Azure Data Lake Storage Gen2 (ADLS), Delta Lake, Azure Synapse Analytics (Dedicated SQL Pool, Serverless SQL), Data Vault
• Proven experience in designing, developing, and deploying ETL workflows and processes for high-volume, complex data environments.
• Extensive hands-on experience with high-performance data processing and data integration strategies.

• Strong hands-on experience with Azure Data Factory (ADF) , including building and managing pipelines for large-scale ETL processes with very good knowledge in Data warehousing
• Advanced proficiency in SQL , including query optimization, complex data transformations, and working with large datasets.
• Experience implementing complex business logic in ETL processes, including data validation, transformation, and enrichment.
• Experience of cloud data storage solutions such as Azure SQL Database , Azure Data Lake , and Azure Blob Storage .
• At least 2 years’ experience of ETL tools and technologies like SSIS , Databricks , Azure Synapse Analytics, PySpark, Synapse Spark Pools
• At least 2 years extensive experience on Azure Data Lake Storage Gen2 (ADLS), Delta Lake
• Experience with performance tuning, troubleshooting, and optimizing ETL pipelines to manage high-volume data.
• Excellent knowledge in creating stored procedure, views, function etc.

• Strong leadership and people management skills, with the ability to mentor and develop team members.
• Excellent communication and interpersonal skills to drive business discussions and collaborate effectively with both technical and non-technical stakeholders.
• Strong problem-solving ability, capable of tackling complex data engineering challenges and implementing efficient solutions.
• Ability to handle multiple priorities, adapt to changing requirements, and meet deadlines.