Big Data Lead

Experience: 6 to 8 years Notice Period: Imm - 30 days Work location: Chennai/Pune/Bangalore Shift timings: 12PM - 10PM IST. Must-Have Skills: Pyspark, Databricks, SQL Nice-to Have Skills: ADF Job Description: Databricks, PySpark and Python project development work experience is a must Develop, design, tune and maintain PySpark scripts using Databricks Notebook. Expertise in Bigdata Eco systems like HDFS and Spark Experience in GitHuB repository Data Warehouse/Data Marts/Data Modelling/Analytics experience is a must Able to convert the SQL stored procedures to Python code in Pyspark frame work using Dataframes. Implementing data ingestion pipelines from multiple data sources using Azure Data Factory, Azure Databricks and other ETL tools. Developing Big Data and non-Big Data cloud-based enterprise solutions in PySpark and SparkSQL and related frameworks/libraries. Developing scalable and re-usable, self-service frameworks for data ingestion and processing. Integrating end to end data pipelines to take data from data source to target data repositories ensuring the quality and consistency of data. Processing performance analysis and optimization. Collaborate with business users, support team members, and other developers throughout the organization to help everyone understand issues that affect the data warehouse Good experience on customer interaction is required. Possesses good interpersonal and communication skills.