Data Engineer

Looking for immediate joiners only Experience; - 2 yrs to 6 Yrs

• Primary Skills: - Python,
• Pyspark
• Scala / Java
• Azure Databricks
• Sql

Work mode: - Remote Role: Data Engineer Description: We are seeking a skilled Databricks Data Engineer with expertise in Scala to design, implement, and maintain scalable data solutions on the Databricks platform. The ideal candidate will have a strong background in data engineering, distributed computing, and cloud technologies, with a focus on leveraging Databricks and Apache Spark for data processing and analytics. The project in which the person will be assigned is a data engineering effort to support one of clients with timeseries data migration. Responsibilities: · Design and Implementation: Develop and maintain robust data pipelines using Databricks Scala, ensuring efficient data processing and transformation. · Data Modeling: Create and optimize data models to support analytical workloads and reporting needs. · Collaboration: Work closely with cross-functional teams, including analysts, and business stakeholders, to deliver data-driven solutions that meet business objectives. · Performance Optimization: Monitor and enhance the performance of existing data pipelines and applications, implementing best practices for scalability and efficiency. · ETL Development: Design and implement ETL processes to extract, transform, and load data from various sources into Databricks. · Automation: Utilize DevOps practices to automate deployment processes and monitor data pipelines for reliability. · Documentation: Maintain clear documentation of data architecture, processes, and workflows to ensure knowledge sharing within the team. Skills: · Programming Languages: Proficiency in Scala is required; experience with Python or SQL is a plus. · Databricks Experience: Hands-on experience with the Databricks platform and Apache Spark for big data processing. · Cloud Platforms: Strong knowledge of Azure in relation to data services. · Distributed Computing: Understanding of distributed computing principles and big data technologies. · Data Warehousing: Familiarity with data warehousing concepts and tools. · Timeseries DB: Knowledge of timeseries DB like influx db · Columnar Data store: Knowledge of columnar datastore like Apache Druid or Imply Polaris