Senior Data Engineer

Exp: 8+ Years Required Skills: Databricks (Expert) + Pyspark (Advanced) + MongoDB (Advanced) + Scala (Understands) As a Data Engineer (Cloud Native), you will be responsible for designing, developing, and maintaining our cloud-based data infrastructure and systems. Your primary focus will be on leveraging Azure storage, Databricks, NoSQL databases like MongoDB, and other cloud native technologies to build scalable and efficient data pipelines. You will collaborate closely with cross-functional teams, analysts, and software engineers, to ensure optimal data flow and availability for analytics and reporting purposes. Experience Range: 8+ years of experience in relevant skills and technologies. Responsibilities: • Design, develop, and maintain cloud native data infrastructure and systems using Azure storage, Databricks, NoSQL databases like MongoDB, and related technologies. • Develop and implement data ingestion and transformation processes to ensure efficient and accurate data flow from various sources into the data platform. • Develop event based pipeline that process data in real-time and apply business logics • Collaborate with cross-functional teams to understand data requirements and develop appropriate data models and schemas. • Optimize data pipelines for performance, scalability, and reliability, ensuring high availability and low latency. • Implement and maintain data security and privacy measures, adhering to industry best practices and compliance standards. • Monitor and troubleshoot data pipelines, identifying and resolving issues proactively to ensure uninterrupted data flow. • Work closely with analysts to understand data usage patterns and optimize data structures and schemas for analytics and reporting purposes. • Implement and maintain data orchestration using tools like Azure DevOps and Kubernetes, ensuring smooth deployment and management of data processes. • Stay up to date with the latest trends and advancements in cloud native data engineering technologies, evaluating and recommending new tools and techniques as needed. Qualifications: • Bachelor's degree in Computer Science, Engineering, or a related field. A master's degree is a plus. • Extensive experience working as a Data Engineer , preferably in a cloud native environment. • Strong proficiency in Azure storage, Databricks, NoSQL databases like MongoDB, and related technologies. • Proven experience in data ingestion and transformation methods, including ETL/ELT processes. • Solid understanding of data modeling and schema design principles. • Familiarity with cloud-based deployment and orchestration tools like Azure DevOps and Kubernetes. • Strong analytical and problem-solving skills, with the ability to troubleshoot and resolve complex data engineering issues. • Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams. • Experience with Agile development methodologies is a plus.

• Experience with Metadata Driven Framework is a plus.
• Familiarity with Data Marketplace and Data Mesh architecture is a plus.