Lead Data Engineer

We are an agile tech company based out of the UK headquarters. At Neem, we provide flexible, low-cost (high value!), innovative IT Solutions, Consulting, and Project Management services. Neem Consulting is the delivery partner for companies requiring solutions to generate real business benefits at speed and a cost base that is impossible with large consulting organizations. Neem- We have a presence in more than 14+ countries. We expanded into India and South Africa a few years back as part of our global expansion roadmap. For more information, please visit us at https://www.neemconsulting.com/.We want to talk to you if you thrive in high-performance environments, strive to make a great impact, and work well with ambiguity, urgency, fast-paced work, and low support structures.Neem Consulting Limited is an equal opportunity employer. We welcome and encourage diversity in the workplace regardless of race, gender, religion, age, sexual orientation, gender identity, disability, or veteran status.We Neem'ers believe that core values define you and how you act and demonstrate your personality.At Neem Consulting Limited, our 'Seven C's' define our core values, no matter who or where we are... as a global organization, we work hard to ensure our people carry what it means to be part of the Neem family.Our Core Values- Can-do, Customer-Centric, Confident, Courageous, Convivial, Compassionate, Cutting-Edge.What we have for You/ as You deserve In addition to an outstanding work environment with rapid advancement potential, Neem associates enjoy exceptional compensation and benefits. Flexible working hours.:An upbeat, positive culture. Our values are central to integrity, objectivity, collaboration, results, and a no-limits mindset.Limitless growth. We work with you to help you meet your goals and advance within the company.Performance-based recognition and rewards are based on a transparent, unique appraisal cycle.Lucrative yearly bonus.WFH benefit plans.Medical Insurance coverage/ family.Mobile and internet reimbursements.Training allowances-Job role- Lead Data EngineerImmediate joiners only with a maximum of 30 days' notice.Preferably Pune & Noida onsiteExperience required- 8+ Years.Role Overview: Deliver successful projects in customer environments, such as use case deployments into production, machine learning projects, and large migrations, to deliver on the value proposition. Key Responsibilities- ARCHITECTURE AND DESIGN FOR DATA ENGINEERING AND MACHINE LEARNING PROJECTS Establishing architecture and target design for data engineering and machine learning projects. REQUIREMENT ANALYSIS, PLANNING, EFFORT AND RESOURCE NEEDS ESTIMATION Current inventory analysis, review and formalize requirements, project planning, and execution plan. ADVISORY SERVICES AND BEST PRACTICES Troubleshooting, Performance Tuning, Cost Optimization, Operational Runbooks and Mentoring LARGE MIGRATIONS Assist customers with large migrations to Databricks from Hadoop ecosystems, Data Warehouses (Teradata, DataStage, Netezza, Ab Initio), ETL engines (Informatica), SAS, SQL, DW, Cloud-based Data platforms like Redshift, Snowflake, EMR, etc DESIGN, BUILD AND OPTIMIZE DATA PIPELINES The Databricks implementation will be best in class and flexible for future iterations. PRODUCTION READINESS Assisting with production readiness for customers, including exception handling, production cutover, capture analysis, alert scheduling, and monitoring MACHINE LEARNING (ML) - MODEL REVIEW, TUNING, ML OPERATIONS AND OPTIMIZATION Build and review ML models, ML best practices, model lifecycle, ML frameworks, and deploying models in production. Must-Have: