Senior Data Engineer

Role & Responsibilities: As a Data Engineer, you will play a pivotal role in designing, developing, and optimizing data pipelines on Google Cloud Platform (GCP). Your primary responsibility will be to ensure efficient data flow and seamless integration with GCP services while adhering to best practices and data architecture standards. Should be comfortable working with extremely abstract and ambiguous requirements. Must be capable of handling extremely changing business environments with the capability to wear multiple hats at once. Must be able to consult improvisations in various old school data engineering methods. Should be a very good team player. Design and Develop Data Pipelines: Create robust and efficient (infrastructure and cost efficient) data pipelines on GCP to facilitate the seamless flow of data. Cross-functional Collaboration: Collaborate with cross-functional teams/stakeholders to understand data requirements and deliver high-quality data solutions in the least amount of time. Collaborate with Data Architects: Work closely with client and team data architects to implement data models and pipelines and ensure their integration with GCP services. Optimize Data Storage Solutions: Utilize BigQuery, Cloud Storage, and other appropriate GCP services to optimize data storage solutions.Troubleshooting: Proactively troubleshoot data and pipeline-related issues to ensure smooth operation and reliability. Skills required: Migration of DAGs from AWS to GCP by handling version upgrade and architectural changes Configuration changes for Hive jobs and spark jobs (Python, Scala) migration based on the cloud environment Collaborate with development and operations teams to streamline the deployment process and ensure continuous integration and continuous deployment practices. Define and enforce best practices for version control, branching, and code promotion in collaboration with development teams. Work closely with cross-functional teams to troubleshoot and resolve deployment issues and optimize release processes. Good to have: Knowledge of LMS and education sector Working knowledge of IICS PRB