Lead Data Engineer - AWS Cloud

NorthBay Solutions is presently in search of a Cloud Data Lead with 8 to 10 years of experience, possessing expertise in AWS, to become a valuable addition to our growing team of analytics specialists. The ideal candidate enjoys tackling intricate problems across a wide range of technologies, is passionate about experimentation, and is seeking a new challenge that pushes the boundaries of their skills. The selected candidate should possess the capability to create and uphold data analytic roadmaps and structures that align with business and technology goals. Demonstrating a natural inquisitiveness and a willingness to delve into underlying data for valuable insights, the candidate will collaborate with functional areas to drive actionable outcomes. Responsibilities for the selected candidate include leading and enhancing both data architecture and pipelines, with a substantial demand for their vital role in optimizing data flow and collection across various teams and products. Objectives of this role

• Develop and implement pipelines that extract, transform, and load data into an information product that helps the organization reach its strategic goals
• Focus on ingesting, storing, processing, and analyzing large datasets
• Create scalable, high-performance web services for tracking data
• Translate complex technical and functional requirements into detailed designs
• Investigate alternatives for data storing and processing to ensure the implementation of the most streamlined solutions

• Develop and maintain data pipelines using ETL processes
• Ensure conformity withdata security and privacybest practices, access control,data-versioning,and regulatory compliance within AWS environments.
• Work closely with the data science team to implement data analytics pipelines
• Work closely with DevOps professionals to incorporate and deploy solutions based on AWS, guaranteeing reliability, scalability, and optimizing performance.
• Analyze the vast number of data stores to uncover insights
• Focus on ingesting, storing, processing, and analyzing large datasets
• Investigate alternatives for data storing and processing to ensure the implementation of the most streamlined solutions
• Capability to comprehend and express requirements to both technical and non-technical audiences.
• Proficient in stakeholder management and communication, involving prioritization, problem-solving, and building interpersonal relationships.
• Convert business requirements into technical specifications and supervise the implementation of solutions based on AWS, collaborating closely with cross-functional teams.
• Create and deploy scalable, secure, and efficient AWS architectures to facilitate machine learning initiatives.

• Strong SQL writing skills
• Experience with Python, PySpark, Delta Lake(Must)
• Experience with streaming frameworks such as Kafka
• Experience with specialized data architecture (e.g., data lake, lake house, data fabric, data mesh, etc.)
• Strong data engineering skills with the Amazon cloud platform
• Demonstrated expertise with AWS services like S3, Lambda, EC2, Redshift/Redshift Spectrum, Glue, EMR, Athena, MSK, DMS, SSM, EventBus, Lake Formation, SageMaker, etc.
• A strong understanding of data warehousing and data modelling techniques, and ETL processes
• An in-depth understanding of large-scale data sets, including both structured and unstructured data
• Knowledge and experience of delivering CI/CD and DevOps capabilities in a data environment
• Be motivated and passionate about technology.
• Team player with strong integrity and professionalism who can work well with others in the group.
• Self-starter with a proven ability to take initiative and strong problem-solving skills.
• Knowledge of industry-wide visualization and analytics tools (ex: Tableau, etc.)
• Good interpersonal skills and a positive attitude
• Should have experience in Jira and Agile development methodologies.
• Must be open to learning and have the ability to grasp new technologies quickly.
• Be motivated and passionate about technology.