Data Engineer

Title: Data Engineer Company: Alifcloud IT Consulting Location: Remote Salary- up to 7 lpa About Alif Cloud IT Consulting: Alif Cloud IT Consulting is a leading provider of Microsoft Services Globally, Working with Microsoft and Distributors directly. Majorly focused on Cloud and Security solutions. We are seeking a highly experienced and skilled Azure Solution Architect to join our dynamic team and lead the design and implementation of Azure-based solutions. Lead the team Project initiation and delivery. Description: As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining the data infrastructure and pipelines that power our organization's data-driven initiatives. You will collaborate closely with data scientists, analysts, and other stakeholders to ensure that our data architecture meets the needs of the business. Responsibilities:

1. Data Pipeline Development:

• Design, develop, and maintain scalable data pipelines and ETL processes to ingest, transform, and load data from various sources into our data warehouse or data lake.
• Optimize data pipeline performance, reliability, and scalability to support the growing volume and complexity of data.

1. Data Modeling and Architecture:

• Design and implement data models, schemas, and data structures to support analytical and reporting requirements.
• Collaborate with data scientists and analysts to understand their data requirements and translate them into efficient data models.

1. Data Integration:

• Integrate data from multiple sources, including databases, APIs, flat files, and streaming sources, ensuring data consistency, integrity, and quality.
• Implement data governance and quality control processes to ensure the accuracy and reliability of the data.

1. Data Warehousing and Storage:

• Design, build, and maintain data warehousing solutions, such as data lakes, data marts, or data warehouses, using technologies like Amazon Redshift, Google BigQuery, or Snowflake.
• Optimize data storage and retrieval mechanisms for performance, cost-efficiency, and scalability.

1. Data Infrastructure Management:

• Manage and monitor data infrastructure components, including databases, clusters, servers, and storage systems, to ensure optimal performance, availability, and security.
• Implement and maintain data security measures, including access controls, encryption, and compliance with data privacy regulations.

1. Data Visualization and Reporting:

• Work closely with business intelligence developers and analysts to enable data visualization and reporting capabilities, integrating data pipelines with BI tools such as Tableau, Power BI, or Looker.

1. Collaboration and Documentation:

• Collaborate with cross-functional teams, including data scientists, analysts, software engineers, and business stakeholders, to understand data requirements and deliver solutions that meet business needs.
• Document data architecture, processes, and best practices to facilitate knowledge sharing and ensure the scalability and maintainability of data solutions.

• Bachelor's degree in Computer Science, Engineering, or a related field. Master's degree preferred.
• Proven experience as a Data Engineer or similar role, with a strong background in data modeling, ETL development, and data warehousing.
• Proficiency in programming languages such as Python, SQL, Java, or Scala.
• Experience with cloud platforms and services, such as AWS, Azure, or Google Cloud Platform.
• Hands-on experience with big data technologies, such as Hadoop, Spark, Kafka, or Apache Airflow.
• Strong understanding of database systems, data architecture principles, and best practices for data management and governance.
• Excellent problem-solving skills, attention to detail, and ability to work in a fast-paced, collaborative environment.
• Strong communication and interpersonal skills, with the ability to effectively communicate complex technical concepts to non-technical stakeholders.

• Optional certifications in relevant technologies, such as AWS Certified Big Data - Specialty, Google Professional Data Engineer, or Microsoft Certified: Azure Data Engineer Associate, are a plus.