Data Engineer - Azure, AWS, Databricks
Job details
Responsibilities:
Design, develop, and maintain scalable and efficient data pipelines using cloud-based technologies, including Azure and AWS.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and implement data solutions.
Implement and optimize ETL processes for the extraction, transformation, and loading of large datasets.
Work with big data technologies such as Apache Spark and Databricks for data processing and analysis.
Ensure data quality, integrity, and security throughout the entire data lifecycle.
Collaborate with cross-functional teams to design and implement data storage solutions based on business needs.
Optimize and tune data pipelines for performance and efficiency.
Stay updated on the latest trends and advancements in data engineering and cloud technologies.
Mentor and provide guidance to junior members of the data engineering team.
Participate in code reviews and ensure adherence to coding standards.
Requirements:
Minimum 5 years of experience in data engineering with a focus on cloud platforms (Azure, AWS).
Bachelor's degree in Computer Science or a related field.
Proficiency in cloud-based data services and tools, including Azure Data Factory, AWS Glue, and Databricks.
Strong programming skills in languages such as Python, SQL, or Scala.
Experience with big data processing frameworks (e.g., Apache Spark).
Knowledge of data modeling, database design, and optimization.
Familiarity with data warehousing concepts and technologies.
Excellent problem-solving and analytical skills.
Ability to work collaboratively in a team environment.
Strong communication and documentation skills.
Preferred Skills:
Industry certifications related to data engineering and cloud platforms (e.g., Microsoft Certified: Azure Data Engineer Associate, AWS Certified Big Data - Specialty).
Experience with containerization and orchestration tools (e.g., Docker, Kubernetes).
Familiarity with data streaming technologies (e.g., Apache Kafka).
Knowledge of data governance and compliance.
Understanding of DevOps practices in the context of data engineering.
#J-18808-Ljbffr IT & Technology
Apply safely
To stay safe in your job search, information on common scams and to get free expert advice, we recommend that you visit SAFERjobs, a non-profit, joint industry and law enforcement organization working to combat job scams.
