Data Engineer

Agilis are currently working exclusively with a key client who are a leading technology consultancy in their search for a Data Engineer. This is a fantastic opportunity to join a fast growing, forward thinking company and helping them take their Data engineering to the next level! Job Description: We are seeking a highly skilled and motivated Data Engineer to join a dynamic team. The ideal candidate will have a strong background in SQL, Python, ETL processes, and data integration, ideally in Databricks. You will play a crucial role in continuing an exciting project designing, developing, and maintaining data infrastructure to ensure the seamless inflow, data sanitation/consolidation and automated report production for clients. Key Responsibilities: Design and Development:

• Design, develop, and maintain scalable ETL pipelines to process and integrate data from various sources.
• Implement data validation routines to ensure data quality and integrity.
• Develop and optimize SQL queries for data extraction, transformation, and loading.

• Integrate data from multiple sources, including APIs & relational databases.
• Collaborate with cross-functional teams to gather and understand data requirements.

• Design and maintain relational database schemas to support business needs.
• Ensure efficient storage, retrieval, and management of large datasets.

• Develop and maintain APIs for data access and integration.
• Utilize tools like Postman for API testing and documentation.
• A good understanding of working with APIs: Ensure robust and efficient API integration and management.

• Manage permissions and access controls within Databricks to ensure data security and compliance

• Work with data analysts to provide clean and well-structured data for analysis.
• Develop and maintain documentation for data processes and workflows.
• Develop and maintain automatic report production to ensure seamless delivery of critical data

• Collaborate with colleagues to gather requirements and translate them into technical specifications.
• Communicate effectively with team members to ensure alignment on data initiatives

• Bachelor's degree or equivalent experience in Computer Science, Information Technology, or a related field.
• Proven experience as a Data Engineer or in a similar role.
• Strong proficiency in SQL or Python or ideally both.
• Experience with ETL processes and tools.
• Knowledge of data validation routines and data integration techniques.
• Familiarity with relational database design and management.
• Experience with API development and testing using tools like Postman.
• Experience of Databricks or similar data platforms desirable
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills.