ETL Test Engineer - Automation/Manual Testing

Job Description : As an ETL Test Engineer, you will play a critical role in ensuring the quality, accuracy, and integrity of data as it moves through the Extract, Transform, and Load (ETL) pipeline. This involves validating data at various stages of the ETL process, including the extraction from source systems, transformation through logic processes, and loading into the target systems, ensuring that all data conforms to business rules and is free of errors. You will engage both manual and automated testing techniques to ensure that the ETL processes operate as expected and meet the business requirements. You will work closely with data engineering teams, collaborating on a sustainable and efficient testing approach, while also being involved in automation to streamline the testing process.This role requires a blend of technical skills, domain knowledge, and an understanding of various tools, technologies, and methodologies related to data testing. The candidate will need to stay abreast of current trends and best practices in data testing, including data automation, cloud environments, and Agile methodologies.Key Responsibilities : 1. Data Validation and Testing : - Conduct thorough data validation scenarios, ensuring that the data being extracted from source systems, transformed according to business logic, and loaded into target databases is accurate and aligned with expected outcomes.- Focus on validating the ingestion pipelines and the data transformation processes. This includes verifying that transformation rules are correctly applied, and data accuracy is maintained during each stage of the pipeline.2. Automation of Testing : - Develop and execute automated tests to test ETL workflows across multiple platforms and technologies. This automation will help reduce the manual effort and improve testing efficiency across the development lifecycle.- Use appropriate automated testing tools to ensure that testing is seamless and repeatable, and integrate them into continuous integration/continuous deployment (CI/CD) pipelines for efficiency.3. Collaboration and Issue Management : - Work closely with data engineering teams, data architects, and business stakeholders to understand the business requirements and the expected outcomes of the ETL processes. You will ensure the tests reflect these requirements accurately.- Handle test case management and maintain documentation on testing procedures. Report issues and defects in a clear and structured manner to relevant teams, ensuring timely resolution.4. Metadata Verification : - Test systems that capture, verify, and maintain metadata for data quality, lineage, and traceability. Ensure that metadata is correctly recorded at all stages of the ETL pipeline.- Ensure that the systems under test capture and report the field-generated metadata that is required for regulatory compliance and future data audits.Technology Familiarity : 1. Backend and Frontend Technologies : Have a working knowledge of various backend and frontend technologies, libraries, and web servers. Understanding the integration points between frontend applications, backend services, and the data pipeline is essential.2. Databases and APIs : Have experience with databases like DB2, MySQL, PostgreSQL, Snowflake, and cloud data platforms like AWS S3 and Azure. In addition, knowledge of APIs, web services, and microservices is important for validating data flows between systems.3. UI/UX Design : Be familiar with testing the user interface (UI) and user experience (UX) design of web-based applications that interact with data, ensuring that the data displays accurately to end-users.4. Modern Testing Practices : - Ensure testing best practices and modern methodologies are followed, with an emphasis on Agile software development. - Experience with Agile environments is necessary, as this will ensure timely testing cycles and collaboration with the development teams.- Use modern test development tools such as GitLab and other integrated products to streamline and automate testing procedures, managing test cases, and tracking defects.5. Cloud Testing & Migration : - Experience in testing applications and data on cloud platforms, such as AWS and Azure. You will be involved in testing cloud deployments, migrations, and ensuring that the cloud architecture is effectively tested across different stages of development.- Work on cloud migration testing, ensuring data is accurately moved from legacy systems to the cloud while maintaining data integrity and quality.Required Skills and Qualifications : - Insurance Domain Knowledge : Knowledge of the insurance domain is highly valuable, as understanding the specific data requirements and processes in this industry is essential for testing.- Mainframe Background : Strong experience with mainframe systems, particularly in extracting files from Mainframe systems and comparing them with databases like DB2. This expertise is crucial for working with legacy data and integrating it into modern data systems.- Python : Proficiency in Python for scripting and automating data-related testing. Python is commonly used in data automation, and familiarity with libraries and frameworks related to testing is important.- Database Knowledge : Expertise in databases such as DB2, Aurora, SQL, MySQL, PostgreSQL, and cloud data platforms like Databricks and Snowflake. A strong grasp of SQL is essential for writing test queries and validating data against expected results.- Power BI, SSRS, SSIS : Experience with Power BI, SSRS, and SSIS is necessary for report testing, ensuring that reports are generated accurately and reflect the transformed data.- Messaging Services : Understanding of SNS, SQS, and Kafka for handling event-driven architectures and messaging systems that may be part of the ETL processes.- ETL Tools : Experience with Informatica or similar ETL tools for developing, managing, and testing data pipelines.- AWS Cloud : Familiarity with AWS Cloud, particularly S3 for data storage and processing, is critical for cloud-based data testing.Additional Preferred Skills : - API/Microservices Testing : Hands-on experience with RestAssured and other testing tools for testing APIs and microservices that interact with ETL processes.- Data Visualization Testing : Expertise in testing BI reports and data visualization tools for accurate representation of data in reports.- Cloud Data Testing : Experience in both AWS Cloud and Azure Cloud data testing, ensuring data in the cloud environment is valid and consistent.- Automation Testing : Proficiency in Selenium (Java-based testing) and Tosca for ensuring data integrity in automated test scenarios. (ref:hirist.tech)