Senior ETL Developer (Hybrid)

Senior ETL Developer (Hybrid)Contract Duration 2 YearsJob ResponsibilitiesSenior ETL Developer will design, develop and implement ingestion framework from Oracle data source to Azure Data Lake - initial load and incremental ETL. Tools used are:Azure Data Factory (good knowledge required) to maintain pipeline from Oracle to Azure Data LakeAzure Databricks/PySpark (good Python/PySpark knowledge required) to build transformations of raw data into curated zone in the data lakeAzure Databricks/PySpark/SQL (good SQL knowledge required) to develop and/or troubleshoot transformations of curated data into datamart modelReview the requirements, database tables, and database relationships - Identify gaps and inefficiencies in current production reporting environment and provide recommendations to address them in the new platform.Design ingestion framework and CDC - tools used are Oracle Golden Gate and Azure Data FactoryPrepare design artifactsWork with IT partner on configuration of Golden Gate - responsible to provide direction and "how to ".Maintain dynamic pipeline for ETL ingestion to add new tables and data elementsData design - physical model mapping from data source to reporting destination.Understand the requirements. Recommend changes to the physical model to support ETL design.Reverse engineer and document existing SQL logic to improve design effortAssist with data modelling and updates of source-to-target mapping documentationDevelop scripts for the physical model, and update database and/or data lake structure.Access Oracle DB, SQL Server, and Azure environments, using SSIS, SQLDeveloper, Azure Data Studio, Azure Data Factory, Databricks and other tools to develop solution.Proactively communicate with business and IT experts on any changes required to conceptual, logical and physical models, communicate and review timelines, dependencies, and risks.Development of ETL strategy and solution for different sets of data modulesUnderstand the Tables and Relationships in the data model.Create low level design documents and test cases for ETL development.Senior ETL Developer will create the workflows and pipeline designDevelopment and testing of data pipelines with Incremental and Full Load.Develop high quality ETL mappings/scripts/notebooksDevelop and maintain pipeline from Oracle data source to Azure Data Lake and Databricks Sql WarehouseDevelop ETL to update datamarts built in Databricks Sql WarehousePerform unit testingEnsure performance monitoring and improvementPerformance review, data consistency checksTroubleshoot performance issues, ETL issues, log activity for each pipeline and transformation.Senior ETL Developer will review and optimize overall ETL performance.End-to-end integrated testing for Full Load and Incremental LoadPlan for Go Live, Production Deployment.Create production deployment steps.Configure parameters, scripts for go live. Test and review the instructions.Create release documents and help build and deploy code across servers.Go Live Support and Review after Go Live.Review existing ETL process, tools and provide recommendation on improving performance and reduce ETL timelines.Review infrastructure and remediate issues for overall process improvementKnowledge Transfer to Ministry staff, development of documentation on the work completed.Document work and share the ETL end-to-end design, troubleshooting steps, configuration and scripts review.Transfer documents, scripts and review of documents to Ministry.Must Have:7+ years using ETL tools such as Microsoft SSIS, stored procedures, T-SQL (Must Have)2+ Azure Data Lake and Databricks, and building Azure Data Factory and Azure Databricks pipelines (Must Have)2+ years Python and PySpark (Must Have)Ideal Candidate should have:Oracle Golden GateSQL ServerOracleAbility to present technical requirements to the businessEvaluation Criteria:Design Documentation and Analysis SkillsDemonstrated experience in creating both Functional Design Documents (FDD) & Detailed Design Documents (DDD).Experience in Fit-Gap analysis, system use case reviews, requirements reviews, coding exercises and reviews.Experience in the development and maintaining a plan to address contract deliverables, through the identification of significant milestones and expected results with weekly status reporting.Work with the Client & Developer(s) assigned to refine/confirm Business RequirementsParticipate in defect fixing, testing support and development activities for ETL and reportingAnalyze and document solution complexity and interdependencies by function including providing support for data validation.Development, Database and ETL experienceDemonstrated experience in database and ETL development (7+ years)Experience in developing in an agile Azure DevOps environmentExperience in application mapping to populate data warehouse and dimensional data mart schemasDemonstrated experience in Extract, Transform & Load software development (7+ years)Experience in providing ongoing support on Azure pipeline/configuration and SqlServer SSIS developmentExperience building data ingestion and change data capture using Golden GateAssist in the development of the pre-defined and ad-hoc reports and meet the coding and accessibility requirements.Demonstrated experience with Oracle and SqlServer databasesProficient in SQL and PythonImplementing logical and physical data modelsKnowledge TransferThe Developer must have previous work experience in conducting Knowledge Transfer and training sessions, ensuring the resources will receive the required knowledge to support the system. The resource must develop learning activities using review-watch-do methodology & demonstrate the ability to prepare and present.Development of documentation and materials as part of a review and knowledge transfer to other membersDevelopment and facilitation of classroom based, or virtual instructor led demo sessions for DevelopersMonitor identified milestones and submission of status reports to ensure Knowledge Transfer is fully completedNote:Hybrid role - 3-days per week on siteBenefits:Medical, Vision, and Dental Insurance Plans401k Retirement Fund#gttca#LI-GTT