Data Engineer
Job details
About the job
Company Profile Minfy is an award-winning cloud native Systems Integrator with specialised capabilities in end to end digital transformation. Minfy offers full bouquet of consulting and delivery services working backwards from business imperatives. Minfy has exclusive digital maturity frameworks to translate business goals into Intelligent application/platform engineering, unlocking data value for organisation competitiveness and leveraging the power of cloud infrastructure. Mission
Minfy’s core mission is to augment human capacity in customer organizations. It enables its customers to become future ready. Minfy not only helps customers visualize, but actualize their future. It recognizes that each customer and their context is unique, and the journey must be tailormade. Minfy highlights:
Minfy has won AWS India Partner of year awards three times.
Recognised as Fastest growing companies by the Financial times
Minfy has operations in the US, Australia, South-East Asia and India
Key practices include: AI/ML, Data & Analytics, Intelligent Application Engineering, Cloud Migration and Modernisation and Site Reliability Engineering
Industry Focus: Logistics, Manufacturing, HCLS, FSI, Aerospace & Satellite
Leadership team from Global Majors – AWS, Microsoft, Deloitte, Capgemini, Infosys, PWC Experience eligibility and Location
Experience Level: 5 to 12 Years
Work location: Remote Job Description
Role & Responsibilities:
Data Architecture: Design and implement data architecture solutions on AWS, including data lakes, data warehouses, and streaming data pipelines.
ETL Development: Develop and maintain ETL (Extract, Transform, Load) processes to ingest, process, and transform data from various sources into a usable format for analytics and reporting.
Data Modelling: Create and maintain data models that support analytical and reporting requirements. Optimize data structures for performance and scalability.
AWS Services : Leverage a wide range of AWS services such as S3, Redshift, Glue, EMR, Athena, Kinesis, Lambda, and more to build data solutions.
Data Quality : Implement data quality checks and monitoring to ensure data accuracy, completeness, and reliability.
Performance Optimization : Continuously optimize data pipelines and infrastructure for improved performance, cost efficiency, and scalability.
Security and Compliance : Implement data security and compliance best practices, including data encryption, access controls, and auditing.
Documentation : Maintain comprehensive documentation of data engineering processes, configurations, and workflows.
Collaboration : Work closely with data scientists, analysts, and other stakeholders to understand their data requirements and provide data engineering support.
Troubleshooting : Diagnose and resolve data engineering-related issues in a timely manner. Skills Required:
Proven experience as a Data Engineer with a strong focus on AWS cloud platform.
In-depth knowledge of AWS services and tools for data engineering and analytics.
Proficiency in programming languages such as Python, Java, or Scala.
Strong SQL skills and experience with relational databases and data warehousing concepts.
Experience with big data technologies such as Hadoop, Spark, and Kafka.
Understanding of data governance, data quality, and data privacy.
Excellent problem-solving and troubleshooting skills.
Strong communication and collaboration skills.
AWS certifications (e.g., AWS Certified Data Analytics, AWS Certified Data Warehousing) are a plus.
5+ years of experience in data engineering / Data integrations.
Formative years may be in development projects in data warehouses, datamarts, ETL, data modelling, data rich middlewares, and distributed computing solutions.
3+ years in Cloud services with at least 1 implementation/development project
Last 3 years need to be in big data engineering – Cloud Integrations, Serverless, S3/BLOB, Hadoop, HBase, Hive, Spark, Spark streaming, Kafka in-memory database systems, Databases (NoSQL and SQL)
Should have programmed extensively in python, Pyspark, scala, java or .net.
Should have managed an industry standard program in cloud building data pipelines, data migrations or analytics pipelines.
Good to have exposure to Apache airflow.
Should have worked as software solution provider at the level of solution architect, technology evangelist in medium to large ISV.
Direct experience of at least 3 years in design and delivery of bigdata solutions in healthcare vertical. Good to have:
Experience with serverless computing using AWS Lambda.
Familiarity with containerization and orchestration using Docker and Kubernetes.
Knowledge of DevOps practices and CI/CD pipelines.
Experience with data orchestration tools like Apache Airflow.
Apply safely
To stay safe in your job search, information on common scams and to get free expert advice, we recommend that you visit SAFERjobs, a non-profit, joint industry and law enforcement organization working to combat job scams.
