Intermediate Machine Learning Engineer, AI Powered: AI Framework

Intermediate Machine Learning Engineer, AI Powered: AI Framework Location: Remote GitLab is an open core software company that develops the most comprehensive AI-powered DevSecOps Platform, used by more than 100,000 organizations. Our mission is to enable everyone to contribute to and co-create the software that powers our world. An overview of this role: Are you passionate about building robust frameworks to evaluate and ensure the reliability of AI models? As a Machine Learning Engineer on GitLab’s AI framework team, you’ll play a critical role in shaping the future of AI-powered features at GitLab. This is an exciting opportunity to work on impactful projects that directly influence the quality of GitLab’s AI capabilities. Responsibilities:

1. Design and implement technical evaluators for LLM assessment.
2. Contribute to evaluation infrastructure consolidation efforts.
3. Build scalable evaluation pipelines and frameworks.
4. Develop and manage datasets and evaluation metrics.
5. Collaborate with feature teams to integrate validation solutions.
6. Optimize performance across ML evaluation systems.
7. Support improvements to GitLab’s AI-powered tools through validation.
8. Ensure all solutions align with GitLab’s infrastructure and security protocols.

What You’ll Bring:

1. Proven experience designing and implementing LLM evaluation systems.
2. Strong understanding of ML model architectures, including public vs. private implementations.
3. Expertise in ML evaluation metrics and dataset management.
4. Demonstrated ability to build production-grade ML infrastructure.
5. Practical experience with Python-based ML frameworks and evaluation tools (e.g., Langsmith, ELI5).
6. Excellent problem-solving skills with an engineering mindset.
7. Ability to collaborate in an asynchronous, remote-first environment.
8. Familiarity with open-source development and contribution is a plus.

About the team: The AIF team ensures that AI models across GitLab are reliable and well-validated. We focus on building robust evaluation frameworks, consolidating tools, and streamlining processes to scale validation efforts across GitLab’s AI infrastructure. How GitLab will support you:

1. Benefits to support your health, finances, and well-being.
2. All remote, asynchronous work environment.
3. Flexible Paid Time Off.
4. Team Member Resource Groups.
5. Equity Compensation & Employee Stock Purchase Plan.
6. Growth and development budget.
7. Parental leave.
8. Home office support.

Please note that we welcome interest from candidates with varying levels of experience; many successful candidates do not meet every single requirement. If you’re excited about this role, please apply and allow our recruiters to assess your application. Company: GitLab Level of experience: Senior (5+ years of experience)#J-18808-Ljbffr