ML Ops-инженер

About Toloka

At Toloka AI we create data that powers leading GenAI models and innovations. We work with frontier labs, big tech, renowned AI startups, enterprises and non-profit research organizations worldwide. We use a combination of Experts + Crowd + Tech Platform to teach AI models to reason and evaluate their efficacy and safety. We have experts in more than 50 different domains—from doctors and lawyers to physicists and engineers—and boast one of the most diverse global crowds, representing over 100 countries and speaking 40+ languages. We are a well-funded startup with an enviable portfolio of clients including Anthropic, Amazon, Microsoft, Poolside, Recraft, and Shopify.

Recently, we secured strategic investment led by Bezos Expeditions and Nebius Group with participation from Mikhail Parakhin, CTO of Shopify and board advisor to leading GenAI companies, who now serves as our Chairman of the Board. Our remote-first team is globally distributed around the world: USA, UK, the Netherlands, Serbia, and more.

 

About the Team

We are the ML team inside Toloka — we build the machine-learning products that power the platform itself, so every project running on Toloka is faster, cheaper, and more reliable.

A few examples of what we own:

• LLM QA — the core technology behind Toloka's automated quality-check mechanism. Every annotation flowing through Self-Service is reviewed by an LLM agent we design, train, and operate.
• Model distillation and fine-tuning — adapting frontier and open-source models to Toloka's tasks to hit the right quality at the right cost.
• Evaluation, benchmarking, cost modeling, and model selection across providers.

We own the full chain. The same team designs the ML solution, ships it to production, keeps it running 24/7, analyzes the results coming back from real projects, and feeds that signal into the next iteration. No hand-off between research, engineering, and operations — it's all us.

 

About the Position

As a Senior ML Engineer, you will design, train, and deploy the AI agents that drive our core products. You will focus on end-to-end ML tasks—including fine-tuning and Reinforcement Learning (RL)—to build resilient agentic workflows in Python. You will own the full lifecycle of your models, closing the loop from research and benchmarking to production scaling and monitoring.

 

What you’ll do

• Train, fine-tune, and distill ML models (including RL approaches) to power autonomous AI agents.
• Build and operate agentic workflows in Python, handling complex reasoning and hybrid human-expert interactions.
• Own evaluation and benchmarking, selecting foundational models and establishing cost models.
• Manage the full ML lifecycle: design solutions, ship to production, and monitor real-time signals.
• Implement observability metrics tailored for agent logic, model performance, and system reliability.

 

What we're looking for

• 3+ years of experience in ML: Strong background in model training, fine-tuning, and Reinforcement Learning (RL);
• 1+ year in Agent Development: Practical experience building, evaluating, and launching autonomous AI agents;
• Agentic Frameworks: Proven experience working with frameworks such as LangChain, LlamaIndex, or AutoGen;
• Open-Source LLMs: Practical knowledge of model distillation and adapting open-source models (e.g., Llama, Mistral);
• Python Mastery: Advanced proficiency in Python with a drive to apply disciplined software engineering standards to ML;
• End-to-End Ownership: Ability to work across the entire chain, from research to production operations;
• Language: Fluency in English (B2 or above).

 

What we can offer

• You will be part of an international, dynamic environment that drives innovation and sets new standards in the AI and technology sector.
• Competitive compensation package including base salary, bonus, and ESOP.
• Paid PTO and benefits will vary depending on location.
• We offer a full remote or hybrid model (if you are based in NL or Serbia).
• IT setup and home office allowances.

 

Equal Opportunity Employer:

Toloka is committed to providing equal opportunity and fostering an inclusive environment. We welcome applications from all qualified individuals and do not discriminate on the basis of race, religion, color, national origin, sex, sexual orientation, gender identity, age, marital status, veteran status, disability, or any other characteristic protected by applicable law. Selection decisions are made based on qualifications, merit, and business need.

 

[Important Notice] Scam Alert Regarding Fake Job Postings

It has come to our attention that an individual or group is fraudulently impersonating Toloka to post fake jobs and solicit personal information from applicants. Please be aware:

• Official Communication: Our recruiting team will only contact you from an official "toloka.ai" email address. We will NEVER use Gmail, Yahoo, Tolokainc, toloka.inc, or other personal or seemingly business email accounts.
• Our Process: We will never ask for your bank account details, credit card number, or any fees as part of the application or interview process.
• Official Listings: All legitimate job openings are posted on our official careers page: https://toloka.ai/careers#job-list

What to do: If you see a suspicious job posting or have been contacted by someone you suspect is a scammer, please do not provide any personal information. Instead, report the incident to us directly at security@toloka.ai and report the profile/post to LinkedIn.We are taking this matter very seriously and are working with the appropriate parties to resolve it.

Thank you for your vigilance!

To learn how we collect, use, disclose, and store personal data, check out our Privacy Notice.

At JetBrains, code is our passion. Ever since we started, back in 2000, we’ve been striving to make the strongest, most effective developer tools on earth. Today, AI-powered assistance and agents are becoming a core part of how developers work in our IDEs.

We’re building multi-step coding agents that can understand large codebases, plan changes, call tools, and iterate with the user. As a Research Engineer in the Agentic Models team, you’ll be responsible for the models, training loops, and evaluation pipelines that power these agents.

You’ll work at the intersection of SFT and RL-style post-training, and product-driven evaluation, using our distributed GPU and MapReduce clusters to ship models into JetBrains products.

As part of our team, you will:

• Design, implement, and maintain SFT and RL post-training pipelines for multi-step coding agents.
• Train and adapt LLMs for agent workflows, including planning, tool use, and multi-step interactions inside JetBrains IDEs.
• Build and develop evaluation and simulation environments where coding agents can act, be measured, and compared on realistic developer tasks.
• Design evaluation frameworks and metrics for agent behavior, analyze traces and logs, and close the loop from evaluation back into training, data, and reward design.
• Analyze training and evaluation results to propose and implement improvements to model architectures, training recipes, and datasets.
• Work with large-scale infrastructure, including distributed training on GPU clusters and large MapReduce-style data processing for pre-training and fine-tuning datasets.
• Collaborate closely with research, product, and infrastructure teams to turn high-level product visions into concrete models, experiments, and shipped features. 

We’ll be happy to bring you on board if you have:

• Extensive hands-on experience training LLMs (pre-training, fine-tuning, or post-training) in a research or production setting.
• Deep expertise in modern deep learning frameworks such as PyTorch, and specialized LLM training stacks (e.g. Megatron, NeMo, verl, or similar).
• Strong theoretical and practical understanding of LLM fundamentals: architectures, tokenization, data pipelines, batching, mixed precision, distributed training, and debugging unstable runs.
• The ability to own projects end to end, starting from a high-level problem or product pain point and overseeing it through the design, experimentation, implementation, and iteration phases.
• A product-aware mindset – you care about how developers actually use agents and can translate product needs and failure modes into modeling and evaluation work.
• At least 3 years of Python experience writing clean, maintainable code in modern ML codebases.

Our ideal candidate would have experience with:

• ML orchestrators and workflow tools such as Kubeflow, Dagster, Airflow, ZenML, and/or job schedulers like Kubernetes or SLURM.
• Large-scale data and training pipelines, e.g. MapReduce-style clusters, multi-node GPU training, or workloads on the order of 1M+ CPU/GPU hours.
• Designing and maintaining evaluation pipelines for LLMs or agents, including metrics, dashboards, experiment tracking, and automated regression checks.
• AI agent development, such as tool-using agents, planners, or multi-step coding workflows, and familiarity with agentic frameworks or patterns.
• Experiment tracking and observability using tools like Weights & Biases, MLflow, Langfuse, or similar.
• Inference optimization and serving optimized models in production.

#LI-KP1

At JetBrains, code is our passion. Ever since we started back in 2000, we have been striving to make the world’s most robust and effective developer tools. By automating routine checks and corrections, our tools speed up production, freeing developers to grow, discover, and create.

JetBrains is evolving beyond standalone developer tools toward a unified, AI-native platform for software development.

AI is no longer just an assistant inside the editor – it is becoming an active participant in how software is planned, built, reviewed, and operated across teams and organizations. This shift introduces new challenges that cannot be solved at the level of individual tools alone: governance, security, cost control, observability, and coordinated work between humans and autonomous agents.

Our goal is to build a platform that enables companies to adopt AI in software development in a structured, scalable, and economically efficient manner without locking them into closed ecosystems. This platform will serve as the execution and governance layer for AI-driven development, deeply integrated with developer tools but designed to work across teams, products, and environments.

We are looking for an experienced ML leader who has created products with an ML backbone, weaving together research, technical excellence, and strong product focus.

We are seeking a professional who excels in three key areas: technology, product vision, and business operations. This role involves extensive cooperation with products across the company – both AI-native and just integrating AI.

In this role, you will: 

• Identify, prioritize, and champion high-impact AI projects that align with JetBrains' product strategy and deliver measurable business value.
• Lead a team of about 50 people across multiple AI initiatives, including hiring and developing team leads, ML engineers, and MLOps specialists.
• Set the technical direction for JetBrains’ ML roadmap,  translating cutting-edge AI advancements into practical product innovations.
• Ensure projects deliver on commitments and create measurable value.
• Set technical standards, architectural direction, and best practices for ML development.

We’ll be happy to have you on our team if you have: 

• The ability to balance long-term R&D with short-term product needs.
• A track record of shipping production ML systems that users actually use.
• Experience building business cases for technical projects.
• A deep understanding of modern ML infrastructure, LLMs, and AI tooling.
• Outstanding stakeholder management skills and are comfortable working with senior leaders.
• The ability to thrive in a fast-paced environment, effectively managing multiple projects simultaneously.
• Excellent management and communication skills.
• Attention to detail in everything you do.

We’d be especially thrilled if you have experience with: 

• Distributed training of multi-billion parameter models.
• CI systems, workflow automation, and experiment tracking.
• The Kotlin programming language.

#LI-KP1 #LI-HYBRID