Automating Risk Model Retrain Loop with Agentic Skills

Coinbase has detailed how its risk machine-learning team automated a previously labor-intensive model retraining process using internal "agentic skills" — reusable workflows powered by coding agents. The company said the change removes days of manual engineering work and introduces formal human checkpoints between each automated stage.

The firm's fraud-detection models evaluate activity in real time and apply friction such as step-up authentication or manual review to high-risk events. Because attackers continuously change their tactics, models must be retrained on the latest signals. Historically, a single retraining cycle took engineers several days to extract data, analyze signals and manage training jobs.

Three-Step Automated Loop

The revamped process is split into three stages, each supported by a dedicated skill:

• Emerging Trend Analysis — Identifies confirmed fraud incidents where the model did not apply downstream friction, making them valuable signals for improvement. The skill runs a discovery query, starts a Databricks notebook for deeper feature analysis, and produces a summary document.
• Feature Enrichment — Compares the current model's feature set against the company's feature store to find signals not yet used, measures their usefulness for the emerging trend cohort, and returns prioritized recommendations grouped into Tiers 1 through 4.
• Model Retraining — Launches baseline training with existing hyperparameters, polls job status, retrieves logs and metrics, designs and submits roughly 24 parallel hyperparameter sweep tasks, and aggregates results into a model training tracker such as a CSV.

Each stage produces a concrete artifact for human review before the next phase begins. The agent cannot independently deploy models to production, and experts can redirect, adjust inputs or stop the workflow at each gate.

The automation is triggered with a single command, e.g. /emerging-trend-analysis start_date=... end_date=..., and runs end-to-end without manual oversight for the training phase.

Lessons Learned

The Coinbase engineers noted several key takeaways from building the skills. They found it pays to stay engaged when an agent hits technical limits, guiding it through difficult parts while still in the loop rather than taking over manually — an upfront "torture test" that becomes cheaper in future cycles.

They also split workflows at human verification thresholds because the judgment required at those boundaries is complex and high-impact. Adding log-reading capabilities for training jobs was critical, letting agents manage long-running tasks autonomously while preserving observability.

The company says the acceleration strengthens its security posture and supports its goal of being the most trusted cryptocurrency exchange.