Scaling and Maintaining ML Models in Production Without an MLOps Team
Rocket Money, the personal finance app formerly known as Truebill, helps users manage their finances by classifying transactions from linked bank accounts. A key challenge is detecting merchants and services from short, cryptic transaction strings. Initially, they used regex-based normalizers and decision tables, which worked for the first four years but became unsustainable as the user base and product scope grew.
The Journey Toward a New System
After attempts with traditional machine learning solutions, including a bag-of-words model with per-class architecture that suffered from maintenance issues, Rocket Money started fresh. They built labeling queues, validation datasets, and drift detection tools using Retool. After testing various model topologies, they settled on a BERT family model for text classification, handling over 4,000 classes.
Solving Domain Challenges by Partnering with Hugging Face
Domain-specific challenges include noise from merchants, payment processors, and user behavior changes. With a small ML team, they needed a production-ready model serving solution without building in-house MLOps. They evaluated a hand-rolled solution, AWS SageMaker, and Hugging Face's Inference API. Since Rocket Money uses GCP for data and Vertex AI for training, exporting to AWS was cumbersome. Hugging Face set up quickly and handled traffic within a week, leading them to proceed.
Integration, Evaluation, and the Final Selection
After a three-month evaluation, they gradually increased transaction volume to Hugging Face's hosted models, running load tests based on worst-case scenarios. The Inference API proved capable of handling their bursty load. Beyond technical fit, the partnership with Hugging Face was strong. A shared Slack channel enabled rapid issue resolution, and the Hugging Face team acted as invested partners.
Collaboration and Future Plans
"Overall, the experience of working hand in hand with Hugging Face on model deployment has been enriching for our team and has instilled in us the confidence to push for greater scale," said Nicolas Kuzak, Senior ML Engineer at Rocket Money. The collaboration continues as they scale their ML capabilities.