Stability AI launches StableVicuna, the first open-source RLHF chatbot

StableVicuna-13B is an open-source conversational AI model and the first large-scale open chatbot trained using reinforcement learning from human feedback (RLHF). Built as an aligned assistant, it handles everyday tasks like basic math, code generation, and grammar correction while matching human conversational preferences far better than raw instruction-tuned models.

What it does and how it was built

While previous open models stopped at supervised instruction fine-tuning due to the complexity of alignment, StableVicuna-13B adopts the classic three-stage RLHF pipeline on top of Vicuna-13B, which itself is an instruction-tuned LLaMA-13B checkpoint.

First, the team applied further supervised fine-tuning using a mix of three datasets: OpenAssistant Conversations (OASST1) with 161,443 messages across 35 languages, GPT4All with 437,605 prompts and responses, and Alpaca with 52,000 instruction examples.

Next, they trained a reward model using the open-source trlX library. The reward model was trained on major human preference corpora, including 7,213 preference pairs from OASST1, 160,800 helpfulness and harmlessness labels from Anthropic HH-RLHF, and 348,718 collective preferences from the Stanford Human Preferences (SHP) dataset across 18 subject domains.

Finally, the team ran Proximal Policy Optimization (PPO) through trlX to align the fine-tuned model against the reward model.

How to get it and run it

Due to Meta's original LLaMA licensing terms, the model cannot be distributed as standalone weights. Instead, Stability AI released StableVicuna-13B on Hugging Face Hub as a weight delta.

To run it locally, you need access to the original LLaMA-13B weights. You download the weight delta, run the Python merge script provided in the GitHub repository, and reconstruct the final model weights. If you just want to test how it chats, there is a live Gradio demo hosted on Hugging Face Spaces, with a dedicated web interface and a Discord bot planned for the Stable Foundation server.

Why it matters

StableVicuna is a major milestone for open-source AI because it demystifies the post-training alignment recipe that made ChatGPT so effective. It demonstrates that combining accessible tooling like trlX with public preference datasets can produce a functional RLHF pipeline outside proprietary labs.

For builders and researchers looking to align their own open-source models, StableVicuna provides an invaluable blueprint. Casual users might find the weight-merging requirement cumbersome, but trying the hosted demo gives a clear glimpse into where community-driven AI alignment is heading.