Vibe Fine-tuning LLMs

What I Wanted to Validate

I wanted to see if I could reproduce these findings—particularly the claim that rank=16 achieves 99% of rank=256's performance on supervised fine-tuning, and that rank=1 LoRA beats full fine-tuning on reinforcement learning tasks. But instead of spending weeks setting up infrastructure, I decided to use Orchestra and just have a conversation about what I wanted to test.

From Curiosity to Results: How Orchestra Made This Happen

Here's the thing that still feels a bit surreal: I didn't write a single line of training code. I didn't provision GPUs. I didn't debug CUDA errors at 2am. I just had a conversation with Orchestra and explained what I wanted to test:

The Conversations: Setting Up Two Experiments

I ran two separate experiments with two different Orchestra agents. For the supervised fine-tuning experiment, I asked:

For the reinforcement learning experiment, I started a new agent and said:

We went back and forth for about 20 minutes. I clarified which hyperparameters to use, what metrics to track, which baselines to compare against. The agent asked clarifying questions about dataset sizes, training duration, evaluation strategy. It felt more like collaborating with a research engineer than using a tool.

Conversation with Orchestra: defining experiments, clarifying hyperparameters, and discussing baselines

What the Orchestra Agent Did: The Full Workflow

1. Writing the Code

Orchestra generated complete training scripts for both experiments: SFT with LoRA at different ranks, and GRPO with LoRA vs full fine-tuning. It structured the code with proper experiment tracking, checkpointing, and evaluation loops. Set up configurations for Llama 3.2 1B on Tulu3 and Qwen2.5-0.5B on GSM8k.

2. Debugging and Testing

Before running full experiments, the agents ran test runs on GPU with small sample sizes and just a few training steps to catch bugs early. This caught issues in the GRPO reward function during testing—things that would have caused silent failures during full training runs.

3. Provisioning GPUs

Once tests passed, Orchestra automatically provisioned the necessary GPUs via Modal: 4x H100s for the SFT experiment, 1x H100 for the RL fine-tuning. It handled all the infrastructure setup—Docker containers, environment configuration, dependency installation. I didn't touch a single config file.

4. Running GPU Experiments in Parallel

Both experiments ran overnight in parallel. SFT with rank=16 and rank=256. GRPO with rank=1 LoRA vs two full FT baselines (low LR and high LR). All metrics were logged through Orchestra's internal SDK and rendered in real time.

5. Monitoring Progress

The agent continuously monitored training metrics. It detected when full FT with high LR flatlined at 0% correctness and flagged it. It noticed when LoRA hit 100% format compliance and predicted strong correctness would follow (it did).

Real-time monitoring of training metrics showing the agent detecting and flagging issues

6. Plotting Results

When experiments finished, Orchestra generated 11 publication-ready plots: training/eval loss curves for SFT, correctness over time for GRPO, format compliance trends, total reward progression.

The agent automatically generating publication-ready plots with side-by-side comparisons

7. Writing the Analysis Report

Finally, it generated a comprehensive markdown report with statistical analysis, key findings, comparison tables, and recommendations. It even offered to create a PowerPoint presentation.

The agent generating analysis reports and offering to create PowerPoint presentations

The Results: We Reproduced Their Findings

Experiment 1: Supervised Fine-Tuning (Rank=16 vs Rank=256)

Our experiment: Llama 3.2 1B on Tulu3 (rank=16 vs 256)

Original paper: Multiple ranks on Llama 3.1 8B & 3.2 1B

Our curves basically overlap with the original paper's findings. Rank=16 achieves 99.4% of rank=256's performance with 16x fewer trainable parameters—exactly validating what Thinking Machines Lab found.

Experiment 2: Reinforcement Learning (Rank=1 LoRA vs Full Fine-Tuning)

Our experiment: Qwen2.5-0.5B on GSM8k (rank=1 LoRA vs Full FT)

Original paper: LoRA ranks vs Full FT on RL tasks

Rank-1 LoRA absolutely demolished both full fine-tuning configurations—exactly matching what Thinking Machines Lab found. Full FT with high LR flatlined at 0%. Full FT with low LR peaked at 43.8% then degraded to 33.3%. Meanwhile, LoRA shot up to 56% by step 50 and stayed stable around 52%. Same story, different model and dataset.

Format compliance—LoRA hit 100% by step 100, while Full FT maxed at 82.3%

Total reward progression (correctness + format + reasoning quality)

The Timeline Comparison

Here is what I got: Production-ready experiments that ran to completion overnight. The code worked. The infrastructure provisioned correctly. The metrics tracked properly. The only "issue" was me changing my mind about reward weights halfway through—and the agent handled the rerun without complaint.

This eliminated weeks of engineering work that would normally block me from exploring areas outside my main expertise. I went from "I'd love to test this but don't have 3 weeks for setup" to "let me validate this today" in a single evening. That's the difference between ideas staying theoretical and actually testing them.

References

1. LoRA Without Regret — John Schulman and Thinking Machines Lab, Sep 2025
2. LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021
3. Group Relative Policy Optimization — Shao et al., 2024
4. GSM8K: Training Verifiers to Solve Math Word Problems — Cobbe et al., 2021
5. Tulu 3: Pushing Frontiers in Open Language Model Post-Training — Lambert et al., 2024

Citation

If you reference this work, please cite:

@article{zhang2025lora_reproduction,
  title={Reproducing "LoRA Without Regret" with Orchestra},
  author={Zhang, Zechen and Liu, Amber},
  journal={Orchestra Research Blog},
  year={2025},
  url={https://www.orchestra-research.com/perspectives/LLM-with-Orchestra}
}

Acknowledgements

We'd like to thank Modal for their generous support in providing cloud compute infrastructure for these experiments. Their platform made it seamless to provision H100 GPUs on-demand, which was essential for running both the supervised fine-tuning and reinforcement learning experiments at scale.