← Back
Hugging Face Blog

Why your fine-tuning job takes longer than the FLOPs say

•9 min read•
#deployment#compute
Why your fine-tuning job takes longer than the FLOPs say
Level:Advanced
For:AI Engineers
✦TL;DR

Predicting end‑to‑end fine‑tuning time with a simple FLOP‑based formula explains little of the observed variance (R² = 0.31, MAE = 12.41 h) on 272 production SFT jobs on NVIDIA H200 GPUs. A CatBoost regressor using only pre‑dispatch features achieves R² = 0.98, MAE = 1.71 h and median error = 0.51 h in 5‑fold cross‑validation by learning nonlinear interactions among evaluation cadence, checkpoint policy, per‑GPU load and fine‑tuning regime. The result shows that data‑driven models can replace back‑of‑the‑envelope FLOP estimates for more reliable GPU scheduling and reduced idle time.

⚡ Key Takeaways

  • FLOP‑based estimate R² = 0.31, MAE = 12.41 h on 272 jobs (7,382 GPU‑hours).
  • CatBoost regressor reaches R² = 0.98, MAE = 1.71 h, median error = 0.51 h (5‑fold CV).
  • Important features: evaluation cadence, checkpoint policy, per‑GPU load, fine‑tuning regime.
  • Overheads such as environment setup, model download, tokenization, evaluation, and checkpoint uploads dominate beyond raw compute.
  • MFU (Model FLOPs Utilization) typically 30 %–60 % and is unknown before dispatch, limiting closed‑form estimates.
💡 Why It Matters

Engineers can replace inaccurate FLOP calculations with a trained predictor to schedule fine‑tuning jobs more tightly, cutting idle GPU time and improving throughput on shared clusters.

✅ Practical Steps

  1. Use the described feature set (job configuration, model metadata, evaluation cadence, checkpoint policy, per‑GPU load, fine‑tuning regime) to train a CatBoost regressor for duration prediction before dispatch.
  2. Validate the predictor with 5‑fold cross‑validation to ensure R² ≈ 0.98 and low MAE.

Want the full story? Read the original article.

Read on Hugging Face Blog ↗

More like this

New in AWS Quick: More capable flows, an updated web experience, and agents that work together

AWS ML Blog•#agents

Three insights you may have missed from theCUBE’s coverage of the Dell Technologies New York Symposium

SiliconANGLE AI•#deployment

Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face Blog•#llm

NVIDIA and Microsoft Expand Accelerated Computing From Azure to Windows PCs

NVIDIA Blog•#compute

EXPLORE AI NEWS

Daily hand-picked stories on LLMs, RAG, agents and production AI — curated for engineers who ship.

BROWSE NEWS

GET THE WEEKLY DIGEST

Join engineers getting the Monday signal-over-noise AI breakdown. No spam, unsubscribe anytime.

LEARN AI ENGINEERING

Curated courses, research papers, repos and tutorials built for engineers leveling up in AI.

START LEARNING