🔍 Read the full analysis: How To Understand Fine-Tuning Time Beyond FLOPs on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face reports that a CatBoost model predicted the wall-clock duration of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. Its best tested FLOP-based estimate had a 12.41-hour error, but the comparison covers one production environment and does not show that using the model saved GPU time or reduced delays.
Hugging Face reports that a CatBoost model predicted the end-to-end duration of 272 supervised fine-tuning jobs with a mean absolute error of 1.71 hours in five-fold cross-validation. The result was substantially more accurate than the best tested FLOP-based estimate, but comes from one shared production GPU pool and has not yet been tested in a live scheduling trial.
The jobs ran on shared NVIDIA H200 GPUs and accounted for about 7,382 GPU-hours. Their durations ranged from 1.6 to 172.4 hours. The dataset included model families up to 15 billion parameters, LoRA and full fine-tuning, varied dataset sizes, and jobs assigned one, two or four GPUs. The pool contained 8 to 32 H200s across one to four nodes, with other training and inference workloads running at the same time.
The target was wall-clock time from GPU allocation to job completion; queue time was excluded. Using configuration and model information available before dispatch, the predictor included details such as evaluation cadence, checkpoint settings, per-GPU token volumes and fine-tuning regime. It also used a FLOP-based duration estimate as one of its inputs. The study says the selected CatBoost model used 24 features.
In stratified five-fold cross-validation, CatBoost recorded a 1.71-hour mean absolute error, a 0.51-hour median error and a 3.52-hour root mean squared error. Its reported R² was 0.98. The FLOP-based estimate performed best at a tested 30% Model FLOPs Utilization (MFU), with a 12.41-hour mean absolute error and R² of 0.31. At 45% and 60% MFU, the baseline’s error was 13.01 and 14.62 hours, respectively. Hugging Face says testing MFU settings from 25% to 65% did not resolve the baseline’s shortcomings.
Why Fine-Tuning Takes Longer Than FLOPs Say
Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.
What the estimates capture
The target is time from GPU allocation to job completion; queue time is excluded.
The training loop
FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.
The whole job
CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.
Other runtime costs
Setup, data handling, network and storage can add time alongside training.
Prediction error in the tested data
Lower error means predicted duration was closer to observed duration on average.
What this could mean for GPU scheduling
More useful duration estimates could help teams plan scarce GPU capacity.
- GPUs may sit idle until another job starts.
- Operators can misjudge when capacity will be free.
- Other work can be delayed.
- Delivery dates may shift.
How far the result reaches
The preview reports cross-validation on one corporate production environment.
Runs used shared NVIDIA H200 GPUs and varied configurations.
Accuracy on other clusters, software stacks and workload mixes is not established.
The preview reports prediction error, not savings in GPU time or scheduling delays.
Better Estimates for GPU Scheduling
Runtime estimates influence when operators dispatch queued work and how they allocate scarce accelerators. If a job finishes earlier than expected, GPUs may sit idle until another job starts; if it runs longer, it can delay other work and shift delivery schedules. A more accurate prediction could help teams plan allocations and reduce these mismatches.
The reported error gap is large within this dataset, but it measures prediction accuracy, not operational savings. The preview does not report a controlled deployment showing that the model reduced idle GPU intervals, costs or job delays. Whether more accurate estimates produce those benefits depends on how a scheduler uses them and whether the predictions remain accurate on its workloads.
Products Worth Considering

Logitech MX Creative Console with 9 Customizable Keys, Graphite

Logitech MX Master 4 Ergonomic Wireless Mouse with Haptics – Graphite

Jackery Explorer 300 Portable Power Station, 292Wh
As an affiliate, we earn on qualifying purchases.
What FLOP Estimates Leave Out
A common way to estimate dense Transformer training compute uses parameter count and training-token count, then divides by GPU capacity and an assumed Model FLOPs Utilization. MFU represents the share of theoretical peak throughput achieved in practice. Hugging Face says its comparison tested a range of MFU assumptions rather than relying on just one.
That type of calculation estimates training compute, while this study measures a broader job lifecycle. Setup, model downloads and loading, tokenization, scheduled evaluations, checkpoint uploads and final storage can add wall-clock time. The preview says these overheads vary with job configuration and may compete for network and storage capacity on a multi-tenant cluster. That helps explain why configuration metadata could be useful, but does not by itself establish that the model will generalize elsewhere.
“Everything is known before dispatch.”
— Hugging Face, in the study preview
How Far the Results Travel
The reported scores come from cross-validation on 272 runs in one corporate production environment. The preview does not establish accuracy on other GPU types, clusters, software stacks or workload mixes, and cross-validation within this dataset is not the same as testing on independently collected sites or a later, unseen period.
The full paper has not yet appeared, and the preview gives no date for its release. It also does not report a live scheduling trial or enough information to determine how often the predictor might need retraining as configurations and shared-cluster conditions change. The amount of GPU time, money or delay that a scheduler might save remains unreported.
Full Paper and External Tests
Hugging Face says a full version will be posted on arXiv, but has not provided a publication date. The paper may offer additional detail about the dataset, feature selection and evaluation design.
For wider scheduling use, the key tests would be validation on workloads from different periods or environments, followed by a trial inside an operational scheduler. Those results could show whether the reported accuracy holds beyond the original pool and whether better forecasts translate into fewer delays or idle GPU intervals.
Key Questions
What did Hugging Face’s model predict?
It predicted end-to-end runtime for supervised fine-tuning jobs, measured from when a job received GPUs until it finished. Queue time was not included.
How did it compare with the FLOP-based estimate?
In five-fold cross-validation, CatBoost had a 1.71-hour mean absolute error. The best tested FLOP-based setting, at 30% MFU, had a 12.41-hour error. These figures apply to the study’s dataset.
Does the result prove the model will work on other clusters?
No. The preview reports results from one production environment. Performance on other GPU types, software stacks and workloads has not been established.
Did the model reduce GPU costs or delays?
The preview reports prediction errors, not the results of a live scheduling deployment. It does not quantify whether using the model reduced GPU costs, idle time or delays.
When will the full paper be available?
Hugging Face says it plans to post the full version on arXiv, but the preview gives no publication date.
Primary source: Hugging Face · via ThorstenMeyerAI.com
