The Definitive Fine-Tuning Guide for 3D Generation Models
This guide releases fine‑tuning scripts and container images for adapting the TRELLIS.2 3‑D generation model to custom assets, using LoRA adapters via Hugging Face’s PEFT library. In experiments on a 188‑item footwear subset of the Google Scanned Objects (GSO) dataset, LoRA improves sole geometry that the pretrained model gets wrong. Engineers can run the provided Docker‑based workflow locally or on Hugging Face Jobs to fine‑tune only the structure and geometry components while freezing the rest of the pipeline.
⚡ Key Takeaways
- Fine‑tuning was evaluated on 188 footwear objects from the GSO dataset.
- LoRA adapters are inserted into the structure and detailed‑geometry stages of TRELLIS.2, leaving background removal, encoding, and material generation frozen.
- Using PEFT’s low‑rank adapters reduces trainable parameters and GPU memory compared with full‑model fine‑tuning.
- The workflow integrates BiRefNet for background removal and DINOv3‑L for image encoding, both required to be accessible via a HF_TOKEN.
- The pretrained DINOv3 model is gated; access must be granted to the Hugging Face account before training.
By targeting only the most task‑relevant components, engineers can quickly specialize a powerful 3‑D generator for niche domains (e.g., accurate furniture or footwear geometry) without the cost of full model retraining.
✅ Practical Steps
- Upload your mesh dataset to the Hugging Face Hub, then run the provided fine‑tuning script with your HF_TOKEN and the dataset identifier as arguments.
- Choose either the local Docker command or a Hugging Face Job to launch the containerized training, ensuring DINOv3‑L access is enabled.
- After training, replace the original TRELLIS.2 components with the saved LoRA adapter weights for inference.
Want the full story? Read the original article.
Read on Hugging Face Blog ↗