Skip to content
Search

Qwen

lastmod 2026-03-09created 2025-02-028 edits

See also

Qwen3.5 fine-tuning guide

The Qwen3.5 model family (0.8B–122B) can be fine-tuned on text and vision with Unsloth, an open-source framework for LLM fine-tuning and reinforcement learning

  • Unsloth offers 1.5x faster training and 50% less VRAM than FlashAttention-2, enabling efficient training with a bf16 LoRA setup
  • The 0.8B, 2B, and 4B models can be tried for free via Colab notebooks, and notebooks for the 27B and 35B models are provided for A100 environments
  • MoE models (35B, 122B, etc.) get 12x faster training, 35% less VRAM, and 6x longer context length with the latest kernels
  • After training, models can be exported to various deployment formats such as GGUF, vLLM, Ollama, LM Studio, and SGLang

Qwen3.5 fine-tuning overview

  • The Qwen3.5 model family (0.8B, 2B, 4B, 9B, 27B, 35B‑A3B, 122B‑A10B) can be fine-tuned with Unsloth
    • Supports both text and vision
    • Qwen3.5‑35B‑A3B bf16 LoRA runs on 74GB VRAM
  • Unsloth provides 1.5x faster training and 50% less VRAM usage
    • VRAM usage: 0.8B (3GB), 2B (5GB), 4B (10GB), 9B (22GB), 27B (56GB)
  • The 0.8B, 2B, and 4B models can be tried with free Google Colab notebooks
  • To preserve reasoning ability, a dataset with at least 75% reasoning examples is recommended
  • Full Fine-Tuning (FFT) is also possible, but VRAM usage increases 4x

Training environment and setup

  • Qwen3.5 is a multilingual model supporting 201 languages
  • Reinforcement Learning (RL) and Vision RL (VLM RL) are also supported through Unsloth
  • A100 Colab notebooks provided: Qwen3.5‑27B, Qwen3.5‑35B‑A3B
  • For local training, update to the latest version
    • Command: pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo
  • transformers v5 required; older versions do not work
  • Initial training may be slow due to Mamba Triton kernel compilation (especially on T4 GPUs)
  • QLoRA (4-bit) training is not recommended

MoE model fine-tuning (35B, 122B)

  • Supports Qwen3.5‑35B‑A3B / 122B‑A10B / 397B‑A17B models
    • 12x faster training, 35% less VRAM, 6x longer context length
  • bf16 LoRA or Full Fine-Tuning recommended
  • MoE QLoRA 4-bit not recommended due to BitsandBytes limitations
  • Unsloth MoE kernels enabled by default; the backend can be switched with UNSLOTH_MOE_BACKEND
  • Router-layer fine-tuning is disabled by default for stability reasons
  • Qwen3.5‑122B‑A10B bf16 LoRA requires 256GB VRAM
    • For multiple GPUs, set device_map = "balanced" or see the multiGPU guide

Quickstart

  • Text-only SFT (supervised fine-tuning) example provided
  • Qwen3.5 has a Causal Language Model + Vision Encoder architecture
    • Vision dependencies (torchvision, pillow) must be installed
  • Using the latest Transformers version is recommended
  • GRPO training can be done with Unsloth inference after disabling fast vLLM
  • If OOM (out of memory) occurs
    • per_device_train_batch_size=1, reduce max_seq_length
    • Keep gradient_checkpointing="unsloth" to save VRAM and extend context
  • MoE bf16 LoRA loader example provided

Vision fine-tuning

  • Supports vision fine-tuning of multimodal Qwen3.5 models
    • Qwen3-VL GRPO/GSPO RL notebooks can be used (just change the model name)
  • Can choose vision-only or text-only training
    • Selective fine-tuning among Vision, Language, Attention, and MLP layers
    • All enabled by default
  • For multi-image training, see the separate multi-image vision guide

Saving and deploying models

  • Supports various deployment methods such as llama.cpp, vLLM, llama-server, Ollama, LM Studio, and SGLang

Saving to GGUF

  • Unsloth supports saving directly to GGUF format and uploading to Hugging Face
  • If performance degrades during inference, the main cause is usually a wrong chat template or EOS token

Saving for vLLM

  • vLLM 0.16.0 does not support Qwen3.5
    • Requires 0.170 or later, or a Nightly version
  • Can save 16-bit or LoRA adapters only
  • For details, see Unsloth's inference guide

Qwen3.5 local run guide

  • Qwen3.5 local run guide | GeekNews
  • Alibaba's Qwen3.5 model family comes in sizes from 0.8B to 397B, with multimodal hybrid reasoning and 256K context
  • Unsloth provides all Qwen3.5 models as Dynamic 2.0 GGUF quantizations, which can be run locally via llama.cpp or LM Studio
  • You can switch between thinking and non-thinking modes; small models (0.8B–9B) default to non-thinking mode
  • Required RAM/VRAM and recommended settings (temperature, top_p, etc.) are listed for each model, and the 27B and 35B models can run even on a 22GB Mac
  • Unsloth GGUF improves performance with an enhanced quantization algorithm and imatrix data, and is not compatible with Ollama

Favorite site

author Zer0 sitemap