Fine-tuning Gemma 3 4B for Text-to-SQL with QLoRA
Evaluated by running the SQL and comparing results — not by vibes.
The problem
Turn natural-language questions into SQL for a banking database, using a model small enough to fine-tune on a free Colab T4.
The build
QLoRA — a 4-bit quantized base with trainable LoRA adapters in the attention and MLP projections, so roughly 0.1–1% of parameters train and save to a tiny adapter file. Full SFT loop: dataset formatting, chat template, prompt masking, gradient checkpointing, training, evaluation, inference. The schema deliberately mirrors the Customer Spending & Risk Analytics project, so the model writes SQL for that exact database.
Architecture
— one teal highlight: the execution-accuracy checkWhat I found
Execution accuracy is the honest metric: run the generated SQL against the database and compare result sets, base model versus fine-tuned. String-matching the SQL rewards syntax mimicry; executing it rewards correctness.
The reps
Fitting a 4B model onto a free T4 forced every practical trick — quantization, checkpointing, masking the prompt from the loss — which is the actual engineering content of the project.
Results
- Gemma 3 4Bfine-tuned on a free T4
- QLoRA4-bit base + LoRA adapters
- Execution accuracybase vs fine-tuned, objective
What's verified
Every case study carries this blockWhat was measured, and what was not. Nothing in the second column is estimated into the first.
The Colab notebook runs top to bottom on a T4; execution-accuracy evaluation is part of the notebook.
No production serving path — this is a training and evaluation project.