In this paper, we present evidence that SLMs still have a quality advantage for certain domain tasks that require structured output, despite the emergence of large-scale language models (LLMs) such as GPT-4 making the benefits (faster inference, lower cost) of fine-tuning small-scale language models (SLMs) for real-world applications less clear. By comparing SLM fine-tuning with LLM prompting on a low-code JSON workflow generation task, we find that good prompting can produce reasonable results, but fine-tuning only improves quality by an average of 10%. In addition, we reveal the limitations of the model through systematic error analysis.
