Embarrassingly Simple Self-Distillation Improves Code Generation

6 Steps

1. 1

   Select a Code-Generating LLM: Choose an LLM that excels at code generation, accessible via an API or local deployment. Ensure it supports configurable sampling parameters like temperature and truncation.

2. 2

   Define a Code Generation Task: Prepare a set of programming prompts or problems for your LLM. These should be representative of the code you want the model to generate and improve upon.

3. 3

   Generate Multiple Candidate Solutions: For each prompt, query the LLM multiple times (e.g., 5-10 times) to generate diverse candidate solutions. Crucially, set a higher `temperature` (e.g., 0.7-1.0) to encourage creative and varied outputs.

4. 4

   Apply Truncation and Filtering: During generation, or post-generation, apply truncation strategies (e.g., `top_p`, `top_k`) to focus on higher probability tokens while still maintaining diversity. Filter out clearly non-viable or syntactically incorrect solutions.

5. 5

   Select Best Solutions for Self-Distillation: Implement a simple evaluation metric (e.g., pass/fail on provided test cases, static analysis for correctness, or even manual review) to identify the 'best' performing generated solutions for each prompt. These become your self-generated 'teacher' examples.

6. 6

   Distill the Model (Optional but Recommended): Use the selected 'best' solutions and their corresponding prompts as a small, high-quality dataset to fine-tune or 'distill' your original LLM. This reinforces the desired generation patterns and completes the self-improvement loop.