Fine-tuning ChatGPT

Image generated by DALL-E 2

I’ve been spending some time trying to fine-tune ChatGPT’s answers, followed a few articles like this one by Amogh Agastya (which is really well explained) and the official docs. My intention is to sum up the steps to achieve it, and explain what could happen in each of them.

Before getting into the process, lets define a few things, starting with fine-tuning:

“Fine-tuning is the process of adapting a pre-trained model to a new task or dataset, by training it on a smaller dataset. This allows the model to learn the characteristics of the new task or dataset, and improve its performance on that task.”

Yes, I quoted the previous definitions because that’s what ChatGPT has to say about fine-tuning 😲. Fine-tuning is a really good solution when working with very specific tasks (such as chatting about

There are two main types of fine-tuning:

• Few-shot fine-tuning: This type of fine-tuning is used when the new task or dataset has a small number of examples, usually in the order of a few hundred or less.
• Zero-shot fine-tuning: This type of fine-tuning is used when the new task or dataset has no or very few examples. In this case, the model is not trained on the new task or dataset, but instead, the pre-trained weights of the model are used directly to make predictions on the new task or dataset.

After those clarifications we can get right to business. First, we need to get our hands on a dataset which has at least a couple hundred examples (as we’ve learned, that’s for the few-shot fine-tuning to be accurate) according to the official docs. This is not as easy to do, most of all since it has to be properly formatted. Ideally, the file would be a JSONL file with the following format:

{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
{"prompt": "<prompt text>", "completion": "<ideal generated text>"}
...

Luckily, OpenAI provides a tool that takes several formats and returns a JSONL with the right structure. The only caveat here is that the file (CSV, TSV, XLSX, JSON or JSONL) needs to have a prompt column and a completion column. It’s not an issue if the file has other columns, but those two need to be there.

To be able to use OpenAI toolset, we first need to create an API_KEY associated to our account, which is pretty straight forward. Just go to your account and click on the create new secret key button. Make sure you copy the key as you won’t be able to retrieve it later.

Now lets move to a brand new terminal, and run the following commands:

1. Install the openai cli → pip3 install --upgrade openai
2. Expose your API_KEY → export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
3. Run the file through the OpenAI formatter tool → openai tools fine_tunes.prepare_data -f <LOCAL_FILE> , where <LOCAL_FILE> is the path to your CSV, TSV, XLSX, JSON or JSONL file. The cli will either abort the operation if any of the previously mentioned conditions are not fulfilled, or just throw warnings indicating that things can be changed in the process. Here are a few examples of those:

• Some of the additional columns/keys contain `COLUMN_NAME` in their name. These will be ignored, and the column/key `COLUMN_NAME` will be used instead. This could also result from a duplicate column/key in the provided file.
• Your data does not contain a common separator at the end of your prompts. Having a separator string appended to the end of the prompt makes it clearer to the fine-tuned model where the completion should begin. See https://beta.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more detail and examples. If you intend to do open-ended generation, then you should leave the prompts empty
• Your data does not contain a common ending at the end of your completions. Having a common ending string appended to the end of the completion makes it clearer to the fine-tuned model where the completion should end. See https://beta.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more detail and examples.

And I could go on with those, but basically you just need to press ‘y’ and follow along.

— NOTE: when answering ‘y’ to — Add a suffix separator ` ->` to all prompts [Y/n] — will require you to provide -> at the end of every prompt to let ChatGPT know your prompt is done. —

The result of running the steps mentioned above, we end up with a file YOUR_FILE_NAME_prepared.jsonl which is the output for running the command.

The second step is to have the CLI (there are other options, but this one is the most straight-forward of all) create our fine-tuned model.

The way to do it is basically by running

openai api fine_tunes.create -t <YOUR_FILE_NAME_prepared.jsonl> -m <BASE_MODEL> .

BASE_MODEL can be one of the four models OpenAI provides (Ada, Babbage, Curie or Davinci).

— NOTE: Creating the fine-tune model has a cost. I’ve ran it with a ≃ 100 prompt file and it was about $4.00, which were deducted from the free $18.00 you’re given to test the API. —

This step can take a few minutes/hours depending on the length of the system queue and the dataset provided.

If by any reason the stream is interrupted, you can run

openai api fine_tunes.follow -i YOUR_MODEL_ID

As a cool fact, OpenAI lets you change the name of your model, which at first looks something like ft-AbCdefgHijk85lmnopqrSTuVwxyZ0993 (not so nice to look at 😆) by running openai api fine_tunes.create -t <YOUR_FILE_NAME_prepared.jsonl -m <BASE_MODEL> —suffix <YOUR_COOL_MODEL_NAME>

Once it’s done, the CLI will prompt you with the following:

Job complete! Status: succeeded 🎉
Try out your fine-tuned model:

openai api completions.create -m <BASE_MODEL> -p <YOUR_PROMPT> #Remember here to add -> at the end of every prompt

And that’s it!!

Basically what’s left to do is either try it out in the OpenAI playground and select your fine-tuned model (explained in the image below) to prompt the chat the specific questions you expect to have answered like provided, or just hit the API from your app.

OpenAI Chat playground

Summary

1. Create an OpenAI account
2. Generate your API_KEY (and copy it right away!)
3. Set up your env → pip3 install --upgrade openai export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
4. Prepare the dataset (any of the following formats CSV, TSV, XLSX, JSON or JSONL) with a prompt and completion columns
5. Run the file through the OpenAI formatter tool → openai tools fine_tunes.prepare_data -f <LOCAL_FILE>
6. Create the fine-tuned model openai api fine_tunes.create -t <YOUR_FILE_NAME_prepared.jsonl> -m <BASE_MODEL> . If you want a specific name, add the --suffix <YOUR_COOL_SUFFIX> . This can take a few minutes/hours depending on demand and dataset size.
7. Follow the creation stream (also useful if it gets interrupted somehow)openai api fine_tunes.follow -i YOUR_MODEL_ID
8. Test the model in the playground or start querying it openai api completions.create -m <YOUR_MODEL_NAME> -p <YOUR_PROMPT> #Remember to add -> at the end of every prompt

Hope this was helpful! Let me know your thoughts on the comments.

— Fede