Fine-Tuning LLMs with 7B or more parameters require substantial hardware resources. One option is to build and on-premise computer with powerful and costly GPUs. The other option is to use cloud environments, including free services, like Collab and Kaggle, and paid services, like Replicate and Paperspace. These environments offer Jupyter notebooks in which you can run your LLM fine-tuning code. However, these environments have constraints and limitations that need to be considered, such as the maximum amount of time that a notebook can run.
This article contains eight tricks when working with such cloud environments. You will learn how to inspect the cloud environment, define workloads to run on CPU or GPU, how to save and export training results as well as preventing sessions timeouts.
This article originally appeared at my blog :
# Transformers installation
!pip install -U transformers==4.30 tensorflow==2.15
!pip install accelerate==0.27.2 peft==0.10.0 bitsandbytes==0.43.0 trl==0.8.1 datasets==2.1.0
!pip install einops==0.7.0 fsspec==2024.2.0
Binary Version Pinning and Execution
Some projects require you to use a specific version of an installed binary, such as Python.
Running an internet search reveals a plethora of methods, dating back several years into the past, and include using Linux install commands, pipx, pyenv and conda.
In environments where conda is available, you can install a specific Python version as shown:
!conda create -n py3.8 -y \
&& source /opt/conda/bin/activate py3.8 \
&& conda install python=3.8 -y \
&& python --version
When using this specific binary, you need to consider that each command in a Jupyter notebook s essential a one-off command. Therefore, you need to prepend all commands with the desired binary, and chain the commands together, like this:
!source /opt/conda/bin/activate py3.8 \
&& python --version \
&& cd llm-evaluation \
&& pip install -r requirements.txt \
Prevent Data Logging to External Providers
Some cloud environments automatically enable external telemetry data to be captured and send.
On Kaggle, the wandb library is installed, which is invoked during training automatically. If you do not need it, you can uninstall it with this command:
!pip uninstall wandb -y
Alternatively, you can set an environment variable.
import os
os.environ["WANDB_MODE"] = "offline"
When using HuggingFace trainer library, disable all telemetry with this:
args = TrainingArguments(
...
report_to=None,
)
Execution
Periodically Save Training Artifacts
Some cloud environments do not guarantee a default runtime duration. Therefore, you should save training results automatically & periodically.
With the HuggingFace Trainer library, use this:
training_args = TrainingArguments(
output_dir="./llama-7b-qlora-instruct",
save_steps=1,
)
With Tensorflow, you need to create a Checkpoint and CheckpoinManager object, and pass them to the trainer.
# source: https://www.tensorflow.org/guide/checkpoint
ckpt = tf.train.Checkpoint(step=tf.Variable(1), optimizer=opt, net=net, iterator=iterator)
manager = tf.train.CheckpointManager(ckpt, './tf_ckpts', max_to_keep=3)
def train_and_checkpoint(net, manager):
#...
for _ in range(50):
example = next(iterator)
loss = train_step(net, example, opt)
ckpt.step.assign_add(1)
save_path = manager.save()
Manually Export Training Artifacts
Output data resides in the virtual machine instance of the cloud provider. To get this data out, you have several cloud-provide specific and agonistic solutions. A list ordered by "most-generic" to "very specific":
Download via GUI
Some environments offer and option to download files from a dedicated directory path. First, create a zip file via a bash command, e.g. !zip -r file.zip "/kaggle/working/llama-7b-qlora-instruct/checkpoint-80". Second, download this zip file.
In Collab, you can access the file explorer via the GUI. Or you can trigger a Download dialog to open by executing his snippet:
from google.colab import files
files.download(zipfile_name)
In Kaggle, you can also use the GUI, or open a clickable link with this code:
from IPython.display import FileLink, display
display(FileLink(zipfile_name)
Upload to Cloud Storage
Another option is to upload the results to a cloud storage repository. Thereby, it is crucial that you trust the environment with providing required access credentials.
For accessing Google Storage, use the following snippet. It creates an inline-tile that starts an interactive login and then mounts the drive at the specified mount point.
from google.colab import drive
drive.mount('/content/gdrive')
For accessing Amazon cloud storage, use the boto3 library:
s3_client = boto3.client('s3')
response = s3_client.upload_file(file_name, bucket, object_name)
Prevent Session Timeout
Most cloud environments have an Idle timeout, which means that after a certain period where you do not engage with the site, the environment will be stopped, and your results lost. The key is to implement browser interactivity with a script. Open the browser console, then run the following script:
document.body.addEventListener('click', () => {
console.log("click");
});
const click = () => {
const simulate = new MouseEvent('click', {
view: window,
bubbles: true,
cancelable: true,
clientX: 100,
});
document.body.dispatchEvent(simulate);
}
const sleep = (delay) => new Promise((resolve) => setTimeout(resolve, delay))
const repeatedClick = async () => {
while (true) {
click();
await sleep(60000);
}
}
repeatedClick();
This will keep the session active even when you un-focus the browser window.
Conclusion
When fine-tuning or evaluating LLMs in cloud environments, several restrictions apply. This blog post includes a set of tricks and best-practices to make these environments work more robust for your projects. You learned how to inspect the hardware, libraries and binaries, then how to apply strict version pinning, and finally how to periodically and automatically save results and prevent a sessions timeout.
SOCIAL SHARE CARD GENERATOR