Stacking fine-tuning

I can see myself wanting to optimise the linguistic reasoning inside of an LLM using LoRA with extra data that it hadn’t seen in pretraining.
But what happens if would then want to tweak the HHH of the outputs using RLHF and again LoRA?
Can I stack LoRAs at inference time or merge the first LoRA weights into the base weights (LLM*LoRA1) and then run the RLHF LoRA on this updated LLM?

As far as I understand the RLHF is run during training with LORA. Once training has finished you can use LLM + LORA weights, but those all are frozen now, and inference can happen with these frozen weights.

Hi, thanks for the quick response.
RLHF + LoRA makes sense to me, but what I meant is that in Week2 we learned that LoRA can be used for normal PEFT, for example if we want a pretrained model to learn new insights based on our internal input-output training data.
What do we do if we want to:

  • use FLAN-T5
  • fine tune it for our domain with PEFT LoRA
  • and then we want to fine tune the outputs to reduce toxicity - again with PEFT LoRA