Can Lora be combined with Prompt Tuning in a 1-2 punch design? (sort of)
Absolutely, you can do LoRA first and then prompt tuning to help with the context! But to be frank I am not sure if it has any effect because you already have specialised the LLM to a certain task with LoRA.