Costs to finetune T5

I’m just going through the Scaling Instruction-Finetuned Language Models paper and I’m wondering if it is affordable for the average joe to finetune an LLM and get a model that’s good at zero shot general tasks.

On page 10 the authors state:
Flan-T5-XL is only 3B parameters and achieves a MMLU score of 52.4%, surpassing GPT-3 175B’s score of 43.9%. That’s awesome. GPT-3 was already impressive.

On page 4 the authors state:
For example, we only use 0.2% of the pre-training compute to instruction-finetune Flan-PaLM 540B…

On slide 144 of last weeks lecture notes I see that T5 3B took 100 Petaflop/s-days to pre-train.

On Runpod a regular 80GB NVIDIA A100 costs 1.79 dollars /hr (on demand)

Does that mean that
The cost of pretraining: 1.79 (runpod price) *24 * 2 * 100 (pre-train compute according to slides)
times the factor of finetuning: * 0.002
we could potentially spend as little as 17.18 dollars?

The paper states “approximately 512 v4 TPU chips for 37 hours”.

I believe that’s for PaLM 540B