I am doing lab for week 2 - Lab 2 - Fine-tune a generative AI model for dialogue summarization
At step 2.2 of lab during training model the python console is getting killed. I tried this many times and same error. I am using Kernal as mentioned in lab (Data science 3, ml.m5.2xlarge, python3)
This is output on console.
>>> trainer.train()
/opt/conda/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
warnings.warn(
0%| | 0/1 [00:00<?, ?it/s]Killed
sagemaker-user@studio$ ```
Inside the the TraininingArguments method put optim=“adamw_torch”, lets see if that overcomes the issue. Hi @chris.favila is there some update that maybe be needed to do some changes on the notebook!
Hi
Its still the console gets killed after using extra new param inside TrainingArguments method.
>>> training_args = TrainingArguments(
... output_dir=output_dir,
.. optim="adamw_torch",
... learning_rate=1e-5,
... num_train_epochs=1,
... weight_decay=0.01,
... logging_steps=1,
... max_steps=1
... )
>>>
>>> trainer = Trainer(
... model=original_model,
... args=training_args,
... train_dataset=tokenized_datasets['train'],
... eval_dataset=tokenized_datasets['validation']
... )
>>> trainer.train()
0%| | 0/1 [00:00<?, ?it/s]Killed
sagemaker-user@studio$
Hi Prashant. Unfortunately, I can’t reproduce the issue. Can you provide a screenshot of the failure message together with the instance type? Something like the picture below but with the training failed as you mentioned. Will forward it to our engineer for checking. Thanks!
