Compute and display cost with non-zero w and b
Do I need to initialize test_w and test_b in the main code? in either case i am getting following error:

Cost at test w and b (non-zeros): nan

Out[cell#]: <function main.compute_cost(X_train, y_train, test_w, test_b, *argv)>

No, you do not.

Only add your code where it says to. Do not modify anything else.