Prevent resource exhaustion in gradient descent

Course Q&A Machine Learning Specialization Supervised ML: Regression and Classification
1

This gradient descent form optional lab I would like to ask why he add

if i<100000:

it says to prevent resource exhaustion. If I deleted that condition that’s not effect. in both situations the i it will stop depends on what’s num_iters is

def gradient_descent(x, y, w_in, b_in, alpha, num_iters, cost_function, gradient_function): 
    """
    Performs gradient descent to fit w,b. Updates w,b by taking 
    num_iters gradient steps with learning rate alpha
    
    Args:
      x (ndarray (m,))  : Data, m examples 
      y (ndarray (m,))  : target values
      w_in,b_in (scalar): initial values of model parameters  
      alpha (float):     Learning rate
      num_iters (int):   number of iterations to run gradient descent
      cost_function:     function to call to produce cost
      gradient_function: function to call to produce gradient
      
    Returns:
      w (scalar): Updated value of parameter after running gradient descent
      b (scalar): Updated value of parameter after running gradient descent
      J_history (List): History of cost values
      p_history (list): History of parameters [w,b] 
      """
    
    # An array to store cost J and w's at each iteration primarily for graphing later
    J_history = []
    p_history = []
    b = b_in
    w = w_in
    
    for i in range(num_iters):
        # Calculate the gradient and update the parameters using gradient_function
        dj_dw, dj_db = gradient_function(x, y, w , b)     

        # Update Parameters using equation (3) above
        b = b - alpha * dj_db                            
        w = w - alpha * dj_dw                            

        # Save cost J at each iteration
        if i<100000:      # prevent resource exhaustion 
            J_history.append( cost_function(x, y, w , b))
            p_history.append([w,b])
        # Print cost every at intervals 10 times or as many iterations if < 10
        if i% math.ceil(num_iters/10) == 0:
            print(f"Iteration {i:4}: Cost {J_history[-1]:0.2e} ",
                  f"dj_dw: {dj_dw: 0.3e}, dj_db: {dj_db: 0.3e}  ",
                  f"w: {w: 0.3e}, b:{b: 0.5e}")
 
    return w, b, J_history, p_history #return w and J,w history for graphing
2

Hi @khaldon,

It does not control when the loop stops. If it had been designed to stop the loop earlier, it would have been in something like this

if i >= 100:
    break

Check out what it does in the if-block, it appends new values to two lists - J_history and p_history. The “resource” that it is trying to prevent exhausting was the memory consumed by the two lists - it limits the number of items stored by the lists. In other words, with the if-statement, we won’t store more than 100,000 values into each of the lists, otherwise, theoretically, it can keep storing until the memory in our system gets exhausted, if I unintentionally set num_iters to a very large number.

Check also that in this program, J_history and p_history are the two things that can grow in memory size, so we want limits on them.

Cheers,
Raymond

3

you mean that if I put num_iters > 100000 it will stop appending each of J_history and p_historybut if num_iters < 100000 such as 10000 or 50000it will append J_history and p_history corresponding to len(num_iters -1)

4

The way the program was written will stop appending new values after 100,000 iterations.

        if i<100000:      # prevent resource exhaustion 
            J_history.append( cost_function(x, y, w , b))
            p_history.append([w,b])

Ask yourself this: will the if statement be satisfied when i becomes larger than 100000? The answer is No. Since it is not satisfied, the append statements won’t be triggered.

Raymond