Week 1 video

Course Q&A Machine Learning Specialization Advanced Learning Algorithms
1

image
image1741×1058 158 KB

In the video of Data in TensorFlow, Andrew mentioned that the last row x=np.array([200,17]) represent neither a row nor a column because it is a 1D vector/array. Then he said “although by convention we may write x as a column like this.” Not sure
what he meant by that, and there is no more annotation on the slide as he speaks.

2

In the real world, 1D sequence of numbers is neither a row nor a column vector. A row or column vector is in fact a two dimensional array (in which one of the two dimensions is 1.

Prof. Andrew Ng mentioned conventionality as far as I remember

To multiply a matrix by a vector which happens most of the time, the dimensions need to match up.

It usually makes more sense to use a column vector x for an expression like Ax. If x happens to be a row vector, you can take the transpose and write AxT, but that gets cumbersome. It is more of esthetic reason honestly while doing complex operations in linear algebra and but not a mandatory one.

1 Like
3

Three stances to be used as required:

I am working with “sequences of numbers”

When writing down the sequence, orientation is a matter of taste. Both

\left[1,2,3\right] or \left[ \begin{array}{c} 1 \\ 2 \\ 3 \end{array} \right]

are fine, one just sticks to either one or the other.

Dot/Scalar product can be computed as the sum of pairwise multiplication of the elements of two sequences that have the same length.

I am working with “matrixes”

Now we can have things like

\left[ \begin{array}{cc} 1 & 2 \\ 3 & 4 \end{array} \right]

“Orientation” of a sequence matters now (it is an additional attribute of the sequence) and

\left[1,2,3\right] is not the same as \left[ \begin{array}{c} 1 \\ 2 \\ 3 \end{array} \right]

Dot/Scalar product of two sequences of equal length is computed as the matrix multiplication of a “1-row matrix” on the left times a “1 column matrix” on the right:

\begin{bmatrix} 1 & 2 & 3 \end{bmatrix} \times \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} = 1^2 + 2^2 + 3^2 = 14

whereas commuting the operands of \times yields

\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \times \begin{bmatrix} 1 & 2 & 3 \end{bmatrix} = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 3 & 6 & 9 \end{bmatrix}

I am working with some computer implementation

For example, Numpy.

Now additional subtleties may arise in representation, both when writing code and printing. I once did this diagram:

notes on matrix and tensor
notes on matrix and tensor1300×920 122 KB

1 Like
4

Hi @dtonhofer,

I believe that the shape of 3-D tensor presented on the diagram should be (r, n, m):

>>> x = np.arange(2 * 3 * 5).reshape(2,3,5)
>>> x
array([[[ 0,  1,  2,  3,  4],
        [ 5,  6,  7,  8,  9],
        [10, 11, 12, 13, 14]],

       [[15, 16, 17, 18, 19],
        [20, 21, 22, 23, 24],
        [25, 26, 27, 28, 29]]])
>>> [s // 8 for s in x.strides]
[15, 5, 1]

Contiguous array is stored in memory such that the last axis (axis 2) changes fastest. This means that elements that differ only along the last axis are placed next to each other.

In NumPy, the .strides attribute provides the number of bytes you need to step through the array to move from one element to the next along a given axis. If the array’s data type is 8 bytes (e.g., 64-bit integers or floats), we can divide the strides by 8 to get the step size in terms of elements. This means that moving one step along axis 0 (the first dimension) requires jumping 15 elements in memory. Moving one step along axis 1 (the second dimension) requires jumping 5 elements. Moving one step along axis 2 (the third dimension) requires jumping 1 element. Therefore, the shape (r, n, m) corresponds to r blocks (slices), n rows per block, m columns per row.

BTW the plural for matrix is matrices :smiling_face:

1 Like
5

Hi @flyunicorn,

You may find the following thread helpful:

6

Hmm… I was thinking more in conceptual terms rather than memory layout (as done by the underlying BLAS library) … let me think about this :thinking: a revised diagram will be needed.

7

@dtonhofer
Even conceptually, as you can see from the structure of the data in the printed array, the shape (r, n, m) aligns well with how we’d naturally interpret the tensor: each of the r outer elements is a 2D matrix (or “slice”), each slice contains n rows, each row contains m columns.
For example, x[0] gives us the first 2D slice:

>>> x[0]
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])

This is a matrix with 3 rows and 5 columns, which clearly matching the (n, m) shape of each slice. So even without considering memory layout, the data is organized in a way that naturally corresponds to (r, n, m). I think that a revised diagram would definitely help to ground the intuition better.

1 Like