In the course Neural Networks and Deep Learning, Week 2, lesson Vetorizing Logictic Regression’s Gradient
the equation for cost:
is this equation referring to matrix dot product or matrix elemental multiplication?
The lab for the Programming Assignment is saying to use dot, but I am getting an error (shape not aligned) when using dot.
You actually have your choice here when computing the cost of whether to use dot product style matrix multiplication or elementwise multiplication followed by a sum.
The key point in either strategy is to realize the shape of the input data. You can put in some print statements to check that both A and Y are row vectors with shape 1 x m, where m is the number of samples.
So you can compute A * Y elementwise for the first term there and follow that with np.sum to add up the products. The advantage of using the dot product is that it’s less code and more efficient at runtime because it computes the multiplication and the sum at once. But then the trick is that you can’t “dot” 1 x m with 1 x m, because the “inner” dimensions must match on a dot product. You need to transpose the second vector so that you are dotting 1 x m with m x 1, which gives you a 1 x 1 result (equivalent to a scalar).
This question has come up periodically over time. Here’s a thread that covers all this and has links to a number of other relevant threads if my quick explanation above isn’t sufficient.
Also note that you filed this under AI for Python, so I used the little “edit pencil” on the title to move it to DLS C1 W2.
To save you the effort of going through that tree of links I gave above, here’s a more directed view:
This post most directly addresses just your particular question.
This thread is worth a look because it discusses the general notational conventions for the different types of matrix multiply.
And here’s a fun one that shows examples of how to apply the transpose in different ways when you have 1 x m vectors and what can go wrong. 
