Question / confusion about YOLO

[aiai1]

I have seen Andrew’s lectures many times over and over and still struggling to understand all the details about YOLO mostly regarding what happens when an object spans more than one grid cell. The only answer I find repeatedly given this question is the canned lines “only one grid cell is responsible for predicting the object” or “object is assigned to only one grid cell” without giving any detail as to how this is actually accomplished. Bit question is… when number of grid cells is more than 3x3, say 19x19, it is quite clear that more than one grid cell will predict the objects with high probability. But during training of YOLO we are forcing it to stop predicting the object in cells to it was not “assigned”. In other words, aren’t we training it to ignore major parts of object that fall in non-assigned grid cells, and making it just use the part that falls in “assigned” grid cell? Wouldn’t that confuse the network and ultimately degrade its prediction ability???

[lambliu]

The training method of assigning an object to a single, center-based grid cell is not about forcing the network to ignore pixels. Instead, it is a highly effective training regularization that forces the network to:

1. Unambiguously identify the location of the object’s center.

2. Use its broad receptive field to aggregate information about the entire object (even the parts in neighbor cells).

3. Produce only one high-confidence prediction per object, greatly simplifying the output and improving speed.

While non-assigned cells are trained to be silent (Objectness=0), their visual input still contributes indirectly to the final accurate prediction made by the responsible cell.

[ai_curious]

Also see

Week 3: finding the correct cell in YOLO

In the example above there will be 2 grid cells with non-empty training data and 18 with empty. The grid cells with object centers contained within them have a ground truth bounding box that includes a perimeter around the entire object, not just those parts of the object within the grid cell that contains its center. The grid cells that don’t contain an object’s center will have no information about any objects. During training, incorrect grid cell center and bounding box shape predictions ar…

And

How does a cell detect a bounding box bigger than itself, YOLO?

This is correct, but it is important to see how that happens, which is described in one of the replies above in this thread. At training time you know the ground truth bounding box location, the training image dimensions, and the number of grid cells. From this, you can calculate which pixel is the ground truth bounding box center, and map that to one specific grid cell. That grid cell is the given a 1 for object presence, and the other grid cells 0. Grid cell center location / object prese…

Or even

https://www.coursera.org/learn/convolutional-neural-networks/lecture/fF3O0/yolo-algorithm

You can easily start with Y the correct shape and initialized to 0. Then, overwrite just the locations that have training labels. It is straightforward to determine in which grid cell the center of the object is located. First, determine how many pixels map to each grid cell. Then, from your bounding box label, determine the pixel coordinate of the object center. From those two numbers, determine how many grid cell units the object is offset from the origin. In the autonomous driving exerci…

[aiai1]

So you are saying that :

1. Objectness is zero for all neighboring gridcells when centroid does not lie in them EVEN when they contain large parts of the object?
2. Objectness is 1 for gridcell in which centroid lies and it considers/uses large parts of the object from neighboring gridcells due to convolutional application of sliding window as discussed in this slide?
3. 

[TMosh]

Yes, and Yes.

[ai_curious]

@aiai1 I agree with these assertions from @TMosh . However there are nuances.

First, the assignment of 1 and 0 as the objectness value happens at labeled training data creation time; population of Y. The output of forward propagation, \hat{Y} , contains predicted values, not assigned values.

Second, YOLO works because the input to forward propagation is the entire input image, X, not a sub-region, which is the case in sliding windows. The grid cells are not decompositions of that input. They influence the shape of the network output, not its input. Labeled ground truth bounding box shapes in Y are not constrained to lie within a single grid cell, and neither are the predicted bounding boxes in the network output \hat{Y}

Finally, despite best efforts, training and runtime prediction are imperfect. It is possible that more than one grid cell of \hat{Y} will end up with a non-zero objectness predicted value (and bounding box shape and location and class) for the same object in the input image. During training the cost function attempts to correct this. At runtime, we use non-max suppression to disambiguate.

[ai_curious]

@aiai1

without giving any detail as to how this is actually accomplished

Quick question regarding YOLO algorithm

Here is a little more detail mapped specifically to the image in the OP. This grid is 3x3, with what appears to be 2 anchor boxes (one taller-than-wide, one wider-than-tall). The YOLO-independent ground truth would have 2 labels, one for the person, one for the car. When converted to YOLO input for training, there would need to be 3 * 3 * 2 == 18 locations, each holding (1 + 4 + C) == 8 values; one for p_c, 2 each for bounding box location and shape, and C class indicators (as a one hot vecto…

Yet another example of how, actually, this is accomplished. YOLO is a step up in complexity from what has been covered previously in the lectures, hopefully these historical threads help clear things up a little.