I got lost in terms of the meaning of
strategy.reduce and what it does.
Here is a relevant code from the lab 3.
strategy.reduce(tf.distribute.ReduceOp.SUM, per_replica_losses, axis=None)
Andrew explains somewhere in the lecture the ‘map reduce’: it does aggregate losses.
strategy.reduce do the same thing as ‘map reduce’?
If so, I think that it is a very confusing name: why is this method called ‘reduce’, if it aggregate somethings?
If not, please advise me what ‘reduce’ does in the case of
I would appreciate it if anyone can do mentoring on this.
@Michio_Suginoo, you’re exactly right. The term “reduce” comes from the terminology used to describe the “map reduce” approach for working with distributed data. The general idea is that “mapping” is used to help us determine how to distribute the data across multiple processors that will be working in parallel. Then the “reduce” part will aggregate the work of all those processors.
The terms have been around for a while, and may not be the best, but I think the concept behind using the word “reduce” comes from the idea of “reducing” the set of parallel processes back down to one.
Thanks @Wendy for your note.
Now, I understand the historical background the issue.