I agree that the exception trace is pretty hard to read and doesn’t convey enough information to tell which actual line of code is “throwing”. The only definitive thing you can tell is that you must have a Numpy function someplace in your code where you really need a TF function. It could be anywhere, since all your previous functions will get called as part of the training. For example, do you use numpy .T to do the transpose anywhere or call np.reshape instead of tf.reshape? Of course you might well ask “if it was wrong in earlier code, why didn’t it fail the earlier tests?” But now that we are running training, the rules are different: when you’re doing backpropagation in TF, it’s all being handled for you automatically, but it requires that every operation in the compute graph be a TF operation. For numpy operations, TF can’t compute the gradients. I’m not sure that’s the explanation, since in the cases I’ve seen of including numpy operations it gives a more explicit message about not being able to compute gradients. FWIW here is another recent thread that shows at least one example of the kind of error you can get by mixing numpy operations in a TF compute graph.
If that’s not enough of a clue to find the error, then we can use the “in case of emergency, break glass” method and take a look at your code. I’ll send you a DM about how to do that.