W2 assignment: defining metric vs defining optimizer

[Goudout]

Hi again,

In the end of W2 assignment, we define an optimizer and a metric for our pretrained network where we start learning on the last few layers.

We define the optimizer via optimizer = tf.keras.optimizers.Adam(...) but the metric by metrics = ['<name of metric>']. Why is metrics = ['tf.keras.metrics.Accuracy()']incorrect (and also, why do we specify an array?)?

I’m confused, for the hundredth time this afternoon

By the way, I can’t thank you enough for all the help you’re giving me, so thank you…!

Cheers

[TMosh]

Not everything in Keras is entirely logical.

In one case you’re defining a text parameter, in the other you’re instantiating an object.

[balaji.ambresh]

Here you go:

1. See the explanation of metrics here
2. Accuracy checks for exact match.

[anon57530071]

I think I understand your confusion, since the string in here is a kind of “shortened form”, not an exact name of metrics. If “string” is given, Keras guesses what it means… If ‘accuracy’ or ‘acc’ is specified, then, Keras selects “categorical_accuracy” as a default. Depending to other conditions, Keras may select ‘sparse_categorical_accuracy’. So, if you really want to use a specific metrics, it is better to specify a function name, not a string.
In your case, function name is inside “quotation”, i.e, string. So, Keras tries to guess, but it may not be able to find an appropriate one, although an answer is almost there.
What you need is to remove quotation to explicitly declare it is a function name.
And, you can specify multiple metrics. So, square brackets are for creating a list of multiple metrics.

Hope this helps.

[Goudout]

Thank you for your answer,

But i’m not entirely sure I understand you correctly. I did try without the quotes, and here are 2 errors I got trying different thing:

If you have any idea on how to use tf.keras.metrics.Accuracy I’m interested.

[Goudout]

Hi again.

Thanks again for your aswers. I think I tried giving it an instance of metric.
From the 2.3 API doc, it seems that my first try in previous message should work:

Then I saw that the the verifier script expected exactly 'accuracy',

so I commented out that line, but still, the network is not learning properly. So it doesn’t seem to work.

Of course the string solution seems easier to use, butI’d like to understand why it is not, since I feel like I’m following the API doc (which is evidently not the case, but idk why) and it doesn’t work

[anon57530071]

I do not know your current status, but, at least Keras accepts this.

Setting Keras metrics is really confusing, since, I believe, it merged different implementations.

Historically, Keras WAS a high level API to run on Tensorflow, PlaidML and Theano. (And, I was using PlaidML as a backend for Keras.)
Keras had default metrics to set in here. (In V1, they prepared slightly different metrics. So, for this explanation, I use V2 APIs.) Available metrics are;

binary_accuracy
categorical_accuracy
sparse_categorical_accuracy
top_k_categorical_accuracy

Of course, only 4 metrics can not cover all. So, Keras allows users to develop their own function. I think you can see this in Keras document.

def mean_pred(y_true, y_pred):
return K.mean(y_pred)

model.compile(optimizer=‘rmsprop’, loss=‘binary_crossentropy’, metrics=[‘accuracy’, mean_pred])

In those Keras default implementation, we just set a class name, not an instance. This is one type.

Then, in 2020, Keras and Tensorflow were merged. (Personally, I was very disappointed, since Keras is no longer multi-backend support API.)
The challenge for them is how to merge different types of metrics classes in here.
So, (I guess) they decided to set Tensorflow classes as an instance in metrics definition just like
tf.keras.metrics.Accuracy() in the above case.

This causes confusions, of course. Please look at this.

Both work as “categorical accuracy”, but the first one comes from Tensorflow, and the second one comes from Keras.

In net, original Keras metrics and custom metrics do not use parenthesis, but Tensorflow classes need parenthesis.

Then, additional confusions come from ‘accuracy’, ‘acc’, …
Those are special strings that Keras guess what does it mean. Then, Keras starts to look at several environment and selects one of these.

binary_accuracy
categorical_accuracy
sparse_categorical_accuracy

So, there is a chance that a metrics, that you may not expect, is selected. In this sense, it is better to set your metrics with in either Keras style or Tensorflow style. (By the way, just setting ‘accuracy’ works most of cases, though…)

That’s the whole story.

[anon57530071]

By the way,

so I commented out that line, but still, the network is not learning properly. So it doesn’t seem to work.

Why do you think it is not working ? Learning heavily depends on the network design including loss functions.

Keras document clearly said that,

A metric is a function that is used to judge the performance of your model.
Metric functions are similar to loss functions, except that the results from evaluating a metric are not used when training the model.

So, usually, we revisit our network design and selection of loss function. Hyperparameter tuning is, of course, for that purpose. In some cases, just increasing the number of iterations (epoch) get a good result.

Hope this helps.

[Goudout]

Wow, thanks for the history of Keras x Tensorflow,and your explanations. Interesting stuff.

Blockquote Why do you think it is not working ? Learning heavily depends on the network design including loss functions.

Because when I run the script of the assignment, then the accuracy is stuck at 0 But at least the metrics is accepted!