Thresholds in TFMA Model Validation

Course Q&A Machine Learning Engineering for Production(MLOps) Machine Learning Modeling Pipelines in Production
1

in C3_W4_LAB_1_TFMA section Model Validation, it is emphasized to set thresholds by

    metrics_specs=[
        tfma.MetricsSpec(
            metrics=[
                tfma.MetricConfig(class_name='ExampleCount'),
                tfma.MetricConfig(class_name='BinaryAccuracy',
                      threshold=tfma.MetricThreshold(
                          value_threshold=tfma.GenericValueThreshold(
                              # Ensure that metric is always > 0.9
                              lower_bound={'value': 0.9}),
                              # Ensure that metric does not drop by more than a small epsilon
                              # e.g. (candidate - baseline) > -1e-10 or candidate > baseline - 1e-10
                              change_threshold=tfma.GenericChangeThreshold(
                                  direction=tfma.MetricDirection.HIGHER_IS_BETTER,
                                  absolute={'value': -1e-10}
                                  )
                              )
                      )
            ]

but what is the metric threshold and its lower_bound here?! when I increase or reduce the lower_bound, the result of validation doesn’t change?

2

Please clear something for me. Why does the markdown above the code cell not make sense?

Below, you will re-configure the EvalConfig settings to compare two models: a candidate and a baseline. You will also validate the candidate’s performance against the baseline by setting a tmfa.MetricThreshold on the BinaryAccuracy metric. This helps in determining if your new model can indeed replace your currently deployed model.

In this case, the metric of interest is BinaryAccuracy. We want the candidate model to have accuracy of at least 90% as indicated by lower_bound.