In Ungraded Lab M4_UGL1_1. in point 4. Evaluation Step – Preferred Domains- The objective evaluation is not required on per-example ground truth. As we have same ground truth that whether the response carries references of predefined list of preferred domain. This is same for all scenarios/example and not specific to examples. So this case should fall in -lower-left category of quadrant(Quadrant with two axes of evaluation)
Hi vaaCoder,
Thanks for your post.
The way I understand it is that when there are exact values to be matched, there is a per example ground truth, and when there are no exact values to be matched (e.g., length being less than or equal to 10) there is no per example ground thruth.
As the preferred domain is specified exactly, the output is compared to the exact values of that preferred domain, which would make it a case of per example ground truth. One could argue that even if the preferred domain contains exact values to be matched, the exact output is not compared to a per example ground truth, because it only needs to fall into the preferred domain and the specific output components are not matched exactly. So it depends on which part of the output is being considered. If only the domain is evaluated there is a per example ground thruth (for the domain). If the full output is evaluated there is not (for the full output).