Error in Graded Lab1, exercise 2

[Guillermo_Acosta]

Cannot make exercise 2 work because result = constrained_gen(formatted) returns str instead of JSON object so it raises exception:

'str' object has no attribute 'issue_type'

As outlines documentation says, Outlines generation functions output plain strings by design, even when you restrict the model to a JSON schema. The library guarantees that the generated string adheres precisely to your constraints, but you must manually parse the returned text into a Python dictionary or object.

It’s impossible to edit ungraded cells. I tried to do an workaround with a custom constrained_gen function that parses the output to JSON before return it, but the string object I get before parsing is truncated (I don’t receive the full json string), and it raises:

 Invalid JSON: EOF while parsing a string at line 1 column

[Deepti_Prasad]

hi @Guillermo_Acosta

Are you encountering error? for exercise 2

[Guillermo_Acosta]

Hi. I encounter that in table every result fails:

Strategy 2: Constrained Decoding — Results

#	Issue	Status	Type	Severity	Time
0	add dream dataset	fail	-–	-–	0.0s
1	concatenate_datasets loads all the data into	fail	-–	-–	0.0s
2	load_metric from local “glue.py” meet error ’	fail	-–	-–	0.0s
3	Implement Dataset to JSON	fail	-–	-–	0.0s
4	How to choose proper download_mode in functio	fail	-–	-–	0.0s

I digged into it and I found it’s because of the exception I mentioned 'str' object has no attribute 'issue_type'. The code silently fails

[Deepti_Prasad]

@Guillermo_Acosta

this issue is already reported.

please wait for staff response to update, there needs to be changed in codes in the subsequent cell for the results to be json.

As Tom has notified the staff already again, but probably due to weekend, there might be delay in the update, so please wait until staff response on your thread.

[4eburek]

I found the issue, actually 4 of them:

1. constrained_gen returns str, that’s the error message about

2. max_new_tokens must be set, without it JSON cuts mid-value

3. repetition_penalty must be set, otherwise at certain example response turns into a crazy loop: {“issue_type”: “bug”, “severity”: “high”, “affected_components”: [“dataset”], “has_reproduction_steps”: true, “environment”: {“os”: "Linux/Windows/MacOSX/Linux/Windows/MacOSX/Linux/Windows/MacOSX/Linux/Windows/MacOSX/Linux/Windows/MacOSX/Linux/Windows/MacOSX

4. result is a dictionary, result.issue_type, result.severity dont work

import json
result = constrained_gen(formatted, max_new_tokens=1024, repetition_penalty=1.3)
result = json.loads(result.strip())

cd_results.append({“success”: True, “time”: elapsed, “issue_type”: result[“issue_type”], “severity”: result[“severity”]})

Ideally schema must have constraints:

from pydantic import Field, constr
class IssueEnvironment(BaseModel):
    python_version: Optional[constr(max_length=16)] = None
    pytorch_version: Optional[constr(max_length=16)] = None
    rocm_version: Optional[constr(max_length=16)] = None
    os: Optional[constr(max_length=16)] = None


class ParsedIssue(BaseModel):
    issue_type: Literal[“bug”, “feature_request”, “documentation”, “question”]
    severity: Literal[“critical”, “high”, “medium”, “low”]
    affected_components: List[constr(max_length=64)] = Field(max_length=32)
    has_reproduction_steps: bool
    environment: IssueEnvironment

Without constraints even with above fixes I get:*
“os”:“Linux/MacOS/X11/LinuxX64/WINDOWS_XP-WIN32_Windows_NT50_WINDOWS_VISTA_WINNT98_UNIX_LINUX_MINGW_GCC_LinuxMingw_xgmaux_winntxp_ntwinvista_windows_msys_gnu_wsl_darwin_osx_linux_macos_debian_rhel_centos_fedora_redhat_openbsd_freebsd_suse_opensolaris_aix_hpux_hp_uvx_ius_amd64_powerpc_alpha_armel_bpf_archlinux_linaro_eabi_embedded-linux-gnueabihf_cortex-a7aarch64_vfp-neon-vfpva_thumbipaiaqaiopapipm_avr_hppa_sparc_ppc_sh_stmdsp_xtensa_microblaze-microblazemips_niosii_thunderx_kmicroblakumqttmp_zynq-zynqu+zkvmzvvppclnxarmhf_linaro_abi_ibrix_trustedlinaro_tegra-trusty-som-xilinx-armeb_generic-poky-raspberrypi-barebox-alpine-chroot_jessie-debootstrap_buildrootsudoers-denyall-root-owner-only-everyone-can-read-all-the-files-in-this-directory-permissions-on-executable-filesthere-is-no-way-to-change-these-defaults-without-modifying-source-code-of-os-or-using-specific-tools-like-cryptsetup-and-so-forth-but-you-will-be-aware-that-most-users-do-not-know-howto-use-it-properly-if-at-alland-unless-your-system-has-special-purpose-security-measures-installedlike-hypervisor-based-firewallsor-enforced-user-level-access-controlthen-lackingpropersecuritymechanismsyoucannotguarnetthatyourdatawillbeprotectedfromunauthorizedaccessors-intermediariesin-transit-aswellasinterferencebymalicioussoftwarewhichtakeadvantageofanyweaknessesintheprotectedcomputervendor-supplieddefaultconfigurationwhichmaynotcomplywiththebestpracticesrecommendedbythesecuritycommunitybuttheydonotcoverallspecialcasesforexampleiftherewerenootherwaytosecureanapplicationthanusingitsproprietaryencryptionalgorithmsothereisanysuchcase”*

I am surprised that lab wasn’t tested before course went online and issue persists for a few weeks now, sloppy job

[TMosh]

@4eburek

I am surprised that lab wasn’t tested before course went online and issue persists for a few weeks now, sloppy job

The course was tested extensively.

Most likely, something has changed in the backend, and broken compatibility with the notebook.

[TMosh]

Update: Course staff are investigating the issue.

[jan.ravnik]

Hi, I have tested the lab and tried running it with and without the solution and it seems to me that all is working well. The solution code for exercise 2 is actually given in the markdown. Did you use that exact code when it failed, or you used another solution? As far as I know, the solution in the markdown works.

[tnabil]

Hi @jan.ravnik,

Yes, using the exact code in the markdown causes the issue highlighted by all the posts above, which is that the model returns a string, not an object, which causes the code that is trying to access attributes on the returned object, such as result.issue_type to fail.

Additionally, as @4eburek highlighted, not specifying max_new_tokens causes the output to be truncated, producing invalid JSON.

Could you please retest the lab and adjust it accordingly?

[tnabil]

Hi @jan.ravnik,

I think I understand now what you meant when you said the lab was working well. Despite the issues highlighted above, submitting the lab still results in a pass with a 100 score, so it looks like the grader is not impacted by those issues. However, this is not to say that the they do not affect the learning experience.

I, personally, spent a very long time trying to understand the results I’m getting and where the issues with the code are. The fact that most of the cells are frozen makes this even harder because one has to work around this while troubleshooting.

I think revisiting the lab to address the feedback will make the experience much better for future learners.

[Sutapa_Chattopadhyay]

I found the same issue where

1. max_new_tokens must be set

[Deepti_Prasad]

i thought they updated the lab, do you still get the fail test from the subsequent test cell @Sutapa_Chattopadhyay ?

[jan.ravnik]

Hi,

sorry this took so long. It is fixed now. The issue was only in the code that tests the answer, but not in the actual exercise or the grader.

It was indeed that the max tokens were not set, so no valid JSON was ever returned.

Thanks everyone for valuable outputs and discussion!

Best, Jan