The yml file for Great Expectations does not match with what is in the notebook

Hello Team,

Is it just me that has a mismatch with the “great_expectations.yml” file after running the command “great_expectations init”… I am trying to follow the instructions in the notebook for using S3 buckets for the different stores but the instruction mentions seeing a placeholder in the yml file that I will fill but when I ran it, I dont see anything like that, in fact there are no places to fill any placeholders, so I am manually editing the yml file to try and fit into what I see in the images in the jupyter notebook.

Please is this the same experience for everyone or am I missing something.

This is the screenshot of the yml file without any modification when I ran initialized great expectations.

This is the link to the classroom: Great Expectations Lab

UPDATE: So I figured you are actually supposed to update the yml file to fit into the context provided by the notebook, since we want to change the store location to S3

Hello @damolavictor,
That is correct if you follow the instructions in part 1 to setup the files:

  1. after great_expectations init you get the great_expectations.yml file, think of it as template you will use in step 1.3
  2. find the bucket names for artifacts and docs you will use in the next step
  3. if you follow the instructions you will replace the parts of the great_expectations.yml with the :
    class_name: TupleS3StoreBackend instead and your S3 bucket name from the cloudformation outputs

@Georgios , thanks for confirming.

Now I am having this issue when I run the command “great_expectations store list”, I have checked my yml file and I dont seem to find the issue:

Traceback (most recent call last):
  File "/home/ec2-user/environment/jupyterlab-venv/bin/great_expectations", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/cli/cli.py", line 146, in main
    cli()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 1694, in invoke
    super().invoke(ctx)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/cli/store.py", line 13, in store
    ctx.obj.data_context = ctx.obj.get_data_context_from_config_file()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/cli/cli.py", line 43, in get_data_context_from_config_file
    context: FileDataContext = toolkit.load_data_context_with_error_handling(  # type: ignore[assignment] # will exit if error
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/cli/toolkit.py", line 420, in load_data_context_with_error_handling
    context = get_context(context_root_dir=directory)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/context_factory.py", line 263, in get_context
    context = _get_context(**kwargs)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/context_factory.py", line 302, in _get_context
    file_context = _get_file_context(
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/context_factory.py", line 383, in _get_file_context
    return FileDataContext(
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/file_data_context.py", line 64, in __init__
    self._scaffold_project()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/file_data_context.py", line 91, in _scaffold_project
    if self.is_project_scaffolded(self._context_root_directory):
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/serializable_data_context.py", line 513, in is_project_scaffolded
    and cls.config_variables_yml_exist(ge_dir)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/serializable_data_context.py", line 260, in config_variables_yml_exist
    config = yaml.load(f)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/main.py", line 434, in load
    return constructor.get_single_data()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/constructor.py", line 119, in get_single_data
    node = self.composer.get_single_node()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 76, in get_single_node
    document = self.compose_document()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 99, in compose_document
    node = self.compose_node(None, None)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 223, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 223, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 223, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 223, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 113, in compose_node
    if self.parser.check_event(AliasEvent):
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/parser.py", line 146, in check_event
    self.current_event = self.state()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/parser.py", line 632, in parse_block_mapping_value
    if not self.scanner.check_token(KeyToken):
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/scanner.py", line 1794, in check_token
    while self.need_more_tokens():
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/scanner.py", line 211, in need_more_tokens
    self.stale_possible_simple_keys()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/scanner.py", line 360, in stale_possible_simple_keys
    raise ScannerError(
ruamel.yaml.scanner.ScannerError: while scanning a simple key
  in "/home/ec2-user/environment/gx/great_expectations.yml", line 77, column 3
could not find expected ':'
  in "/home/ec2-user/environment/gx/great_expectations.yml", line 78, column 15
(jupyterlab-venv) voclabs:~/environment $ great_expectations store list
Traceback (most recent call last):
  File "/home/ec2-user/environment/jupyterlab-venv/bin/great_expectations", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/cli/cli.py", line 146, in main
    cli()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 1694, in invoke
    super().invoke(ctx)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/cli/store.py", line 13, in store
    ctx.obj.data_context = ctx.obj.get_data_context_from_config_file()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/cli/cli.py", line 43, in get_data_context_from_config_file
    context: FileDataContext = toolkit.load_data_context_with_error_handling(  # type: ignore[assignment] # will exit if error
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/cli/toolkit.py", line 420, in load_data_context_with_error_handling
    context = get_context(context_root_dir=directory)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/context_factory.py", line 263, in get_context
    context = _get_context(**kwargs)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/context_factory.py", line 302, in _get_context
    file_context = _get_file_context(
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/context_factory.py", line 383, in _get_file_context
    return FileDataContext(
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/file_data_context.py", line 64, in __init__
    self._scaffold_project()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/file_data_context.py", line 91, in _scaffold_project
    if self.is_project_scaffolded(self._context_root_directory):
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/serializable_data_context.py", line 513, in is_project_scaffolded
    and cls.config_variables_yml_exist(ge_dir)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/great_expectations/data_context/data_context/serializable_data_context.py", line 260, in config_variables_yml_exist
    config = yaml.load(f)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/main.py", line 434, in load
    return constructor.get_single_data()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/constructor.py", line 119, in get_single_data
    node = self.composer.get_single_node()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 76, in get_single_node
    document = self.compose_document()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 99, in compose_document
    node = self.compose_node(None, None)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 223, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 223, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 223, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 223, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/composer.py", line 113, in compose_node
    if self.parser.check_event(AliasEvent):
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/parser.py", line 146, in check_event
    self.current_event = self.state()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/parser.py", line 632, in parse_block_mapping_value
    if not self.scanner.check_token(KeyToken):
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/scanner.py", line 1794, in check_token
    while self.need_more_tokens():
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/scanner.py", line 211, in need_more_tokens
    self.stale_possible_simple_keys()
  File "/home/ec2-user/environment/jupyterlab-venv/lib64/python3.9/site-packages/ruamel/yaml/scanner.py", line 360, in stale_possible_simple_keys
    raise ScannerError(
ruamel.yaml.scanner.ScannerError: while scanning a simple key
  in "/home/ec2-user/environment/gx/great_expectations.yml", line 77, column 3
could not find expected ':'
  in "/home/ec2-user/environment/gx/great_expectations.yml", line 78, column 15

Hello @damolavictor,
That looks correct however you need to add the actual bucket names from the cloudformation outputs artifact bucket (3 places) and docs bucket (1 place). Hope it helps

Yes I am aware, I didnt want to expose them, since my account ID is in them. Not sure why I still face the error, even after replacing the bucket ID’s

@damolavictor I can see it has the correct indentation just a small typo in the yml file next to profiler store:

profiler_store: <--TYPO
    class_name: ProfilerStore
   ...

You are right, that was the fix, I dont know were that came from, I just assumed it came by default, found it weird but just felt it was needed for the initialization. Thank you for your help, the command runs fine now​:raised_hands:t4:

1 Like

@damolavictor, you could remove the code, glad it fixed. Thanks

1 Like