C3W2 tokenize_labels returns correct shape but wrong values

Course Q&A TensorFlow Developer Professional Natural Language Processing in TensorFlow
, ,
1

Please advise where I should look for my error.

Thanks!

Shape of label_seq_np is: (1780, 1)
Shape of label_seq_np is: (445, 1)
First 5 labels of the training set should look like this:
[[6]
[3]
[4]
[4]
[8]]

First 5 labels of the validation set should look like this:
[[7]
[6]
[5]
[4]
[4]]

Tokenized labels of the training set have shape: (1780, 1)

Tokenized labels of the validation set have shape: (445, 1)

Expected Output:

First 5 labels of the training set should look like this:
[[3]
[1]
[0]
[0]
[4]]

First 5 labels of the validation set should look like this:
[[4]
[3]
[2]
[0]
[0]]

Tokenized labels of the training set have shape: (1780, 1)

Tokenized labels of the validation set have shape: (445, 1)

2

Please click my name and message your notebook as an attachment.

3

Please fix your code in def tokenize_labels.
Why use tokenizer when you are creating a new tokenizer for labels?

Regarding your question about one-hot encoding:

Week 2 in course 3 is about bbc news dataset and has nothing to do with sarcasm.

In this assignment, the labels are not one-hot encoded. This text pulled from the assignment markdown should make that clear:

First 5 labels of the training set should look like this:
[[3]
 [1]
 [0]
 [0]
 [4]]

First 5 labels of the validation set should look like this:
[[4]
 [3]
 [2]
 [0]
 [0]]

If the labels were one-hot encoded, this is how the same markdown would look:

First 5 labels of the training set should look like this:
[[[0. 0. 0. 1. 0.]]

 [[0. 1. 0. 0. 0.]]

 [[1. 0. 0. 0. 0.]]

 [[1. 0. 0. 0. 0.]]

 [[0. 0. 0. 0. 1.]]]

First 5 labels of the validation set should look like this:
[[[0. 0. 0. 0. 1.]]

 [[0. 0. 0. 1. 0.]]

 [[0. 0. 1. 0. 0.]]

 [[1. 0. 0. 0. 0.]]

 [[1. 0. 0. 0. 0.]]]

Does this help figure out the loss function?

4

Thanks for your advice. I’ve fixed the label tokenizer and the loss function (SparseCategoricalCrossentropy) but my codes for the labels are way out. Instead of being in the range of 1 to 5 they are in the hundreds. This causes the model to bail out (see below).

I’ve attached the notebook and would appreciate a further hint

[code removed - moderator]

Model Output:

Epoch 1/30

InvalidArgumentError Traceback (most recent call last)
in
1 model = create_model(NUM_WORDS, EMBEDDING_DIM, MAXLEN)
2
----> 3 history = model.fit(train_padded_seq, train_label_seq, epochs=30, validation_data=(val_padded_seq, val_label_seq))

/opt/conda/lib/python3.8/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.traceback)
—> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb

/opt/conda/lib/python3.8/site-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
56 try:
57 ctx.ensure_initialized()
—> 58 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
59 inputs, attrs, num_outputs)
60 except core._NotOkStatusException as e:

InvalidArgumentError: Received a label value of 781 which is outside the valid range of [0, 5). Label values: 0 0 781 146 146 0 146 643 643 781 0 0 781 781 643 146 146 781 146 0 0 0 643 0 0 781 781 0 146 0 781 0
[[node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits
(defined at /opt/conda/lib/python3.8/site-packages/keras/backend.py:5113)
]] [Op:__inference_train_function_711]

Errors may have originated from an input operation.
Input Source operations connected to node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits:
In[0] sparse_categorical_crossentropy/Reshape_1 (defined at /opt/conda/lib/python3.8/site-packages/keras/backend.py:5109)
In[1] sparse_categorical_crossentropy/Reshape (defined at /opt/conda/lib/python3.8/site-packages/keras/backend.py:3561)

Operation defined at: (most recent call last)

File “/opt/conda/lib/python3.8/runpy.py”, line 194, in _run_module_as_main
return _run_code(code, main_globals, None,

File “/opt/conda/lib/python3.8/runpy.py”, line 87, in _run_code
exec(code, run_globals)

File “/opt/conda/lib/python3.8/site-packages/ipykernel_launcher.py”, line 16, in
app.launch_new_instance()

File “/opt/conda/lib/python3.8/site-packages/traitlets/config/application.py”, line 845, in launch_instance
app.start()

File “/opt/conda/lib/python3.8/site-packages/ipykernel/kernelapp.py”, line 612, in start
self.io_loop.start()

File “/opt/conda/lib/python3.8/site-packages/tornado/platform/asyncio.py”, line 199, in start
self.asyncio_loop.run_forever()

File “/opt/conda/lib/python3.8/asyncio/base_events.py”, line 570, in run_forever
self._run_once()

File “/opt/conda/lib/python3.8/asyncio/base_events.py”, line 1859, in _run_once
handle._run()

File “/opt/conda/lib/python3.8/asyncio/events.py”, line 81, in _run
self._context.run(self._callback, *self._args)

File “/opt/conda/lib/python3.8/site-packages/tornado/ioloop.py”, line 688, in
lambda f: self._run_callback(functools.partial(callback, future))

File “/opt/conda/lib/python3.8/site-packages/tornado/ioloop.py”, line 741, in _run_callback
ret = callback()

File “/opt/conda/lib/python3.8/site-packages/tornado/gen.py”, line 814, in inner
self.ctx_run(self.run)

File “/opt/conda/lib/python3.8/site-packages/tornado/gen.py”, line 775, in run
yielded = self.gen.send(value)

File “/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py”, line 358, in process_one
yield gen.maybe_future(dispatch(*args))

File “/opt/conda/lib/python3.8/site-packages/tornado/gen.py”, line 234, in wrapper
yielded = ctx_run(next, result)

File “/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py”, line 261, in dispatch_shell
yield gen.maybe_future(handler(stream, idents, msg))

File “/opt/conda/lib/python3.8/site-packages/tornado/gen.py”, line 234, in wrapper
yielded = ctx_run(next, result)

File “/opt/conda/lib/python3.8/site-packages/ipykernel/kernelbase.py”, line 536, in execute_request
self.do_execute(

File “/opt/conda/lib/python3.8/site-packages/tornado/gen.py”, line 234, in wrapper
yielded = ctx_run(next, result)

File “/opt/conda/lib/python3.8/site-packages/ipykernel/ipkernel.py”, line 302, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)

File “/opt/conda/lib/python3.8/site-packages/ipykernel/zmqshell.py”, line 539, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)

File “/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 2894, in run_cell
result = self._run_cell(

File “/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 2940, in _run_cell
return runner(coro)

File “/opt/conda/lib/python3.8/site-packages/IPython/core/async_helpers.py”, line 68, in _pseudo_sync_runner
coro.send(None)

File “/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 3165, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,

File “/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 3357, in run_ast_nodes
if (await self.run_code(code, result, async_=asy)):

File “/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)

File “”, line 3, in
history = model.fit(train_padded_seq, train_label_seq, epochs=30, validation_data=(val_padded_seq, val_label_seq))

File “/opt/conda/lib/python3.8/site-packages/keras/utils/traceback_utils.py”, line 64, in error_handler
return fn(*args, **kwargs)

File “/opt/conda/lib/python3.8/site-packages/keras/engine/training.py”, line 1216, in fit
tmp_logs = self.train_function(iterator)

File “/opt/conda/lib/python3.8/site-packages/keras/engine/training.py”, line 878, in train_function
return step_function(self, iterator)

File “/opt/conda/lib/python3.8/site-packages/keras/engine/training.py”, line 867, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))

File “/opt/conda/lib/python3.8/site-packages/keras/engine/training.py”, line 860, in run_step
outputs = model.train_step(data)

File “/opt/conda/lib/python3.8/site-packages/keras/engine/training.py”, line 809, in train_step
loss = self.compiled_loss(

File “/opt/conda/lib/python3.8/site-packages/keras/engine/compile_utils.py”, line 201, in call
loss_value = loss_obj(y_t, y_p, sample_weight=sw)

File “/opt/conda/lib/python3.8/site-packages/keras/losses.py”, line 141, in call
losses = call_fn(y_true, y_pred)

File “/opt/conda/lib/python3.8/site-packages/keras/losses.py”, line 245, in call
return ag_fn(y_true, y_pred, **self._fn_kwargs)

File “/opt/conda/lib/python3.8/site-packages/keras/losses.py”, line 1737, in sparse_categorical_crossentropy
return backend.sparse_categorical_crossentropy(

File “/opt/conda/lib/python3.8/site-packages/keras/backend.py”, line 5113, in sparse_categorical_crossentropy
res = tf.nn.sparse_softmax_cross_entropy_with_logits(

5

This happens because you are using the global variable tokenizer when converting texts to sequences inside tokenize_labels.

1 Like
6

Thanks, that was obvious when I saw it :slight_smile:

I’ve read several documents but am still not clear on variable scope and why “tokenizer” has global scope. The variable “tokenizer” is declared inside function fit_tokenizer, and returned when you call the function. Does that make it global?

Or is it global because it is called by the test for function seq_and_pad?

Thanks!

7

tokenizer is not defined inside def tokenize_labels. Here’s a link to get started on scoping rules in python.