InvalidArgumentError when running a GRU model

AI Discussions
1

I have encountered this error when fitting a GRU model for a text classification task in Google Colab for my university assignment. ( the task is similar to that in Natural language processing in Tensorflow, week 3, C3_W3_Lab_4_imdb_reviews_with_GRU_LSTM_Conv1D, so I have used a very similar code).

My code is below

# Vocabulary size of the tokenizer
vocab_size = 10000

# Maximum length of the padded sequences
max_length = 1200

# Output dimensions of the Embedding layer
embedding_dim = 16

def tokenize_sentences(sentences):    

    # Parameters for padding and OOV tokens
    trunc_type='post'

    padding_type='post'

    oov_tok = "<OOV>"

    vocab_size = 11000#32000

    max_length = 1200#1600

    # Instantiate the Tokenizer class by passing in the oov_token argument

    tokenizer = Tokenizer(filters='!"*#$%&()+,--./:;<=>?@[\\]^_`{|}~\t\n',lower=True, num_words=vocab_size, oov_token=oov_tok)    

    # Fit on the sentences
    tokenizer.fit_on_texts(sentences) 

    # Print the length of the word index
    word_index = tokenizer.word_index

    print(f'number of words in word_index: {len(word_index)}')    

    # Convert sentences to sequences
    sequences = tokenizer.texts_to_sequences(sentences)    

    # Pad the sequences using the post padding strategy
    padded_sequences = pad_sequences(sequences, maxlen=max_length, padding='post',truncating=trunc_type)

    return padded_sequences

sent_tok_train = tokenize_sentences(sent_clean_train)
sent_tok_test = tokenize_sentences(sent_clean_test)
sent_tok_val = tokenize_sentences(sent_clean_val)

# Parameters

embedding_dim = 16

vocab_size = 10000

max_length = 1200

gru_dim = 32

dense_dim = 6

# Model Definition with GRU
model_gru = tf.keras.Sequential([

    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),

    tf.keras.layers.Bidirectional(tf.keras.layers.GRU(gru_dim)),

    tf.keras.layers.Dense(dense_dim, activation='relu'),

    tf.keras.layers.Dense(1, activation='sigmoid')

])

# Set the training parameters
model_gru.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])

# Print the model summary
model_gru.summary()

my model summary

Model: “sequential_2”

Layer (type) Output Shape Param #

embedding_2 (Embedding) (None, 1200, 16) 160000

bidirectional_2 (Bidirectio (None, 64) 9600
nal)

dense_4 (Dense) (None, 6) 390

dense_5 (Dense) (None, 1) 7

=================================================================
Total params: 169,997
Trainable params: 169,997
Non-trainable params: 0

so when I fit the model, I am getting this error with the first epoch:

num_epochs = 30

history_gru = model_gru.fit(sent_tok_train, labels_train, epochs=num_epochs, validation_data=(sent_tok_val, labels_val), verbose=2)

Epoch 1/30
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-91-e2264c2cd665> in <module>()
      1 num_epochs = 30
      2 
----> 3 history_gru = model_gru.fit(sent_tok_train, labels_train, epochs=num_epochs, validation_data=(sent_tok_val, labels_val), verbose=2)

1 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     53     ctx.ensure_initialized()
     54     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 55                                         inputs, attrs, num_outputs)
     56   except core._NotOkStatusException as e:
     57     if name is not None:

InvalidArgumentError: Graph execution error:

Detected at node 'sequential_2/embedding_2/embedding_lookup' defined at (most recent call last):
    File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py", line 16, in <module>
      app.launch_new_instance()
    File "/usr/local/lib/python3.7/dist-packages/traitlets/config/application.py", line 846, in launch_instance
      app.start()
    File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelapp.py", line 499, in start
      self.io_loop.start()
    File "/usr/local/lib/python3.7/dist-packages/tornado/platform/asyncio.py", line 132, in start
      self.asyncio_loop.run_forever()
    File "/usr/lib/python3.7/asyncio/base_events.py", line 541, in run_forever
      self._run_once()
    File "/usr/lib/python3.7/asyncio/base_events.py", line 1786, in _run_once
      handle._run()
    File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
      self._context.run(self._callback, *self._args)
    File "/usr/local/lib/python3.7/dist-packages/tornado/ioloop.py", line 758, in _run_callback
      ret = callback()
    File "/usr/local/lib/python3.7/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 536, in <lambda>
      self.io_loop.add_callback(lambda: self._handle_events(self.socket, 0))
    File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 452, in _handle_events
      self._handle_recv()
    File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 481, in _handle_recv
      self._run_callback(callback, msg)
    File "/usr/local/lib/python3.7/dist-packages/zmq/eventloop/zmqstream.py", line 431, in _run_callback
      callback(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py", line 283, in dispatcher
      return self.dispatch_shell(stream, msg)
    File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
      handler(stream, idents, msg)
    File "/usr/local/lib/python3.7/dist-packages/ipykernel/kernelbase.py", line 399, in execute_request
      user_expressions, allow_stdin)
    File "/usr/local/lib/python3.7/dist-packages/ipykernel/ipkernel.py", line 208, in do_execute
      res = shell.run_cell(code, store_history=store_history, silent=silent)
    File "/usr/local/lib/python3.7/dist-packages/ipykernel/zmqshell.py", line 537, in run_cell
      return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
      interactivity=interactivity, compiler=compiler, result=result)
    File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
      if self.run_code(code, result):
    File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "<ipython-input-91-e2264c2cd665>", line 3, in <module>
      history_gru = model_gru.fit(sent_tok_train, labels_train, epochs=num_epochs, validation_data=(sent_tok_val, labels_val), verbose=2)
    File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1384, in fit
      tmp_logs = self.train_function(iterator)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1021, in train_function
      return step_function(self, iterator)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1010, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1000, in run_step
      outputs = model.train_step(data)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 859, in train_step
      y_pred = self(x, training=True)
    File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/sequential.py", line 374, in call
      return super(Sequential, self).call(inputs, training=training, mask=mask)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/functional.py", line 452, in call
      inputs, training=training, mask=mask)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/functional.py", line 589, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/keras/layers/embeddings.py", line 197, in call
      out = tf.nn.embedding_lookup(self.embeddings, inputs)
Node: 'sequential_2/embedding_2/embedding_lookup'
indices[28,74] = 10176 is not in [0, 10000)
	 [[{{node sequential_2/embedding_2/embedding_lookup}}]] [Op:__inference_train_function_12418]

thank you for any advice.

2

So now it’s our job to do your homework for you in external classes? :nerd_face:

I’m not familiar with that part of the NLP courses, but my reading of the above line is that you’ve told the tokenizer that the vocabulary size is 11000, but the main model thinks it is 10000. What could go wrong? :scream_cat:

Note that the actual error is thrown on 10176 which is > 10000 and < 11000. Hmmmmm.

3

@bluetail, @paulinpaloalto points out a danger of hardcoding and using ‘global’ context variables. Accidents are too easy! One way you can reduce the error rate is, if you decide you really need to use them, always upper case the variable names when you are assigning the magik values. That is,

VOCAB_SIZE = 100

then every time you use it in the code, you’ll need to fully upper case, which will make these stand out from other Python (or any other languages besides maybe 1980’s era FORTRAN) variable names. In a production environment you might think about reading that from a config file so you don’t have to change code just to use a different vocabulary (size). Notice also that in these courses (not sure about your university) the unit tests and autograder will frequently use different sized data sets. If you have hard coded things (not to mention hard coding it to two different values in two different places) the tests are likely to report it to your dismay.

1 Like
4

yes, super mentor! :slight_smile: well spotted. I spotted it myself after a while and then forgot to delete this post. it works now though my val_accuracy metric is really rubbish around 50%.
I have started doing Andrew Ng’s NLP courses to better understand my university course which includes a task on NLP.

5

thank you. Mine is Napier Edinburgh University and I just started a masters in data engineering in January. I have not reached unit tests yet!