Data Generator Logic

Course Q&A Natural Language Processing NLP with Sequence Models
1

I am doing the assignment for 1st week and I found that It is done in this way

tweet = data_pos[pos_index_lines[pos_index]]

My question is that why we cannot make a data generator in this way

data_pos[i : i+batch_size]

What is the logic behind this?

2

Hi @Pirzada

Usually there are many ways (codes) to accomplish the same results. I guess the creators of the course thought the suggested way is more understandable (natural) for learners that are not very familiar with vectorized code and the loops (“while, for”) might be more familiar.

So, there would be problems when the i is near the end of and array (when + batch_size is outside the array) and you should handle that. Also, you should have to convert the tweet to tensor anyway.

So, all in all I think their choice was for a more conventional way.