As mentioned above. I want to ask about 3 problem:
1,We take the article and a part of the summary as input to the model and wait for the probability of the next word. So why does the output have the shape (1, padded_length, vocab_size) instead of (1,vocab_size)?
2, Can we do parallel computation when testing, if we do that the input size will be (1,batch_size, padded_length, vocab_size) ?
3, The decoder has to repeat for each word, so how could it be faster than CNN? Just compare speed. And what about the âtrue transformerâ instead of the model we implement in this assignment when it has to encode before decode.
that the image i croped from notebook
Hey @20020069_Le_Thai_S_n,
Welcome, and we are glad that you could become a part of our community
First, I would like to thank you for creating this thread. I also learnt something new, while trying to curate the answer to your thread.
This is because the model outputs the probabilities corresponding to each of the positions, i.e., if we have padded_length = 100, it will output the probabilities corresponding to each of the 100 positions (or tokens). And the model has been structured to do this exactly, since in order to compute the loss for each of the tokens during training, we need the probabilities corresponding to each of the tokens. I hope this makes sense now.
I donât think it should be an issue. Itâs quite analogous to CNNs being used on multiple examples simultaneously, during inference. We just need to make sure that the padded_length for each of the examples in a single batch is the same. And I guess that should be it.
As to the speed comparisons between transformers and CNNs, I am not sure whether this is a question that we should even think about. Note that CNNs are designed to exploit spatial information, which is not an attribute of any natural language application. On the other hand, sequence models and transformers are designed to exploit temporal information, which happens to be a key attribute of every natural language application.
If you really think this is a question of concern, feel free to train a CNN based architecture and a transformer based architecture for any natural language application, and you can decide for yourself, whether you want a faster CNN-based architecture with a huge drop in the performance or not. Please do share your results with the community.
Honestly speaking, I never thought about this at all prior to your question. I checked this article out, and in that, Text Summarization was included as an application for the Encoder-Decoder architecture. So, what are we missing here? Turns out @arvyzukai has already posted an answer to this question, which you can find here.
Thank you for your previous answers, You did help me a lot. But still, there is some cloud left in my mind.
At question 1, I still donât understand the shape yet. Why do we need to compute the probability for all the tokens that we already know? And if we do that, it means we only use the article for the computation; what is the summary for?
Another question:
I just noticed that we need to pass the embedding layer with a huge article many times. Is this because of the lack of an encoder?
I also remember that Jonas said something about weight loss = 0 for the input. What does it mean?
Cheers.
Son
I believe that my previous response to this was a bit ambiguous. Letâs say that the padded_length = 100 for all the examples in our dataset. Now, consider the training process. Say the first example has article_length = 50 and summary_length = 20; the second example has article_length = 40 and summary_length = 15; and so on.
Since the model needs to work with a batch of examples, most of which are likely to have different lengths of articles and summaries; hence, the model canât produce probabilities from a fixed position (in the first example, it is 51st, in the second example, it is 41st). Hence, we structure the model to produce probabilities for each of the positions, however, we only use the probabilities corresponding to the summary tokens, to compute the loss.
So, the thing to note here is that, although we compute the probabilities for the article tokens as well, we donât use them anywhere, and hence, the shape of the output. I recommend you to go through the section 1.2 of the assignment again, it discusses this point in a great depth.
If you understand the answer to the previous question, this becomes trivial to answer. Here, I believe that by âinputâ, Jonas was referring to the articles. Let me quote something from the assignment as well:
The loss is taken only on the summary using cross_entropy as loss function.
Let me pose you a question, and I guess the answer to this will become trivial to you. In the presence of an encoder, if we have huge articles, do you think that we wonât need to pass them through an embedding layer?
If you still are confused about this, then I recommend you to read about embedding layers and what they do, once again.
Hint: Embedding layers are used in both: Sequential networks like RNNs, GRUs, LSTMs; and Transformer-based networks.
I reread all the assignment again, and I realized that the shape of output is (1, paded_length, vocab_size) or (batch_size, paded_length, vocab_size) , where token_length = article_length + some_summary_token that we output from the previous step+pad token,choosing at token_length position, because it is a part of teacher forcing.
By better guessing the article, we can better guess the summary. By using backward propagation at every output, we improve the model for every next output before it goes too far from the correct answer and take a lot of time for gradient descent if we generate all the output, compute the cost, and then use backward propagation.
Therefore, the shape (1, token_length, vocab_size) at the test and evaluation times may be redundant.
There is nothing to do with input length like you mentioned in the previous answer. Thatâs what I think. Correct me if Iâm wrong.
But the question of masking the âgenerated article partâ is still there. Is that for the sake of training time in this small assignment, or is this the correct way to train this model since we only care about the summary?
That is not entirely correct. In this (C4 W2) assignment the mask for the article is 0. But in general that (also learning from an article) is a good idea and what people usually do they assign some small amount (like 0.05) instead of 0. But in this assignment there is a statement (before part 2):
The loss is taken only on the summary using cross_entropy as loss function.
I usually understand better with actual examples (part 1.2 - Preprocessing for Language Models: Concatenate It!):
Chelseaâs early season form may have led to comparisons with the
Arsenal âInvinciblesâ side, but Gary Neville believes they arenât even
as good as the Chelsea side from 10 years ago. Jose Mourinhoâs side
are currently four points clear at the top of the Premier League, but
after letting leads slip against both Manchester City and United,
their killer instinct has been called into question. âIf a team are
going to be playing for a 1-0 then you better see it out,â Neville
said on Monday Night Football. 'When I saw Jose Mourinho two weeks ago
he talked about the 2005 (Chelsea) team and (compared) the team he had
then to the team he has now and he said the killer instinctâs missing.
Chelsea have dropped more points from winning positions this season
than they did in the whole of 2004/05 . Chelsea took the lead against
both Manchester United and Manchester City, but drew both matches .
'When I look at the statistics they are staggering - 28 times they
(the 2004/05 team) scored first (in Premier League matches), 27 the
season after and they only dropped two and four points (respectively).
âThis team this season, even though theyâre at a really high level,
have scored first seven times and already dropped four points. Theyâve
got to get to that next level.â 'When (Manchester) City went down to
10 men I thought Chelsea let them off the hook and yesterday at 1-0 up
I think Chelsea let United off the hook. âThereâs a mentality shift.
Gary Neville was talking on Sky Sportsâ Monday Night Football show
with Sportsmailâs Jamie Carragher . Robin van Persie scores Manchester
Unitedâs injury-time equaliser against Chelsea at Old Trafford . The
away side had appeared to be in control of the game but were undone
with just moments remaining . 'At Manchester City they went from 55
per cent possession for the 10 minutes before the goal, and the 10
minutes after they went to 26 per cent possession, and City had 10
men. That canât be an instruction from the manager. Thatâs a shift in
the players. 'Yesterday (against Manchester United) they went from 64
per cent (possession before scoring) to 45 per cent (after scoring).
They switch off. âThis is not the manager changing it. The players who
have worked themselves into a 1-0 lead have then sat
deeper.â<EOS><pad>Chelseaare four points clear at the top of the
Premier League . Jose Mourinhoâs side have proved themselves to be
early title favourites . But Gary Neville believes there is still room
for improvement . The former Manchester United defender criticised
their lack of killer instinct . Chelsea dropped points against
both Manchester clubs .<EOS>
You can see that the 1 s are only for the summary.
As for the input_length - we need to determine the input_length for batching, if the batch size is 1 that would be irrelevant, but training is usually done with mini-batch training (some internet explanation). So for that we have an entire section (part " 1.3 - Batching with Bucketing") for saving compute and fitting into memory.
Cheers
P.S. I would suggest addressing one question at a time that way we could find the answers more efficiently
that way we could find the answers more efficiently