Issue in C4 W4 UNQ_C4

Course Q&A Natural Language Processing NLP with Attention Models
1

UNQ_C4

GRADED FUNCTION

def ReformerLM(vocab_size=33000, n_layers=2, mode=‘train’, attention_type=tl.SelfAttention):
“”"
Args:
vocab_size (int): size of the vocabulary
n_layers (int): number of decoder layers
mode (string): setting of the model which can be ‘train’, ‘eval’, or ‘predict’
attention_type(class): attention class to use
Returns:
model (ReformerLM): a reformer language model implemented in Trax
“”"

### START CODE HERE (REPLACE INSTANCES OF 'None' WITH YOUR CODE) ###
# initialize an instance of Trax's ReformerLM class
model = trax.models.reformer.ReformerLM( 
    # set vocab size
    vocab_size=vocab_size,
    # set number of layers
    n_layers=n_layers,
    # set mode
    mode=mode,
    # set attention type
    attention_type=attention_type
)

### END CODE HERE ###
return model

This is my code for UNQ_C$ but getting error in this section

Error:

Failed str_rep_check.
Expected:
Serial[
Serial[
Serial[
ShiftRight(1)
]
Embedding_33000_512
Dropout
Serial[
PositionalEncoding
]
Dup_out2
ReversibleSerial_in2_out2[
ReversibleHalfResidualDecoderAttn_in2_out2[
Serial[
LayerNorm
]
SelfAttention
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderFF_in2_out2[
Serial[
LayerNorm
Dense_2048
Dropout
Serial[
FastGelu
]
Dense_512
Dropout
]
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderAttn_in2_out2[
Serial[
LayerNorm
]
SelfAttention
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderFF_in2_out2[
Serial[
LayerNorm
Dense_2048
Dropout
Serial[
FastGelu
]
Dense_512
Dropout
]
]
ReversibleSwap_in2_out2
]
Concatenate_in2
LayerNorm
Dropout
Serial[
Dense_33000
]
]
LogSoftmax
],
but got:
Serial[
Serial[
ShiftRight(1)
]
Embedding_33000_512
Dropout
Serial[
PositionalEncoding
]
Dup_out2
ReversibleSerial_in2_out2[
ReversibleHalfResidualDecoderAttn_in2_out2[
Serial[
LayerNorm
]
SelfAttention
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderFF_in2_out2[
Serial[
LayerNorm
Dense_2048
Dropout
Serial[
FastGelu
]
Dense_512
Dropout
]
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderAttn_in2_out2[
Serial[
LayerNorm
]
SelfAttention
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderFF_in2_out2[
Serial[
LayerNorm
Dense_2048
Dropout
Serial[
FastGelu
]
Dense_512
Dropout
]
]
ReversibleSwap_in2_out2
]
Concatenate_in2
LayerNorm
Dropout
Serial[
Dense_33000
]
].

Failed str_rep_check.
Expected:
Serial[
Serial[
Serial[
ShiftRight(1)
]
Embedding_100_512
Dropout
Serial[
PositionalEncoding
]
Dup_out2
ReversibleSerial_in2_out2[
ReversibleHalfResidualDecoderAttn_in2_out2[
Serial[
LayerNorm
]
SelfAttention
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderFF_in2_out2[
Serial[
LayerNorm
Dense_2048
Dropout
Serial[
FastGelu
]
Dense_512
Dropout
]
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderAttn_in2_out2[
Serial[
LayerNorm
]
SelfAttention
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderFF_in2_out2[
Serial[
LayerNorm
Dense_2048
Dropout
Serial[
FastGelu
]
Dense_512
Dropout
]
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderAttn_in2_out2[
Serial[
LayerNorm
]
SelfAttention
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderFF_in2_out2[
Serial[
LayerNorm
Dense_2048
Dropout
Serial[
FastGelu
]
Dense_512
Dropout
]
]
ReversibleSwap_in2_out2
]
Concatenate_in2
LayerNorm
Dropout
Serial[
Dense_100
]
]
LogSoftmax
],
but got:
Serial[
Serial[
ShiftRight(1)
]
Embedding_100_512
Dropout
Serial[
PositionalEncoding
]
Dup_out2
ReversibleSerial_in2_out2[
ReversibleHalfResidualDecoderAttn_in2_out2[
Serial[
LayerNorm
]
SelfAttention
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderFF_in2_out2[
Serial[
LayerNorm
Dense_2048
Dropout
Serial[
FastGelu
]
Dense_512
Dropout
]
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderAttn_in2_out2[
Serial[
LayerNorm
]
SelfAttention
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderFF_in2_out2[
Serial[
LayerNorm
Dense_2048
Dropout
Serial[
FastGelu
]
Dense_512
Dropout
]
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderAttn_in2_out2[
Serial[
LayerNorm
]
SelfAttention
]
ReversibleSwap_in2_out2
ReversibleHalfResidualDecoderFF_in2_out2[
Serial[
LayerNorm
Dense_2048
Dropout
Serial[
FastGelu
]
Dense_512
Dropout
]
]
ReversibleSwap_in2_out2
]
Concatenate_in2
LayerNorm
Dropout
Serial[
Dense_100
]
].

Training loop

2

Hi Sumanta_Barman,

You have to embed your trax.models.reformer.ReformerLM in a tl.Serial layer that also includes a tl.LogSoftmax.