_ C4W2 UNQ_C5 how to specify the input parameters for the dot_product_self_attention function

Course Q&A Natural Language Processing NLP with Attention Models
1

I am not sure how to specify the input params for the dot_product_self_attention layer. The function is supposed take three parameters from previous layer. I wonder how to refer to them when calling this function. Thanks!

return tl.Serial(
    tl.Branch( # creates three towers for one input, takes activations and creates queries keys and values
        [tl.Dense(d_feature), ComputeAttentionHeads], # queries
        [tl.Dense(d_feature), ComputeAttentionHeads], # keys
        [tl.Dense(d_feature), ComputeAttentionHeads] # values
    ),
    
    **tl.Fn('DotProductAttn', dot_product_self_attention(?,?,?), n_out=1), # takes QKV**
   ...
)

please help

2

Hi @Ashwin_Kotgire

The “HINT” suggests that you don’t need to call the function in this particular case:

# HINT: The second argument to tl.Fn() is an uncalled function

Note that this is different from the previous definition of AttnHeads and the latter AttnOutput (since these functions themselves return functions).

I hope that was not too confusing :slight_smile:

Cheers