Filtering data output from a layer using the label

Hi I’m relatively new to deep learning and I was trying to implement a domain adversarial classifier in tensorflow. One aspect of that is feeding in to a layer only the data elements with corresponding label equal to a specific value.
Eg. label y can take values 0 or 1
I want a layer inside the model that only passes through the input features to the next layer if the current data element has y =1 , else I want the layer to produce 0s as the output.
I passed in the labels also as an input in addition to xtrain and made a custom layer that does the following.
Thus, I did:
model=Model(inputs=[infeatures,inlabels],outputs=[output])
and model.fit(x=[xtrain,ytrain],ytrain,epochs,batchsize)
However, my issue is that I obviously do not want the labels to be passed in as input during testing.
Is there any other way to do this? Or is it possible to drop certain components of the model architecture during testing?

basically what I’m trying to ask is ,how do we filter input to a particular layer based on the label of the data element. Please respond if anyone has any clue!