Can you add one known variables to the (n-1) layer outputs for Deep learning model

Dear Community,

For example, I have model architecture like below: but somehow, I have one feature I know from RandomForest model works well. Can I add this feature into deep learning model? I want to add this feature together with output from layer (n-1).
Is there a way for me to do it?

Thanks

Hyperparameters

embedding_dim = 64
lstm1_dim = 64
lstm2_dim = 32
dense_dim = 64

Build the model

model = tf.keras.Sequential([
tf.keras.layers.Embedding(tokenizer.vocab_size, embedding_dim),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(lstm1_dim, return_sequences=True)),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(lstm2_dim)),
tf.keras.layers.Dense(dense_dim, activation=‘relu’), #I want to add here.
tf.keras.layers.Dense(1, activation=‘sigmoid’)
])

Print the model summary

model.summary()

Hello @Changbin_Du,

Ofcourse. One way is to have a separated input for the known variable and concatenate it in the right place:

in1 = tf.keras.layers.Input(...) 
in2 = tf.keras.layers.Input(...) # for the known variable

x = tf.keras.layers.Embedding(...)(in1)
x = tf.keras.layers.Bidirectional(...)(x)
x = tf.keras.layers.Dense(...)(x)
x = tf.keras.layers.concatenate()([x, in2])
out = tf.keras.layers.Dense(...)(x)

model = tf.keras.models.Model(inputs=[in1, in2], outputs=out)

Cheers,
Raymond

Thanks so much, Rmwkwok!

That is what I want. I will give a try…

1 Like

Hi, Ray,

Thanks for the help! I have data like
train.head()
ProbeID words mrFastCount Slope
0 AX-169355813 tatg atga tgac gacc accc ccct cctg ctga tgaa g… 0.04 0.428
1 AX-16629260 gaca acat catt attt tttt tttt tttg ttgg tgga g… 0.10 0.613
2 AX-169451789 ctct tcta ctag tagc agcg gcgt cgtg gtgg tggg g… 0.01 0.535
3 AX-169436214 tgcc gcca ccaa caat aatg atgc tgca gcat catg a… 0.05 0.235
4 AX-169450602 tcag cagt agtt gttt tttc ttca tcaa caac aaca a… 0.01 0.439

train.shape
(76373, 4)

X is based on column words only. It will go through the RNN model.
X[0:3,]
array([[110, 34, 154, 207, 209, 191, 79, 55, 20, 177, 71, 165, 195,
97, 7, 177, 28, 39, 18, 12, 44, 100, 155, 38, 152, 83,
59],
[128, 40, 31, 5, 3, 3, 17, 96, 48, 87, 159, 77, 183,
86, 66, 104, 80, 21, 52, 65, 136, 176, 143, 119, 101, 27,
102],
[ 61, 105, 195, 196, 225, 244, 213, 131, 85, 122, 203, 128, 59,
32, 49, 202, 67, 37, 113, 73, 193, 215, 235, 216, 141, 187,
146]], dtype=int32)

X.shape
(76373, 27)
train_y = np.array(train.Slope)

max_length=27
embedding_dim = 10
gru_dim=12
dense_dim = 6
NUM_EPOCHS = 50
BATCH_SIZE=128

#model architecture*****
in1 = tf.keras.layers.Input(shape=(max_length,))
in2 = tf.keras.layers.Input(shape=(1,)) # for the known variable --mrFastCount I want to add in the concatenate layer.

x = tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length)(in1)
x = tf.keras.layers.Bidirectional(tf.keras.layers.GRU(gru_dim,dropout=0.4))(x)
x = tf.keras.layers.Dense(dense_dim, activation=‘relu’)(x)
x = tf.keras.layers.concatenate()([x, in2])
out = tf.keras.layers.Dense(1)(x)

model_gru = tf.keras.models.Model(inputs=[in1, in2], outputs=out)

Set the training parameters

optimizer=tf.keras.optimizers.RMSprop(0.001)
model_gru.compile(loss=tf.keras.losses.Huber(), optimizer=optimizer, metrics=[‘mae’])

Print the model summary

model_gru.summary()

Train the model

history_gru = model_gru.fit([X, train.mrFastCount], train_y, batch_size=BATCH_SIZE, epochs=NUM_EPOCHS, validation_data=([val_padded, valid.mrFastCount], val_y))

I got the following error:

ValueError: Layer “model” expects 2 input(s), but it received 1 input tensors. Inputs received: [<tf.Tensor ‘IteratorGetNext:0’ shape=(None, 27) dtype=int32>]

What is wrong? Thanks

Hello Changbin,

According to the error message we need to focus on the inputs… You said:

model_gru.fit([X, train.mrFastCount], train_y, ...)

Let’s make sure all inputs and y are numpy arrays, and make sure all inputs have the same number samples (same zeroth axis length), and make sure the first & second inputs have the shapes of (None, 27) and (None, 1) as required by your architecture. Note that None represents any number of samples.

I think your 2nd input’s shape is not (None, 1) .

Cheers,
Raymond

Thanks so much, Ray!

As you suggest to make sure all inputs and y are numpy arrays, I made the following changes:
train_count = np.array(train.mrFastCount)
val_count = np.array(valid.mrFastCount)

#x = tf.keras.layers.concatenate()([x, in2]) # it cause error
#TypeError: concatenate() missing 1 required positional argument: ‘inputs’ and I changed to
x = tf.keras.layers.concatenate(inputs=[x, in2],axis=-1) and it works.

#The following shape parameters cause warning.
in1 = tf.keras.layers.Input(shape=(None, max_length))
in2 = tf.keras.layers.Input(shape=(None, 1)) # for the known variable

#WARNING:tensorflow:Model was constructed with shape (None, None, 27) for input KerasTensor(type_spec=TensorSpec(shape=(None, None, 27), dtype=tf.float32, name=‘input_17’), name=‘input_17’, description=“created by layer ‘input_17’”), but it was called on an input with incompatible shape (None, 27).
#WARNING:tensorflow:Model was constructed with shape (None, None, 1) for input KerasTensor(type_spec=TensorSpec(shape=(None, None, 1), dtype=tf.float32, name=‘input_18’), name=‘input_18’, description=“created by layer ‘input_18’”), but it was called on an input with incompatible shape (None,).

The I changed to
in1 = tf.keras.layers.Input(shape=(None, None, max_length))
in2 = tf.keras.layers.Input(shape=(None, None, 1)) # for the known variable

#WARNING:tensorflow:Model was constructed with shape (None, None, None, 27) for input KerasTensor(type_spec=TensorSpec(shape=(None, None, None, 27), dtype=tf.float32, name=‘input_19’), name=‘input_19’, description=“created by layer ‘input_19’”), but it was called on an input with incompatible shape (None, 27).
#WARNING:tensorflow:Model was constructed with shape (None, None, None, 1) for input KerasTensor(type_spec=TensorSpec(shape=(None, None, None, 1), dtype=tf.float32, name=‘input_20’), name=‘input_20’, description=“created by layer ‘input_20’”), but it was called on an input with incompatible shape (None,).

The I used the following, it works.
in1 = tf.keras.layers.Input(shape=(max_length,))
in2 = tf.keras.layers.Input(shape=(1,)) # for the known variables

Appreciated for the efforts and great help! Thanks, Ray!

Please print the shapes of each input.

max_length=27
vocab_size = 258

embedding_dim = 10
lstm_dim = 12
dense_dim = 6

NUM_EPOCHS = 50
BATCH_SIZE=128

in1 = tf.keras.layers.Input(shape=(max_length,))
in2 = tf.keras.layers.Input(shape=(1,)) # for the known variable

x = tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length)(in1)
x = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(lstm_dim,dropout=0.4))(x)
x = tf.keras.layers.Dense(dense_dim, activation=‘relu’)(x)
x = tf.keras.layers.concatenate(inputs=[x, in2],axis=-1)
#x = tf.keras.layers.concatenate()([x, in2])
out = tf.keras.layers.Dense(1)(x)

model_lstm = tf.keras.models.Model(inputs=[in1, in2], outputs=out)

Set the training parameters

optimizer=tf.keras.optimizers.RMSprop(0.001)
model_lstm.compile(loss=tf.keras.losses.Huber(), optimizer=optimizer, metrics=[‘mae’])

Print the model summary

model_lstm.summary()

Train the model

history_lstm = model_lstm.fit([X, train_count], train_y, batch_size=BATCH_SIZE, epochs=NUM_EPOCHS, validation_data=([val_padded, val_count], val_y))

Model: “model_17”


Layer (type) Output Shape Param # Connected to

input_33 (InputLayer) [(None, 27)] 0

embedding_79 (Embedding) (None, 27, 10) 2580 [‘input_33[0][0]’]

bidirectional_73 (Bidirectiona (None, 24) 2208 [‘embedding_79[0][0]’]
l)

dense_158 (Dense) (None, 6) 150 [‘bidirectional_73[0][0]’]

input_34 (InputLayer) [(None, 1)] 0

concatenate_4 (Concatenate) (None, 7) 0 [‘dense_158[0][0]’,
‘input_34[0][0]’]

dense_159 (Dense) (None, 1) 8 [‘concatenate_4[0][0]’]

==================================================================================================
Total params: 4,946
Trainable params: 4,946
Non-trainable params: 0


Layer (type) Output Shape Param # Connected to

input_33 (InputLayer) — [(None, 27)] — 0 —

embedding_79 (Embedding) — (None, 27, 10) — 2580 — [‘input_33[0][0]’]

bidirectional_73 (Bidirectiona — (None, 24) — 2208 — [‘embedding_79[0][0]’]
l)

dense_158 (Dense) — (None, 6) — 150 — [‘bidirectional_73[0][0]’]

input_34 (InputLayer) — [(None, 1)] — 0 —

concatenate_4 (Concatenate) — (None, 7) — 0 — [‘dense_158[0][0]’,
‘input_34[0][0]’]

dense_159 (Dense) — (None, 1) — 8 — [‘concatenate_4[0][0]’]

==================================================================================================
Total params: 4,946
Trainable params: 4,946
Non-trainable params: 0

Oh, sorry I misunderstood. It’s good that your model works now. Great work @Changbin_Du

Hi, Ray,
Here is the input shape for the X and train_count variable.

Train the model

print(X.shape)
(76373, 27)
print(train_count.shape)
(76373,)
history_lstm = model_lstm.fit([X, train_count], train_y, batch_size=BATCH_SIZE, epochs=NUM_EPOCHS, validation_data=([val_padded, val_count], val_y))
Epoch 1/50
597/597 [==============================] - 17s 18ms/step - loss: 0.0118 - mae: 0.1163 - val_loss: 0.0106 - val_mae: 0.1116
Epoch 2/50

Hey Changbin,

I wanted to take back my question but I realized too late that I had misunderstood you. But thank you, thank you for sharing with me the shapes, and that you are training your model :slight_smile: Your data looks like some DNA molecules, and it’s a regression problem. I hope you will train something very interesting :wink:

Cheers,
Raymond