Deploying the Chest X-Ray Medical Diagnosis with TensorflowJS

Course Q&A AI For Medicine AI for Medical Diagnosis
1

Hi,

I am trying to deploy the model from the Week 1 assignment, the one for predicting diseases based on a Chest X-Ray. I am trying to deploy it with TerraformJS, so that I can have a static web page.

However, I am not getting the expected results. If anyone who is more familiar with Terraform/TerraformJS than myself could give me a hand, I would appreciate.

This is the javascript code

async function run() {
  // Import model
  image = document.getElementById("example_image");
  const model = await tf.loadLayersModel('http://localhost/ml/saved_model.tfjs/model.json');
  // Infere
  tensorImg = tf.browser.fromPixels(image).resizeNearestNeighbor([320, 320]).toFloat().sub(tf.scalar(127)).div(tf.scalar(127)).expandDims();
  const prediction = model.predict(tensorImg).dataSync();
  document.getElementById("prediction").innerText = prediction;
  alert("Done!");
}

run();

and the HTML:

<html>
  <head>
    <meta content="text/html;charset=utf-8" http-equiv="Content-Type">
    <meta content="utf-8" http-equiv="encoding">
    <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@2.0.0/dist/tf.min.js">
      import * as tf from '@tensorflow/tfjs';
    </script>
    
  </head>
  <body>
    Hello World
    <img id="example_image" src="00008270_015.png"/> 
    <div id="prediction">prediction</div>
    <script src="./app.js"> </script>
  </body>
</html>

The predictions I am getting for 00008270_015.png are:

0.005539100617170334,0.03543306514620781,0.015558389946818352,0.12826170027256012,0.4334452152252197,0.06943535804748535,0.12194258719682693,0.09075117111206055,0.04148877039551735,0.07736095041036606,0.15867561101913452,0.06838551163673401,0.010514311492443085,0.08459567278623581

They don’t look like what we get in the assignment … so I was wondering what could I be doing wrong? I suspect is how I deal with the sample… I tried scaling and normalizing but seems is not enough to get the same predictions.

Many thanks in advance

jordi

2

I think the problem is that I am not applying the same transformations to the Image. How could I apply the same transformations if I don’t have the training set available to do the “image_generator.fit(data_sample)” ?

3

I found out how to get the mean and the stdev from our assignment by adding this code to the get_test_and_valid_generator :

    print(image_generator.mean)
    print("DEBUG: std")
    print(image_generator.std)

and then I got that the mean is 126.0662 and the stdev is 63.458977, so I changed the js code to:


  training_avg = 126.0662;                                                                               
  training_stdev = 63.458977;                                                                            
  tensorImg = tf.browser.fromPixels(image).resizeNearestNeighbor([320, 320]).toFloat().sub(tf.scalar(training_avg)).div(tf.scalar(training_stdev));
4

Still I am getting this prediction vector for 00008270_015.png , which I don’t this is the expected one, but I can’t see what I am doing wrong…

0: 0.0037838786374777555
​
1: 0.029516877606511116
​
2: 0.021119052544236183
​
3: 0.20689532160758972
​
4: 0.4981258809566498
​
5: 0.09233061224222183
​
6: 0.2959546744823456
​
7: 0.06867408752441406
​
8: 0.03128588944673538
​
9: 0.1287572979927063
​
10: 0.26408666372299194
​
11: 0.19884125888347626
​
12: 0.004785448778420687
​
13: 0.08029219508171082

5

This is the js code

async function run() {                                                                                   
  // Import model                                                                                        
  image = document.getElementById("example_image");
  const model = await tf.loadLayersModel('http://localhost/ml/saved_model.tfjs/model.json');             
  // Infere
  canvas = document.getElementById("canvas_1");                                                          
  // Normalize?                                                                                          
  training_avg = 126.0662;                                                                               
  training_stdev = 63.458977;                                                                            
  tensorImg = tf.browser.fromPixels(image).resizeNearestNeighbor([320, 320]).toFloat().sub(tf.scalar(training_avg)).div(tf.scalar(training_stdev));
  // tf.browser.toPixels(tensorImg, canvas);    
  const prediction = model.predict(tensorImg.expandDims());
  prediction_data = prediction.dataSync();
  document.getElementById("prediction").innerText = " Cardiomegaly: " + prediction_data[1] + "\n Mass: " + prediction_data[9] + "\n Pneumotorax: " + prediction_data[12] + "\n Edema: " + prediction_data[3];
  console.log("X");
  console.log(tensorImg.expandDims().dataSync());                                                        
  console.log("prediction");                                                                             
  console.log(prediction_data);                                                                          
}

run();
6

I am comparing it with this python code

x, y = train_generator.__getitem__(0)
prediction = model.predict(x)
print(prediction[0])

which outputs:

[0.07961479 0.07197064 0.03906971 0.6105395  0.4109465  0.3004871
 0.6297872  0.63894725 0.05621344 0.2562762  0.49528223 0.3899192
 0.02828875 0.20088965]
7

Also, if I print the X , they look diferent :thinking:

notebook:
x,y = train_generator.getitem(0)
print(x)

[[[[ 0.25049764  0.25049764  0.25049764]
   [ 0.17381273  0.17381273  0.17381273]
   [-0.07157898 -0.07157898 -0.07157898]

javascript:
tensorImg = tf.browser.fromPixels(image).resizeNearestNeighbor([320, 320]).toFloat().sub(tf.scalar(training_avg)).div(tf.scalar(training_stdev));
console.log(“X”);
console.log(tensorImg.expandDims().dataSync())

Float32Array(307200) [ 1.2280974388122559, 1.2280974388122559, 1.2280974388122559, 1.1020317077636719, 1.1020317077636719, 1.1020317077636719, 0.9286913275718689, 0.9286913275718689, 0.9286913275718689, 0.8026256561279297, … ]

Is this the right way to compare it?
If X is different, does this mean I am not doing the image transformation correctly?

or I am misunderstanding what do I have to compare? Because the prediction that I printed does not resemble what we are seeing when we do the gradcam thing. In the gradcam we see Cardiomelagy p = 0.004 , Mass p = 0.097, Pneumotorax p = 0.033, Edema p = 0.005.

The expected values are all 0 for this image, and we can see this when we do

x,y = train_generator.__getitem__(0)
prediction = model.predict(x)
print(y[0])

But in the previous message we’ve seen that in the notebook some prediction values are higher than 0.6 :thinking:

[0.07961479 0.07197064 0.03906971 0.6105395  0.4109465  0.3004871
 0.6297872  0.63894725 0.05621344 0.2562762  0.49528223 0.3899192
 0.02828875 0.20088965]

Which is not expected.

What am I misunderstanding?

thanks in advance

8

I bit more research.

I look into the original image 00008270_015.png. The first pixel is 216. I checked this with this python code:

im = Image.open(IMAGE_DIR + "00008270_015.png") 
print(im.load()[0,0])

216

With Javascript, if I take the original image without any rescaling, I also get 216:

console.log(tf.browser.fromPixels(image).toFloat().dataSync())

[216,....]

If I rescale to 320x320, I get 204. I would say this can be expected from the rescaling.

Then, the normalized pixel is 1.228, which is the result for substracting the man and dividing by the standard deviation:

  tensorImg = tf.browser.fromPixels(image).resizeNearestNeighbor([320, 320]).toFloat().sub(tf.scalar(training_avg)).div(tf.scalar(training_stdev));
console.log("normalized");
console.log(tensorImg.expandDims().dataSync());

1.228

All this looks expected, right?

However, when I try this with python and Keras…

original = train_generator2.__getitem__(0)[0][0][0,0,0]
normalized = train_generator.__getitem__(0)[0][0][0,0,0]
print("original: {} normalized:{}".format(original, normalized))
print("expected normalized: {}".format((original-62.622482)/126.8739))

original: 149.0 normalized:0.25049763917922974
expected normalized: 0.6808139262685233

train_generator2 is the same as train_generator but without normalizing.

See, the original pixel is 149, which is very different from the expected 216, and the normalized value is 0.25, which is very different from the expected 0.68. The expected normalized is calculated by substracting the mean and dividing by the standard deviation.

This is the train_generator2 for reference:

def get_train_generator2(df, image_dir, x_col, y_cols, shuffle=True, batch_size=8, seed=1, target_w = 320, target_h = 320):
    """
    Return generator for training set, normalizing using batch
    statistics.

    Args:
      train_df (dataframe): dataframe specifying training data.
      image_dir (str): directory where image files are held.
      x_col (str): name of column in df that holds filenames.
      y_cols (list): list of strings that hold y labels for images.
      batch_size (int): images per batch to be fed into model during training.
      seed (int): random seed.
      target_w (int): final width of input images.
      target_h (int): final height of input images.
    
    Returns:
        train_generator (DataFrameIterator): iterator over training set
    """        
    print("getting train generator...") 
    # normalize images
    image_generator = ImageDataGenerator(
        samplewise_center=False,
        samplewise_std_normalization= False)
    
    # flow from directory with specified batch size
    # and target image size
    generator = image_generator.flow_from_dataframe(
            dataframe=df,
            directory=image_dir,
            x_col=x_col,
            y_col=y_cols,
            class_mode="raw",
            batch_size=batch_size,
            shuffle=shuffle,
            seed=seed,
            target_size=(target_w,target_h))
    
    return generator

why I am not getting 216 when I do train_generator2.__getitem__(0)[0][0][0,0,0]? And then, how is the “normalized value” being calculated?

9

I am answering myself. It was the “shuffle” argument of the ImageDataGenerator: tf.keras.preprocessing.image.ImageDataGenerator  |  TensorFlow Core v2.9.1

Surprisingly the shuffle argument does …well… shuffle :slight_smile: (ironic) . So I was looking at the wrong element. If we do not shuffle, the first has 205, which is totally expected after the resize, and the normalized value is 1.17, which is expected given the mean is 122.4 and standard deviation is 70.33.

10

This is how it looks like:

image
image1317×766 53.9 KB

Source code in GitHub - jordimassaguerpla/chestxray.ai