I’m a lecturer at the Institute of Telecommunications and Informatics of Turkmenistan, currently taking AI for Medical Diagnosis and AI for Medical Prognosis. I’m building ŞYPA, an offline chest X-ray decision-support prototype with a Turkmen/Russian/English interface. It combines three pretrained classifiers (a DenseNet-121 trained on CheXpert, a TorchXRayVision DenseNet-121 and an EfficientNet-B0 for pneumonia), plus Grad-CAM, anatomical segmentation and an automatic cardiothoracic ratio. It has not been validated on local patients yet.
I’d be grateful for advice on four questions:
Evaluation: the three models have different label sets. What is a sound way to evaluate the whole system per pathology (sensitivity, specificity, AUC with bootstrap confidence intervals)?
Screening threshold: in mass screening the prevalence is low, so PPV drops sharply. How would you choose an operating point for screening versus diagnostic use?
Tuberculosis: none of the current models has a TB label. Is fine-tuning a pretrained DenseNet on TBX11K, Shenzhen and Montgomery a reasonable start? How would you limit patient leakage and domain shift when the target is local fluorography images?
Our pneumonia model was trained on the Guangzhou pediatric dataset (ages 1-5). Should we restrict it to children, or is there a good way to adapt it for adults?
your query cannot be responded until one knows what kind of 3 model architecture you are using for the model to detect it’s sensitivity, specificity.
one also needs to know how much dataset you are using, with ratio between cv: train: test, as you mentioned your model has very low prevalence. remember any disease can have lower prevalence, the factor to detect a disease is based on features you have assigned or your dataset has, will lead the better detection of feature to diagnosis outcome.
Until one knows the above, one cannot honestly respond to your queries efficiently.
Thank you for the quick reply! Here are the details.
Models (three image classifiers):
DenseNet-121 trained on CheXpert (14 labels), converted to ONNX. I use the published pretrained weights; I did not retrain it.
DenseNet-121 from TorchXRayVision (densenet121-res224-all, 18 labels, 224x224 input), also used with the published weights. It was trained by the library authors on NIH ChestX-ray14, CheXpert, MIMIC-CXR, PadChest and other public datasets.
EfficientNet-B0, binary (pneumonia vs. normal), fine-tuned by me on the Kermany/Guangzhou pediatric dataset: 5,856 images, standard Kaggle split of 5,216 train / 16 validation / 624 test. Test accuracy is 93.8%, but the test set is 62.5% pneumonia, so I know accuracy alone is not enough.
The outputs are merged into one list of findings, with a decision threshold for each pathology.
Evaluation data: I have not yet evaluated the whole system on a test set of my own, and designing that evaluation is exactly my question. My plan is to use a public set with radiologist-consensus labels that none of the three models was trained on (to avoid leakage), and later anonymized local images labeled by 2-3 radiologists.
Prevalence: sorry, I was unclear. I meant the screened population, not my dataset. In mass fluorography screening only a small share of people have a finding. For example, at 1% prevalence, 90% sensitivity and 90% specificity give a PPV of only about 8%. My question is how to choose the threshold for that setting.
Additional data: I would also value your advice on extending the system. My plan is to keep the three current models unchanged, because changing them would invalidate the calibrated thresholds, and to add separate modules instead:
tuberculosis, trained on TBX11K, Shenzhen and Montgomery;
adult pneumonia, trained on the RSNA Pneumonia Detection Challenge data;
an independent evaluation on the VinDr-CXR test set, which as far as I know none of the three models was trained on.
Is this a sound approach, or would you fine-tune the existing models instead? Are there other datasets you would recommend?
what do you mean by this? are you testing your models on 3 different diagnosis based on a feature. I am confused with the statement mentioning test accuracy being 93.8 and test set is 62.5% pneumonia? is this training outcome percentage??
as far as I can see your data is surely very less, but I cannot confirm on this as I don’t know the pretrained model you are using for pneumonia detection for tb is based on just chest x ray features or conjuncts with clinical/lab findings, ideally a diagnosis threshold having combination of clinical symptoms, with lab findings and radiological features is ideal.
you also mentioned the screening population was quite low which is understandable as the scaling screening matters when it comes where you are screening with your model, like if TB is prevalent in the screening population, or does it have any cofounding factors because remember pneumonia can be also seen post cancer patient too, so ofcourse one needs to fine tune their model even if they are using a pretrained model.
dataset is quite small, based on validation accuracy, it already reflects your validation accuracy is not that good, so test accuracy or prevalence will drop down when you scan on real time data.
Things to check, check your dataset in hand, try to gather atleast 10000 images, divide this into 60:20:20 (train:cv:test) but make sure your dataset has distinctive feature i.r..t. output/disease you want model to detect.
you also mentioned your pneumonia dataset is restricted to 1-5 yr age group, is another issue as to I am not understanding why you limited pneumonia detection with this toddler group.
do you know there are how many types of pneumonia??
A. based on causative factor (,like germs)
Bacterial
viral
fungal
atypical
B. based on environment
community acquired pneumonia (cap)
hospital acquired pneumonia (hap)
ventilator associated pneumonia (vap)
healthcare associated pneumonia (hap)
C special types
aspiration pneumonia
lipoid pneumonia
D. Based on anatomy
lobar pneumonia
bronchopneumonia
interstitial pneumonia
Infact TB has many variant and tuberculosis is diagnosed based on it stages of severity.
Thank you for the detailed reply. Your questions made me go back to my training records, and I found that my earlier post reported the results wrongly. Here is what is actually true.
1. What 93.8% and 62.5% mean
93.8% was not test accuracy. It was the best validation accuracy of an earlier version of my pneumonia model, and the validation set was the original Kaggle “val” folder of only 16 images (8 normal, 8 pneumonia). So that number means very little, and you were right to question it. Sorry for the confusion.
The model that ŞYPA uses now has 85.4% accuracy on the 624-image test set (390 pneumonia, 234 normal, so 62.5% pneumonia). I mentioned 62.5% because a model that always answers “pneumonia” would already score 62.5%, so accuracy alone says little. From now on I will report sensitivity, specificity and AUC with 95% confidence intervals.
2. Three models, not three diagnoses
These numbers are for one model only: my own pneumonia-vs-normal classifier (EfficientNet-B0, 5,856 images). ŞYPA also uses a second model I trained on a CheXpert subset (about 30,000 images, 14 findings) and the public pretrained TorchXRayVision DenseNet-121. Their outputs are combined into one result per finding. So yes, the data behind my own model is small.
3. Why only children aged 1–5
This was not a clinical choice. For my first prototype I used the Kaggle “Chest X-Ray Images (Pneumonia)” dataset because it was free, already labelled and small (about 2 GB). That dataset contains only children aged 1–5 from one hospital in Guangzhou. So the age limit comes from the data, not from a decision to focus on toddlers, and my model should not be used on adults in its current form.
4. Types of pneumonia
Thank you for the classification. ŞYPA answers only one question: “does this image show pneumonia-like changes?” It does not name the type. An X-ray mainly shows the anatomical pattern (your group D). The cause (A) needs microbiology and lab tests, the setting (B) comes from the patient’s history, and the special types (C) also need the history; the image can only hint at them. So the type stays with the doctor. The Kaggle file names even label each case as bacterial or viral, but I don’t think an image-only model should claim the cause.
5. Image only, and no TB
ŞYPA uses only the X-ray image, with no clinical or lab data, and it does not detect TB. I mentioned TB only as a possible future module. I agree that TB has many forms and stages, so a TB module would only flag “suspicious for TB, send for sputum testing”, never give a diagnosis or a stage. ŞYPA is meant as a second reader: the diagnosis stays with the doctor, who combines the image with symptoms and lab results, as you describe.
6. “Low” meant prevalence
By “low” I meant the disease prevalence in a screening population, not the number of people screened. At 1% prevalence, even a model with 90% sensitivity and 90% specificity has a PPV of only about 8%. Your point about confounders, such as opacities in cancer patients, is one more reason not to treat the image result as a diagnosis.
What I am changing because of your advice
Report sensitivity, specificity, AUC (95% CI) and PPV at a realistic prevalence instead of accuracy alone.
Train a separate adult pneumonia model on more than 10,000 images (RSNA Pneumonia, 26,684 images), split 60:20:20 by patient so no patient appears in two sets.
Test on external data that none of the models has seen: VinDr-CXR for adults and VinDr-PCXR for children under 10.
Add a short clinical section to the report (symptoms, temperature, SpO2, basic labs), so the doctor always sees the image result next to the clinical picture.
Keep the current ensemble and thresholds unchanged until a new model does better on the same external test.
One question: if I keep a separate external test set, would you still split the internal data 60:20:20, or would 80:20 (train/validation) be enough?
Thank you again. Your feedback has already made my evaluation more honest and my plan much clearer.
apologies for the delay in response as I was busy in festivities even though I had read your response and I wanted to response when I can response more appropriately.
Your last response is far better description of what you are doing and it helps me in responding better now.
First of all, any ai product especially related to specialised industry that involves human life need to be always created, trained and tested concurrently with a medical consultant as a must requirement, otherwise you as developer are not in only risking chances of failure for your product but also might be cause of setback because of lack of understanding the clinical assessment of a pneumonic patient related to x-ray. So I don’t know if you are exploring a competition on kaggle or creating a model for detection of pneumonia with the aid of x-ray, surely first need to make clear points on what is your end goal, what is achievable goals, and then proceed with any ai product research. divide work into technical, specialised teams, so your work is not held by one setback and each stage is being worked upon by different members even if you are facing challenges and discussing with each other, adding inputs once in a week adds value as well as time efficiency in research work
Now let me respond based on your response pointers
issue here is probably not your just training data being less, the fact that the validation data was created on 6 pneumonia and 6 normal images, tells why your test accuracy is low. your original data is probably overfitting and your testing data is underfitting, causing huge discrepancies. So you need to check or compare the data distribution between the original data and your training dataset. if your dataset is highly imbalanced(which I am suspecting in your case) and the val data was overfitting, any amount of specificity, sensitivity or auc can lead to biased result causing failure of your ai product or model on the screened population when tested leading loss of time and money.
what do you mean by 14 findings, are you stating out of 30,000 images only 14 data points has labelled features, then this confirms my previous statement of your training data being highly imbalanced and not giving you correct results.
Kaggle is one of the best place to explore dataset to work on your research, I agree but as you mentioned dataset is quite less probably because kaagle notebook don’t take much of memory backup like Google colab. this is the most annoying feature about kaggle notebook I dislike.
About choosing a dataset involving only children in your validation set and then testing on adults is another reason for less/lower prevalence of poor testing accuracy. You probably need to use the pretrained model and use your data into training as well as testing (make sure training and testing dataset includes all age groups, then when you test this model on screened population, your dataset can act as validation testset then, probably handling the overfitting issue with the original kaggle dataset, otherwise you need to completely disregard the original kaggle dataset, and divide your dataset into cv, train and test approach (but surely needs larger dataset).
right that’s the basic approach, and I see you understand the clinical difference aspect of detection too, but me explaining the classification was to suggest that you could add images of all the other types chest x-ray that has the causative factor as well as environment factor in your dataset, and label them with features like ground-glass opacities for interstitial pneumonia, lobar pneumonia has silhouette sign, bronchopneumonia has peribronchial thickening which all comes under community acquired pneumonia, same way for hospital acquired and healthcare associated pneumonia would have anatomical based chest x-ray features which if you include in your model, has much better feature detection leading to better accuracy. But this annotation of chest x-ray data needs to be done by either healthcare professionals or radiologist only.
I am skipping response to this as I can see you are working on a pneumonia feature detection ai model, so focus on one thing at a time.
I think I already responded the explanation why your ppv is so low, your dataset is highly imbalanced. So my response would be work on your dataset first before thinking about specificity, sensitivity or auc.
ofcourse if you are having external test set,you can use 80:20 approach. but please read the earlier response here in my comment which explains this query.
To close my response lastly, I have learnt most of the research work shouldn’t just include sensitivity, specificity and auc, but combination of these with f1 score as well as other statistical test based on what kind of data you are working and how your dataset is distributed. I am sharing one of my earlier post response which explains this, see if it helps. hope the lengthy response doesn’t distract you, break it and re-read again and again to understand what I stated, and apply into your research.