I just completed Course 1 - AI for Medical Diagnosis. The learning has been incredible, Thank you Dr Pranav, Dr Andrew Ng & all other mentors. A few Qs:-
1/ How does one translate(transfer learning) these models from lab to a clinical setting? Clinical setting may imply a setting in a developed or developing country.
2/ Let us focus, say on the Chest X-ray Model. Is there a paper that talks about the different digital formats in play for X-rays and how these different formats may or may not affect the model? What kind of test set size is recommended to run a check on?
3/ I am based in Vancouver(Canada), but my roots lie in India. In India(& quite a few countries), X-ray films are in vogue. In a typical clinical setting, photos of X-ray images(digital or film) are taken(via mobile phones) and shared extensively via email/WhatsApp for expert/second opinion. Is there a recommended methodology on how one would take this scenario into account to build a local test set?
4/ Is there a good paper or guiding document on how one should conduct a robust RCT(Randomized Trials) to test the efficacy of these models in a real-world scenario?
5/ One aspect about the dataset we used in the lab, it does not provide data about the patientâs age, sex, other clinically relevant information that might help with the final outcome(aka diagnosis). Is that relevant to improving model predictions? Logically, it seems prudent to say âyes.â Should we add these data points to the local test set? If yes, what is the recommended methodology for doing this?
6/ The datasets we will encounter in a clinical setting will be âraw.â Is there a guiding document that will help us achieve an acceptable level of standardization to make that dataset amenable to the model?
7/ The GRAD-CAM visualization was an excellent tool. How accurate are these? I believe these kinds of tools help engender trust with doctors/radiologists.
That is a great set of questions, @shamsheer_ahmed, and encompasses significant width, not all addressable in a short reply. But let me attempt to give some starting pointers where I can.
1/ How does one translate(transfer learning) these models from lab to a clinical setting? Clinical setting may imply a setting in a developed or developing country.
1a) Let me attempt to answer this at two different levels since I am not sure at what level you are enquiring:
At a high level that is a really wide question to address. It will include looking at a range of issues from IT and Tech Delivery including HCI/last mile, workflow integration etc to organizational transformation, priorities, education, policy and regulation and a host of other challenges. It will also involve the healthcare private sector, academia/research, entrepreneurs, public sector and of course the entire healthcare delivery system.
At the more specific level in the consuming direction, this âtransferâ role is still being done by bio/medical statisticians and the like, with the clinicians taking in only the processed information (of course after proper verification and validation). The process does need to get closer to the clinicians and I think that will continue to happen just as IT is increasingly moving out of the domain of the IT/computer folk and into that of the IT savvy âpower usersâ.
2/ Let us focus, say on the Chest X-ray Model. Is there a paper that talks about the different digital formats in play for X-rays and how these different formats may or may not affect the model? What kind of test set size is recommended to run a check on?
2a) Yes there are different formats for medical imaging. There is a good paper here I found.
There is no best/generic answer to the test set size question. In general the training data is big compared with the test/validation sets. Say training:validation:test at 60:20:20 might be a good place to start.
3/ I am based in Vancouver(Canada), but my roots lie in India. In India(& quite a few countries), X-ray films are in vogue. In a typical clinical setting, photos of X-ray images(digital or film) are taken(via mobile phones) and shared extensively via email/WhatsApp for expert/second opinion. Is there a recommended methodology on how one would take this scenario into account to build a local test set?
3a) Some of this refers to earlier stages of the classic DS pipeline, basically study and pre-processing- acquisition, prepping and storing. This will be followed by feature engineering, modeling, training, evaluation, prediction and the post-processing tasks of interpretation, communication and deployment. You might want to look up DS pipelines irrespective of the domain (here medicine) involved.
4/ Is there a good paper or guiding document on how one should conduct a robust RCT(Randomized Trials) to test the efficacy of these models in a real-world scenario?
4a) Here is one I could search out!
5/ One aspect about the dataset we used in the lab, it does not provide data about the patientâs age, sex, other clinically relevant information that might help with the final outcome(aka diagnosis). Is that relevant to improving model predictions? Logically, it seems prudent to say âyes.â Should we add these data points to the local test set? If yes, what is the recommended methodology for doing this?
6/ The datasets we will encounter in a clinical setting will be âraw.â Is there a guiding document that will help us achieve an acceptable level of standardization to make that dataset amenable to the model?
6a) What you are referring to is the pre-processing stage of the DS pipeline in general. There is a lot of information available online. Here is one link I found⌠Data Preprocessing: Definition, Key Steps and Concepts
7/ The GRAD-CAM visualization was an excellent tool. How accurate are these? I believe these kinds of tools help engender trust with doctors/radiologists.
7a) There is a fair amount of publicly available information of work using Grad-CAM incuding reports on accuracy enhancement. Trust will need not just accurate performance of the visualization but also explainability.
Jaidev(@getjaidev), Thank you for your patience in responding to my laundry list of Qs. I appreciate the very detailed responses. It is a great start. Keep up the good work!