Thanks for the outstanding review. I was the Chief Medical Officer for a startup (GYANT) that makes an online symptom checker, and I'm familiar with the literature you've cited above. We also came to the conclusion that triage safety and accuracy were always going to be more important than diagnostic accuracy, and so that's we focused the majority of our development and QA work. A few additional points we learned along the way
• There are also situations where it may be inappropriate for even a highly accurate symptom checker to disclose a high possibility of a life-changing diagnosis (i.e, pancreatic cancer or new HIV infection).
• The frequently cited Semigran papers are useful as a basis for evaluation methodologies, but some of the vignettes they published are poor choices for OSC testing - for instance, vignettes derived from case reports of rare presentations, or from "gotcha" board exam questions. We and our clinical advisors created our own library of hundreds of vignettes derived from common and important presentations seen in regular clinical practice, rather than rely on the Semigran ones.
• We were surprised how often our clinical advisors would review a vignette case (or a transcript of a real world case that went through our system) and come up with discordant opinions on appropriate triage levels. I remember a number of occasions where 3 reviewers would give us 3 different answers. There's a lot of grey area in medicine!
Thanks for the outstanding review. I was the Chief Medical Officer for a startup (GYANT) that makes an online symptom checker, and I'm familiar with the literature you've cited above. We also came to the conclusion that triage safety and accuracy were always going to be more important than diagnostic accuracy, and so that's we focused the majority of our development and QA work. A few additional points we learned along the way
• There are also situations where it may be inappropriate for even a highly accurate symptom checker to disclose a high possibility of a life-changing diagnosis (i.e, pancreatic cancer or new HIV infection).
• The frequently cited Semigran papers are useful as a basis for evaluation methodologies, but some of the vignettes they published are poor choices for OSC testing - for instance, vignettes derived from case reports of rare presentations, or from "gotcha" board exam questions. We and our clinical advisors created our own library of hundreds of vignettes derived from common and important presentations seen in regular clinical practice, rather than rely on the Semigran ones.
• We were surprised how often our clinical advisors would review a vignette case (or a transcript of a real world case that went through our system) and come up with discordant opinions on appropriate triage levels. I remember a number of occasions where 3 reviewers would give us 3 different answers. There's a lot of grey area in medicine!