Thanks for the outstanding review. I was the Chief Medical Officer for a startup (GYANT) that makes an online symptom checker, and I'm familiar with the literature you've cited above. We also came to the conclusion that triage safety and accuracy were always going to be more important than diagnostic accuracy, and so that's we focused the majority of our development and QA work. A few additional points we learned along the way
• There are also situations where it may be inappropriate for even a highly accurate symptom checker to disclose a high possibility of a life-changing diagnosis (i.e, pancreatic cancer or new HIV infection).
• The frequently cited Semigran papers are useful as a basis for evaluation methodologies, but some of the vignettes they published are poor choices for OSC testing - for instance, vignettes derived from case reports of rare presentations, or from "gotcha" board exam questions. We and our clinical advisors created our own library of hundreds of vignettes derived from common and important presentations seen in regular clinical practice, rather than rely on the Semigran ones.
• We were surprised how often our clinical advisors would review a vignette case (or a transcript of a real world case that went through our system) and come up with discordant opinions on appropriate triage levels. I remember a number of occasions where 3 reviewers would give us 3 different answers. There's a lot of grey area in medicine!
Excellent insights! Thank you for sharing from your experience.
One of the aims of symptom checkers is to help reduce health anxiety by providing more accurate potential condition matches. In the event that a life-changing, anxiety provoking diagnosis is at play, I wholeheartedly agree that this news would be inappropriate for a highly accurate OSC to deliver. Hopefully, the OSC is successful in conveying that the condition MUST be evaluated with urgency given the clinical scenario, without inducing excessive worry. How did you approach a solution here?
Thanks for the outstanding review. I was the Chief Medical Officer for a startup (GYANT) that makes an online symptom checker, and I'm familiar with the literature you've cited above. We also came to the conclusion that triage safety and accuracy were always going to be more important than diagnostic accuracy, and so that's we focused the majority of our development and QA work. A few additional points we learned along the way
• There are also situations where it may be inappropriate for even a highly accurate symptom checker to disclose a high possibility of a life-changing diagnosis (i.e, pancreatic cancer or new HIV infection).
• The frequently cited Semigran papers are useful as a basis for evaluation methodologies, but some of the vignettes they published are poor choices for OSC testing - for instance, vignettes derived from case reports of rare presentations, or from "gotcha" board exam questions. We and our clinical advisors created our own library of hundreds of vignettes derived from common and important presentations seen in regular clinical practice, rather than rely on the Semigran ones.
• We were surprised how often our clinical advisors would review a vignette case (or a transcript of a real world case that went through our system) and come up with discordant opinions on appropriate triage levels. I remember a number of occasions where 3 reviewers would give us 3 different answers. There's a lot of grey area in medicine!
Excellent insights! Thank you for sharing from your experience.
One of the aims of symptom checkers is to help reduce health anxiety by providing more accurate potential condition matches. In the event that a life-changing, anxiety provoking diagnosis is at play, I wholeheartedly agree that this news would be inappropriate for a highly accurate OSC to deliver. Hopefully, the OSC is successful in conveying that the condition MUST be evaluated with urgency given the clinical scenario, without inducing excessive worry. How did you approach a solution here?