Yes! Thanks for pointing this out explicitly, Pete. ChatGPT would definitely have the upper hand in identifying the correct answer if it were trained on the exact case.
In comparing diagnosis and triage accuracy here (or accuracy of any AI model), it's important to keep the training, tuning and validation data sets separate. Otherwise, you run into this very issue.
The Semigran vignettes were very likely to have been included in ChatGPTs training data.
Yes! Thanks for pointing this out explicitly, Pete. ChatGPT would definitely have the upper hand in identifying the correct answer if it were trained on the exact case.
In comparing diagnosis and triage accuracy here (or accuracy of any AI model), it's important to keep the training, tuning and validation data sets separate. Otherwise, you run into this very issue.