Validate clinical phenotype candidates using online medical literature
Jan,2017 – Sep,2017
Working Paper(second author, submitted): “PIVET: A Scaled Phenotype Evidence Generation Framework using Online Medical Literature” (with J. Henderson, J. Ho, B. Wallace and J. Ghosh) Journal of Medical Internet Research(JMIR), 2017.
PIVET(Phenotype Instance Validation & Evaluation Tool) is the upgrade version of a phenotype validation prototype tool(PheKnow-Cloud). This tool examines the clinical significance of candidate phenotypes by conducting Lift analysis of phenotypic items’ co-occurrence on PubMed.
We first generate synonyms for the candidate phenotypes stored in our phenotype database using SNOMED-CT and ICD codes and then rank the synonyms based on overlap between candidate synonyms and most relevant synonyms on PubMed. Then, we perform co-occurrence analysis on all possible subsets of the synonym group and score this process using Lift. After generating the Lift scores, we group the subgroups up by cardinalities and apply classification models to the clinically validated phenotypes. Logistic Regression, k-NN, Lasso and Ridge Regression are implemented and Ridge excels with an F-1 score of 0.91.
The original framework is quite time consuming, usually take 16+ hours to finish the entire process on a 25% PubMed database subset. Meanwhile, it is also only capable of analyzing a single phenotype at a time. In addition, all the candidate phenotypes are generated by ML algorithms with no standard to compare to other than the comments of human experts.
In PIVET, we integrate noSQL(MongoDB) as well as SQL(mySQL) into the system to significantly improve the performance. Meanwhile, we speed up the synonym generation process by switching to using MeSH(Medical Subject Headings by NLM) term(assigning the most relevant MeSH term). We also introduce validated phenotypes from PheKB as the golden standard of our validation tool to improve classification performance. After building the regression model that predicts whether a candidate phenotype is clinically significant, we use PIVET to validate candidate phenotypes produced by Machine Learning algorithms(Marble and Rubik) and evaluate the quality of candidate phenotypes that are identified as “possibly clinically significant” by domain experts.
Using Logistic Regression, k-NN and Ridge Regression, we look at the prediction scores on the possibly clinically significant phenotypes generated by ML algorithms (human expert comments in column 3) and compare them to our prediction score (column 4).