text_analysis.RmdTo start, we will load the package and the sample dataset. The same code below can be used with the entire dataset, but may be a bit slower.
library(ctrialsgov)
library(dplyr)
ctgov_load_sample()The function ctgov_kwic highlights all of the occurances of a term within its context (the few words before and after the term occurs). For example, if we want to show the occurances of the term “bladder” in the titles of the interventional trials we can do this:
z <- ctgov_query(study_type = "Interventional")
ctgov_kwic("bladder", z$brief_title)## ible Local Advanved |Bladder| Cancer
## isplatin Ineligible |Bladder| Cancer Patients
## n in the Overactive |Bladder|.
## men With Overactive |Bladder|
## on Catheter-Related |Bladder| Discomfort
## Non Muscle Invasive |Bladder| Cancer
## l Lung, Biliary and |Bladder| Cancer
## Non-Muscle-Invasive |Bladder| Cancer (NMIBC) Rela
## of Ezetimibe on Gall|bladder| Function
## e (DMSO) in Painful |Bladder| Syndrome/Interstiti
## tment of Overactive |Bladder| or Urge Urinary Inc
## ectrical Control of |Bladder| in Spinal Cord InjuThe function also has an option to include a title along with each occurance that is printed alongside each row. Here we will print the NCT id for each trial:
z <- ctgov_query(study_type = "Interventional")
ctgov_kwic("bladder", z$brief_title, z$nct_id)## [NCT04553939] ible Local Advanved |Bladder| Cancer
## [NCT04046094] isplatin Ineligible |Bladder| Cancer Patients
## [NCT04483817] n in the Overactive |Bladder|.
## [NCT04477265] men With Overactive |Bladder|
## [NCT02228473] on Catheter-Related |Bladder| Discomfort
## [NCT02214602] Non Muscle Invasive |Bladder| Cancer
## [NCT02240238] l Lung, Biliary and |Bladder| Cancer
## [NCT04498702] Non-Muscle-Invasive |Bladder| Cancer (NMIBC) Rela
## [NCT00634140] of Ezetimibe on Gall|bladder| Function
## [NCT00317070] e (DMSO) in Painful |Bladder| Syndrome/Interstiti
## [NCT00174798] tment of Overactive |Bladder| or Urge Urinary Inc
## [NCT00011570] ectrical Control of |Bladder| in Spinal Cord InjuThere are some other options that can be used to change the way that the output is displayed. The default (shown above) prints the results out using the cat function. Other options return the results as a character vector of data frame, which are useful for further post-processing. There is also a flag use_color that prints the term in color rather than with pipes; it looks great in a terminal or RStudio but does not display correctly when knit to HTML.
We can use a technique called term frequence-inverse document frequency (TF-IDF) to determine the most important words in a collection of of text fields. To implement this in R we will use the ctgov_tfidf function:
z <- ctrialsgov::ctgov_query()
tfidf <- ctgov_tfidf(z$description)
print(tfidf, n = 30)## # A tibble: 3,720 × 2
## doc terms
## <int> <chr>
## 1 0 wound|colonization|bacteria|ped|observe
## 2 1 mask|exertion|hypoxia|oxygenation|rest
## 3 2 depleted|conditioning|donor|stem|infusion
## 4 3 passed|tell|molecule|binding|cell
## 5 4 dash|diet|gut|bile|crc
## 6 5 tremelimumab|durvalumab|radiation|hypofractionated|signals
## 7 6 portion|89zr|combination|names|second
## 8 7 apnoea|immediately|ventilation|manually|pressure
## 9 8 tooth|preparation|bridge|dental|prosthesis
## 10 9 cervical|malawian|narrative|screening|grounded
## 11 10 problem|behavioral|stress|plus|immigrant
## 12 11 mindfulness|trainees|burnout|programs|online
## 13 12 t2d|liraglutide|obesity|adolescents|fda
## 14 13 specimen|lumpectomy|specimens|diseased|words
## 15 14 biomechanical|rugby|match|sport|event
## 16 15 cirrhosis|adrenal|outpatients|pathophysiologic|decompensated
## 17 16 ct|tomography|perfusion|emission|rest
## 18 17 operative|persistent|elective|post|opioid
## 19 18 medication|amongst|responders|bariatric|eating
## 20 19 asd|functioning|500|adulthood|adults
## 21 20 knee|injected|osteoarthritis|mild|increasing
## 22 21 basal|carcinoma|compliance|individualized|follow
## 23 22 mv|imt|los|op|icu
## 24 23 malignancies|consultation|gastrointestinal|understand|best
## 25 24 pcr|sore|insufflation|cuff|throat
## 26 25 erosive|esophagitis|healing|healed|endoscopically
## 27 26 thrombosis|ivc|cava|vena|veins
## 28 27 insomnia|cbt|ms|sleep|quantity
## 29 28 rotator|cuff|quicker|repair|healing
## 30 29 diverse|t2dm|clinics|regional|sleep
## # … with 3,690 more rowsThe default takes the lower case version of the terms, but (particularly with acronyms) it may be better to preserve the capitalization of the terms. Here is how we can do that in this example:
tfidf <- ctgov_tfidf(z$description, tolower = FALSE)
print(tfidf, n = 30)## # A tibble: 3,720 × 2
## doc terms
## <int> <chr>
## 1 0 wound|colonization|bacteria|PED|observe
## 2 1 mask|exertion|hypoxia|oxygenation|rest
## 3 2 depleted|conditioning|donor|stem|infusion
## 4 3 passed|molecule|tell|cell|binding
## 5 4 DASH|diet|gut|bile|CRC
## 6 5 durvalumab|radiation|signals|immune|liver
## 7 6 portion|89Zr|Administered|combination|second
## 8 7 apnoea|immediately|ventilation|expiratory|manually
## 9 8 tooth|preparation|bridge|dental|prosthesis
## 10 9 cervical|screening|Malawian|grounded|culturally
## 11 10 Problem|Plus|Management|behavioral|stress
## 12 11 mindfulness|burnout|programs|online|students
## 13 12 T2D|Liraglutide|obesity|adolescents|FDA
## 14 13 specimen|lumpectomy|specimens|diseased|words
## 15 14 biomechanical|rugby|match|sport|event
## 16 15 cirrhosis|adrenal|outpatients|pathophysiologic|decompensated
## 17 16 CT|perfusion|rest|tomography|stress
## 18 17 operative|post|elective|opioid|key
## 19 18 medication|amongst|responders|bariatric|eating
## 20 19 ASD|500|adulthood|adults|measures
## 21 20 knee|osteoarthritis|injected|mild|increasing
## 22 21 basal|carcinoma|compliance|individualized|follow
## 23 22 MV|LOS|IMT|op|ICU
## 24 23 malignancies|consultation|gastrointestinal|understand|best
## 25 24 sore|insufflation|throat|cuff|endotracheal
## 26 25 esophagitis|erosive|healing|healed|endoscopically
## 27 26 thrombosis|IVC|cava|vena|veins
## 28 27 Week|insomnia|CBT|MS|TAU
## 29 28 rotator|cuff|quicker|repair|healing
## 30 29 diverse|Determine|T2DM|Programs|Aim
## # … with 3,690 more rowsWe can also refine the results by including fewer rare terms. The argument min_df specifies the minimal proportion of documents that must contain a term for it to be returned as a keyword; the upper bound can also be specified with the argument max_df.
tfidf <- ctgov_tfidf(z$description, min_df = 0.02, max_df = 0.2)
print(tfidf, n = 30)## # A tibble: 3,716 × 2
## doc terms
## <int> <chr>
## 1 0 effects|sample|pilot|feasibility|open
## 2 1 surgical|higher|use|there|evidence
## 3 2 cell|regimen|method|immune|called
## 4 3 cell|tumors|stop|safe|side
## 5 4 receiving|investigators|participants|level|standard
## 6 5 radiation|immune|cancer|therapy|work
## 7 6 combination|second||dose|recommended
## 8 7 pressure|method|general|positive|end
## 9 8 some|significant|reported|no|therefore
## 10 9 screening|intervention|based|cancer|prevention
## 11 10 plus|support|i|goal|intervention
## 12 11 medical|increased|being|out|10
## 13 12 incidence|us|body|day|insulin
## 14 13 our|technique|analysis|results|evaluation
## 15 14 performance|cognitive|under|changes|aims
## 16 15 whether|determine|prevalence|setting|incidence
## 17 16 coronary|under|assess|comparison|normal
## 18 17 post|procedures|pain|surgical|common
## 19 18 medication|test|weight|loss|treatments
## 20 19 adults|measures|self|outcomes|functional
## 21 20 increasing|moderate|doses|activity|tolerability
## 22 21 follow|cell|up|risk|factors
## 23 22 pre|pulmonary|undergo|decrease|muscle
## 24 23 best|older|multi|adults|how
## 25 24 postoperative|pressure|including|over|effective
## 26 25 dose|explore|assigned|weight|doses
## 27 26 factors|either|without|outcomes|effectiveness
## 28 27 cognitive|i|week|outcomes|complete
## 29 28 both|novel|human|better|impact
## 30 29 diabetes|self|adults|education|
## # … with 3,686 more rowsAny number of text fields can be passed to the ctgov_tokens function; all of the fields for a specific trial are pasted together and treated a single block of text.
Finally, the package also provides a function for producing similarity scores based on the text fields of the studies. Here, we will produce a similarity matrix based on the description field of Interventional, Industry-sponsored, Phase 2 trials.
z <- ctgov_query(
study_type = "Interventional", sponsor_type = "Industry", phase = "Phase 2"
)
scores <- ctgov_text_similarity(z$description, min_df = 0, max_df = 0.1)
dim(scores)## [1] 176 176The returned value is a square matrix with one row and one colum for each clinical trial in the set. We can use these scores to find studies that are particularly close to one another in the words used within their descriptions. Here for example we can see five studies that use similar terms in their descriptions:
index <- order(scores[,100], decreasing = TRUE)[1:5]
z$brief_title[index]## [1] "Huperzine for Cognitive and Functional Impairment in Schizophrenia"
## [2] "Schizophrenia Study In Adults"
## [3] "A Study to Assess Safety and Efficacy of KarXT in Adult Patients With Schizophrenia"
## [4] "A Study of GWP42003 as Adjunctive Therapy in the First Line Treatment of Schizophrenia or Related Psychotic Disorder"
## [5] "A Study to Test the Effectiveness and Safety of Fremanezumab on Patients With Fibromyalgia"Further post-processing can be done with the similarity scores, such as spectral clustering and dimensionality reduction.