MIS 752 · Lab 5 · ShayPachi
Speak the Patient's Language
Semantic search vs. keyword search on a 20-sentence teaching corpus · all-MiniLM-L6-v2 · no PHI
The 30-second read
0.235mean within-topic cosine
0.129mean across-topic cosine
+0.106topic separation
0.917semantic MRR
Retrieval against the gold standard
| Method | Precision@3 | Recall@3 | MRR |
|---|---|---|---|
| Keyword (word overlap) | 0.375 | 0.319 | 0.500 |
| Semantic (embeddings) | 0.417 | 0.542 | 0.917 |
Meaning beats spelling when vocabulary runs out
The keyword baseline returned an empty result page for 5 of 12 queries. Semantic search found no relevant result for one query: “the reading on my meter jumped since supper.”
Example semantic search
Query: chest hurts when i climb stairs
1. car1 · cardiac/patient · cosine 0.787 — My chest feels heavy and tight when I walk up the stairs.
2. car2 · cardiac/clinical · cosine 0.392 — Patient reports exertional chest pressure radiating to the left arm.
3. car4 · cardiac/patient · cosine 0.374 — I get out of breath and my heart pounds for no reason at all.
1. car1 · cardiac/patient · cosine 0.787 — My chest feels heavy and tight when I walk up the stairs.
2. car2 · cardiac/clinical · cosine 0.392 — Patient reports exertional chest pressure radiating to the left arm.
3. car4 · cardiac/patient · cosine 0.374 — I get out of breath and my heart pounds for no reason at all.
Takeaway
The embeddings showed a clear, although not perfect, separation between topics. The mean cosine similarity was 0.235 within the same topic compared with 0.129 across topics, a difference of 0.106. Keyword search survived when patients used vocabulary already present in the corpus; for example, “chest hurts when I climb stairs” achieved 1.00 precision because words such as “chest” and “stairs” directly overlapped with the cardiac sentences. However, keyword search collapsed to an MRR of 0.00 on the patient’s-own-words queries, producing empty pages when there were no shared content words. In a real patient portal, that could incorrectly suggest that no relevant information exists. Semantic search performed much better but still missed “the reading on my meter jumped since supper,” returning no relevant result, likely because those everyday words were too indirect for the diabetes concepts in this small corpus. Outside healthcare, the same pipeline could help FBI or law-enforcement retrieval, connecting everyday descriptions of incidents with more formal investigative terminology.
Scope
This is a teaching demonstration, not a clinical validation. The corpus contains synthetic teaching text, no real patient records and no PHI. Retrieval scores show ranking behavior; they do not establish medical correctness or safety.