The problem
Given a clinical trial report and a short statement about it, decide whether the report entails or contradicts the statement. The reports span eligibility, intervention, results and adverse events in dense medical language, and a small detail can flip the answer. This was SemEval-2024 Task 2 (NLI4CT).
How I built it
What it had to handle.
Numerical reasoning, complex semantics, clinical acronyms, and specialized syntax.
Fine-tuning.
Llama 2 and PubMedBERT fine-tuned with PEFT and LoRA for biomedical text classification.
Retrieval.
Retrieval-augmented generation for evidence retrieval and model interpretation.
One GPU.
LoRA, bfloat16, 8-bit loading and gradient checkpointing fit the 7B model on a single RTX 4090, at about 20 GB with a batch size of 8.
Baselines.
BM25 evidence retrieval, TF-IDF similarity and GPT-3.5 Turbo, to see what fine-tuning actually bought.
Results
66.74%
peak F1 score.
+12%
evidence-alignment accuracy over traditional models, with RAG.
In hindsight
Competitive, not leading: the top submissions reached about 0.85 F1. Mine missed subtle entailments where context mattered most.
