Publication

REMED: Retrieval-Augmented Medical Document Query Responding with Embedding Fine-Tuning

Tianqi Pang, Kehui Tan, Yujun Yao, Xiangyang Liu, Fanlong Meng, Chenyou Fan, Xiaofan Zhang

Published · International Joint Conference on Neural Networks (IJCNN) · 2024 · Second author

Embedding fine-tuning, retrieval, generation, and human evaluation in REMED.
Embedding fine-tuning, retrieval, generation, and human evaluation in REMED. View full-size figure ↗

The problem

Medical document answering depends on retrieving the right evidence before generating a response. General-purpose embeddings may not adequately distinguish specialized medical queries and documents, even when the language model itself is capable of producing fluent answers.

How REMED works

  • Embedding fine-tuning (EM-FT) adapts retrieval representations to the target medical collection. The backbone stays frozen while a trainable head learns from query-positive-negative document examples with a contrastive objective.
  • Retrieval ranks the collection using the adapted embeddings and selects supporting documents for the query.
  • Answer generation conditions the language model on that retrieved evidence, connecting the response to material in the document collection.

The method focuses on improving the evidence supplied to the generator through efficient adaptation of the retriever.

Adapting the retriever before generating an answer

REMED addresses the retrieval step of medical document answering. Instead of fine-tuning the entire embedding model, EM-FT treats its output as the input to a compact trainable head. The backbone parameters remain frozen; the head transforms query and document vectors into a representation better suited to the medical collection.

Figure 5. EM-FT: a frozen embedding backbone and a trainable retrieval head.
Figure 5. EM-FT: a frozen embedding backbone and a trainable retrieval head. View full size ↗

The head uses layer normalization, linear transformations, and a gated linear unit. Contrastive learning pulls a query toward relevant documents and separates it from negative documents. The study also compares a SwiGLU variant, EM-FT(s), and examines L2 regularization.

After adaptation, documents are ranked by their vector similarity to the query. Retrieved evidence is provided to the generation model as context. This keeps the retrieval adaptation distinct from training the answer-generating LLM.

Constructing the two medical collections

Medical Menu Dataset (MMD)

MMD organizes drug-related material into structured text, including drug names, product information, departments, classifications, and relevant attributes. Manual labels identify query-document relevance. The paper splits 561 document instances into 392 training and 169 test instances before query generation.

Figure 3. Construction and structured representation of the medical menu dataset.
Figure 3. Construction and structured representation of the medical menu dataset. View full size ↗

Medical Paper Dataset (MPD)

The paper collection contains 886 papers and 79,966 text entries. Documents are segmented to fit embedding inputs. GPT-3.5 generates candidate queries from paper titles or sampled passages, and BM25-based filtering removes low-relevance query-passage pairs. The paper compares title-derived queries with random-passage queries rather than treating query construction as a fixed detail.

Figure 4. Construction of query-document examples for the medical paper dataset.
Figure 4. Construction of query-document examples for the medical paper dataset. View full size ↗

Retrieval results

The tables report recall, precision, hit rate, and mean average precision (mAP) at Top-K = 10. Each measures a different aspect of retrieving relevant documents, and all are higher-is-better.

Table 1. Medical menu retrieval results at Top-K = 10.
Table 1. Medical menu retrieval results at Top-K = 10. View full size ↗

On MMD, M3E-base gives 0.503 recall and 0.548 precision; EM-FT with L2 gives 0.563 and 0.580. The table compares full-parameter fine-tuning, EM-FT variants, and regularization. The best recall and precision configuration is not necessarily the best configuration on every other metric.

Table 2. Medical paper retrieval results at Top-K = 10.
Table 2. Medical paper retrieval results at Top-K = 10. View full size ↗

On MPD with title-derived queries, E5-base-v2 reports 0.212 recall and 0.526 precision, while EM-FT reports 0.356 and 0.952. Random-passage evaluation is shown separately: EM-FT(s) has stronger results than EM-FT on several metrics in that setting. Preserving both query types avoids reducing the experiment to one favorable row.

Supervision labels and answer evaluation

Table 3. Ablation of supervision labels for medical menu retrieval.
Table 3. Ablation of supervision labels for medical menu retrieval. View full size ↗

The label ablation compares retrieval-model labels with manual drug-name-based labels and their combinations. It tests how the quality and construction of supervision affect adaptation, alongside the regularization and head-architecture comparisons in Table 1.

Figure 1. Example of medical answering with retrieved supporting evidence.
Figure 1. Example of medical answering with retrieved supporting evidence. View full size ↗

The answer example pairs retrieval results with the corresponding generated response. For human evaluation, the paper uses PULSE-7Bv5, top-two supporting passages, and 20 sampled queries, rated by five people including healthcare team members and generalists. The paper reports a mean comparative score of 0.34, with a p-value of 0.0041. This evaluates whether the adapted retrieval improves the resulting response in that experiment; it is not a claim of guaranteed medical correctness.

All publications