Joint Adaptation of Retrieval and Generation in RAG: Domain-Tuned Embeddings and LoRA-Adapted 4B Language Models

Citation: Dmitriy Ulybin, "Joint Adaptation of Retrieval and Generation in RAG: Domain-Tuned Embeddings and LoRA-Adapted 4B Language Models", Universal Library of Engineering Technology, Volume 02, Issue 02.

Copyright: This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Retrieval-augmented generation (RAG) is a common way to ground language models in domain documents, yet systems that must answer within a time budget depend on large generators and a reranking stage. The study aims to substantiate a design in which a domain-tuned embedding model and a compact generator of about four billion parameters, adapted with low-rank adaptation (LoRA), are tuned as one system. The method combines a structured review of 24 sources, comparative analysis, and secondary computation on benchmark tables. The results show that eight distracting passages left four generators with 43% to 97% of their gold-only accuracy, that a 2.7-billion-parameter model had a derived context-dependence indicator of 89%, and that published joint tuning raised exact match by 5 to 14 points. The article proposes a co-adaptation loop, a criterion for retiring the reranker, and a twelve-cell evaluation matrix. The conclusions state when a compact generator can replace a larger one. The material will interest engineers and researchers who build question answering systems with response-time limits.

Keywords: Retrieval-Augmented Generation; Domain Adaptation; Text Embeddings; Dense Retrieval; Low-Rank Adaptation; Compact Language Models; Reranking; Context Noise; Time To First Token; Inference Cost.

https://doi.org/10.70315/uloap.ulete.2025.0202009