Elena Artemova · Елена Артемова · Published · Updated · Purpose of the tool · With application example.
Why eye assessment is not enough
One compelling answer does not show how the system will handle another type of question. After changing the model or searching, you may notice an improvement in the formulation and miss the deterioration of the grounds.
The need for Ragas arises when the comparison needs to be repeated on a specific data set. The tool helps organize the assessment, but does not decide for the team what is considered a useful answer. This agreement depends on the task and the consequences of specific mistakes.
What applies to assessment tools
The July 17, 2025 release of Ragas 0.3.0 includes changes to the metrics assessment and alignment tools. The development of the tool is related not only to the launch of the test, but also to the extent to which its conclusions meet the selected requirements.
For practical work, it is important to separate the measured properties. Completeness of the answer, compliance with the source and correct indication of the absence of data are different requirements. Combining them into one number can hide the flaw, which is the main one for the reader.
When is Ragas really useful?
It makes sense when the application changes regularly, when the team wants to compare results under the same conditions. The test set should include ambiguous questions and situations in which a correct answer is not possible.
Estimation by another language model can help process many results, but does not turn into an independent truth. Controversial decisions of the appraiser must be compared with human analysis. It is useful to understand which formulations of a criterion can change its conclusion.
When a tool won't solve the problem
If the criteria are not defined, automating the estimation will only speed up the production of unclear numbers. If the source documents are incorrect, a good result on a formal metric will not correct their content.
For a small, one-time task, a manual check may be sufficient. Ragas becomes justified when the comparison is repeated and the results need to be saved for team decisions. The main subject of discussion is specific errors and their impact on the task, and not the appearance of the quality report.
Illustrative Example
Compare two methods of searching the archive
- Task
- In a conditional comparison, two search options receive the same questions and the same set of documents. For questions, acceptable reasons for answering are known in advance.
- The role of the tool
- Ragas is used to organize the evaluation of the responses received. Compliance with the found fragments and behavior in the absence of information are considered separately.
- Expected result
- A list of differences in criteria and the controversial answers themselves are expected, which can be sorted out, and not just the overall assessment of the winner.
- What to check
- A human checks discrepancies with the automatic assessment. The data set and criteria must be the same for both options. No measured values are stated here.
Ragas is needed for repeatable comparison. The value of the assessment is determined by the quality of the criteria and verification data.
Source and reasons
Release date:
The primary source describes the instrument and its development. Conclusions about the choice - editorial analysis of the laboratory. An example explains the purpose of the tool; no measurements were carried out on it. Source date verified September 8, 2026.
Questions on the topic
Is it possible to rate an agent with one number?
Such a number may be a summary, but it hides the differences between types of errors. Individual criteria are useful for decisions.
Who determines the correct outcome?
The team responsible for the task, based on sources and recipient requirements. The assessment tool does not replace this agreement.
Are unanswered questions necessary?
Yes. They show whether the system can indicate a lack of information instead of an unsupported conclusion.