Measure before you rank

1. Judge relevance first. For every query, label each possible result: Exact, Partial, or Irrelevant. Ambiguous wording needs the user's intent. Exact should count far more than Partial. Choose weights that fit your users.
2. Reward the order. The same result is worth more near the top than far down. Compare your returned order with the best order from the judgments.
3. Count what is missing. A relevant result that never appears is a loss. Judging only returned results can make a weak ranker look perfect.
4. Inspect the losers. An average can improve while some queries get worse. Review those weaker queries before accepting the change.
5. Check the full trade-off. Also check the first good result, good results found, response speed, and cost. A better ranking still fails if it is too slow or too expensive.

Decide with judgments, order, coverage, and guardrails.