Similarity Is Not Discovery

The sidecar has a regular guest, and it is an embarrassing one. A researcher is reading a paper on retrieval evaluation and opens the side panel. Ranked first is a paper with nearly the same vocabulary: the same benchmarks, the same metrics, the same dataset family. She opens it. It contributes nothing. Same conclusions, weaker experiments, no new evidence, no disagreement, no extension. In the corpus it is the closest thing to what she is reading. As information it is empty. The system did its retrieval job perfectly and its discovery job not at all.

That failure points at a question the previous chapter left open. A sidecar can bring other material beside a source. But when the system puts one piece of information next to another, it is implicitly saying something about how they go together. Proximity reads as endorsement: this belongs here. The question is what, exactly, is being claimed.

the sidecar can bring in other information
        ↓
but "related" says almost nothing
        ↓
similarity is only one relation
        ↓
different relations support different uses
        ↓
the interface must know which relation it is exposing

What similarity measures

Most systems that find “related” material measure closeness. Keywords overlap, or two passages sit near each other in an embedding space, which is a numerical representation in which texts about similar things land close together. That is a useful measurement, and it is a fact about two texts. It is not a statement about what one of them does to the other.

The difference shows up wherever relationships matter. “The treatment improved outcomes” and “the treatment did not improve outcomes” share almost every word, and they say opposite things. A paraphrase and a contradiction can sit at the same distance from a claim. A paper reporting that a result no longer holds under a new protocol can be closer, as text, than the paper whose method actually bears on the reader’s question. Closeness was measured correctly in each case. It was the wrong thing to measure, because the reader needed to know how the candidate relates, and no distance carries that. Two passages can be highly similar and still be useless to one another for the reader’s task.

similarity  ≠  relationship

A vocabulary of relations

If similarity is one relation among many, it helps to name some of the others. A working list, which could be extended or argued with:

similar to      is analogous to     is an example of
supports        contradicts         qualifies
explains        depends on          causes
updates         supersedes          duplicates
extends         implements          replicates

They fall roughly into groups, and each group suggests different uses. Some are about evidence: supports, contradicts, qualifies, replicates. A reader checking a claim wants these. Some are about explanation: explains, depends on, causes, is an example of. A reader trying to understand wants these. Some are about time and succession: updates, supersedes. A reader worried about whether something is still true wants these. Some are about practical use: implements, extends. A reader who wants to build something wants these. And some are about resemblance: similar to, is analogous to, duplicates. These can help a reader see a pattern or avoid reading the same thing twice, and they are the weakest of the lot as grounds for belief.

What a system should do with a candidate depends heavily on which relation it holds. A contradiction deserves scrutiny. A duplicate can usually be collapsed. A dependency decides reading order. An update may replace what the reader had. An analogy is a prompt for thought, to be treated with caution. Treating all of these as “related items” throws away exactly what a reader needs in order to decide what to do.

This also sharpens what similarity does not tell you:

similarity
   ≠ support
   ≠ contradiction
   ≠ causation
   ≠ dependence
   ≠ relevance

Two passages can be near each other without one supporting the other, and far from each other while one contradicts the other. Two things can be related causally without sharing vocabulary. And relevance is not a property of the pair at all: it depends on what the reader is trying to do, which is a separate question that comes later.

Making the relation explicit

The architectural point follows directly. If relationships are what a reader needs, they should be stated, not inferred from proximity. The unit is a relation record, and it need not be large:

item A
relation type
item B
evidence or source for the relation
standing: how much the system trusts this label

For example: this paper contradicts claim 3 of the paper being read, on the evidence of these passages, under this condition, with this confidence. Every part of the record does work. The type says what is claimed. The two items are identified, with direction, because “A updates B” is not “B updates A”. The evidence lets someone check. And the standing allows honest statements such as “a possible contradiction, unconfirmed”, or simply “unknown”, which is a legitimate value and should be treated as one.

This is why the semantic record of the earlier chapter had a field for relationships. Inside one source, relations between its own claims were worth storing explicitly, because renderings that infer them tend to corrupt them. Between sources the argument is the same, scaled up. A relation record is the cross-document version of what the record already does within a document, and it brings the same duties: direction preserved, scope preserved, the path back to the words kept.

Scope deserves particular emphasis. A contradiction outside the relevant scope is not a contradiction of the claim. A result in a different population, a different period or under different assumptions can look like disagreement and be nothing of the sort. Whatever label a system attaches has to preserve the scope it asserts.

Where relations come from

Relations reach the record in three ways, and the route should be visible.

Asserted by a source. A paper says it replicates, refutes or extends a named earlier work. This is the author’s claim about the relation, with the author’s reasons, and it carries a citation.

Classified by the system. A model or a trained classifier reads two passages and proposes a relation. This is the most scalable route and the least reliable, and its output should carry the standing of a proposal.

Adjudicated by a person. A reader or editor confirms or corrects the label. Slow, and the most trustworthy.

Similarity still has a role, but a narrow one. It is good at finding candidates: cheaply surveying a large collection for items that might bear on the current passage. It cannot label them. A useful division of labour has similarity proposing, a relation step deciding what kind of thing each candidate is, and only then some further judgement about whether it deserves the reader’s attention.

find candidates   →   determine the relation   →   decide whether it is worth showing
(similarity can)      (similarity cannot)          (depends on the reader's task)

That ordering is also what keeps the stages improvable independently. Better candidate-finding without a relation step retrieves more near-duplicates faster. A relation step without any judgement of worth surfaces well-labelled irrelevance. And either, without care about timing, interrupts a reader with a contradiction at the wrong moment.

Why the label is hard to get right

None of this makes the labelling easy, and it would be dishonest to suggest otherwise. Relation labelling needs the same care as reading. Surface cues mislead in both directions. A negation word does not make a contradiction: “this does not hold for datasets under a thousand examples” may be a scope condition that agrees with the other paper. And a real contradiction can be phrased without any negation at all. Direction is easy to reverse. Scope, subject and time all change what a relation is.

Earlier work by the author, the RELATE project, explored typed relations between pieces of text, and this chapter draws on that motivation. It does not depend on that project’s design or results. The point stands without them: the label is a claim, claims can be wrong, and a wrong label does more damage than no label, because it comes with the authority of a structure. A system that calls two papers contradictory when they merely use different samples has manufactured a conflict. It is better to say “unknown” than to guess.

It also follows that any claim that typed relations improve discovery has to be tested against serious alternatives, not against raw similarity alone. Showing that a labelled approach beats keyword overlap shows only that labels help. Whether the particular vocabulary, or the particular method for assigning it, adds anything beyond a well-trained ordinary classifier is a separate and harder question, and this book has not answered it.

Where this leaves us

When a system brings one piece of information beside another, it is claiming a relationship, and similarity is the weakest and most ambiguous relationship it could claim. A vocabulary of relations (evidence, explanation, succession, practical use, resemblance) lets the interface say what kind of thing it is showing, and different relations support different uses. The relation becomes an explicit record with two identified items, a type, evidence and a standing, and “unknown” remains an honest value. Similarity finds candidates. It does not label them.

Now the system is transforming and augmenting information at every step: compressing it, rendering it new ways, adding material beside it, and attaching labelled relations. Each of these can quietly go wrong, and a fluent result gives no sign that it has. If the source said “may” and the rendering says “is”, or the relation reversed a direction, or a qualifier was dropped, nothing on the screen announces it. Transforming and augmenting information creates an obligation to measure what survived.

References

• The author’s RELATE project (github.com/ernanhughes/relate), cited as the origin of the typed-relation motivation only. Its results are not relied on here.