Evaluator

class openbiolink.evaluation.evaluation.Evaluator(dl, higher_is_better=True)

Parameters

• dl (DataLoader) – Dataloader containing the OpenBioLink dataset

• higher_is_better (bool) – Boolean which should be set to True if higher scores are considered better, False otherwise.

evaluate(batch_size=100)

Evaluates a model by retrieving scores from the (implemented) score_batch function.

Parameters

batch_size – Integer determining the size of the test batch which is passed to function score_batch

Return type

Dict[str, float]

Returns

Dictionary containing the evaluation results (keys: ‘hits@1’, ‘hits@3’, ‘hits@10’, ‘mrr’)

abstract score_batch(batch)

Abstract function, has to be implemented. Should return two arrays containing the head and tail scores of a batch of test data from a model.

Parameters

batch (Tensor) – Batch of test data. Shape (batch_size,3)

Return type

Tuple[Tensor, Tensor]

Returns

• head_scores: torch.tensor where the value at [i,j] is the score of the triple (j, batch[i][1], batch[i][2]). Shape (batch_size, num_entities)

• tail_scores: torch.tensor where the value at [i,j] is the score of the triple (batch[i][0], batch[i][1], j). Shape (batch_size, num_entities)