The ranking of the submissions is based on the accuracy of pose estimates on the synthetic test set. This score is indicated on the leaderboard in the Best Score column.

We also compute score for submissions on real images of the Tango satellite mockup, to evaluate the transferability of the pose estimation model/algorithm. Even though we indicate this score on the leaderboard (the Real Image Score column), the submissions are ranked based by their score on the synthetic images only.

During the competition, submissions are evaluated on a subset of all test images, in order to avoid 'overfitting' on the test set. At the end of the competition we re-evaluate all submissions on the complete test set, and rank the teams according to this final score.

Pose score calculation

The pose score for one image is calculated in two steps: first we compute separate orientation and translation scores, then we combine these two.

The position score for image i is simply the 2-norm of the position error (difference of estimated and ground truth position vectors), normalized with the ground truth distance of the satellite:

$$ score_{position}^{(i)} = \frac{\left| r_{gt}^{(i)} - r_{est}^{(i)} \right|_{2}}{\left| r_{gt}^{(i)} \right|_{2}} $$

We define the orientation score as the angle of the rotation, that aligns the estimated and ground truth orientations:

$$ score_{orientation}^{(i)} = 2 \cdot arccos\left( \left| \left< q_{est}^{(i)}, q_{gt}^{(i)}\right> \right| \right) $$

The pose score for an image is the sum (1-norm) of the orientation and position scores: $$ score_{pose}^{(i)} = score_{orientation}^{(i)} + score_{position}^{(i)} $$

Finally, the total score is the average of pose scores over all images of the test set: $$ N = \text{number of images in test set} $$

$$ score = \frac 1N \sum^{N}_{i=1}score_{pose}^{(i)} $$