Summary:
Pull Request resolved: https://github.com/facebookresearch/mmf/pull/796

Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/143

Refer to Shir's diff D23716331

It is a subset of his diff, just the alignment part without RAG.

Run f225045496,

r@1_retrieval: 0.6310, r@5_retrieval: 0.8700, r@10_retrieval: 0.9250
r@1_rev_retrieval: 0.5022, r@5_rev_retrieval: 0.7870, r@10_rev_retrieval: 0.8632

Result reported in the paper
r@1_retrieval: 0.651, r@5_retrieval: 0.873, r@10_retrieval: 0.926
r@1_rev_retrieval: 0.506, r@5_rev_retrieval: 0.788, r@10_rev_retrieval: 0.867

Running shir's code on FAIR dev machine
r@1_retrieval: 0.6290, r@5_retrieval: 0.8620, r@10_retrieval: 0.9210,
r@1_rev_retrieval: 0.4944, r@5_rev_retrieval: 0.7852, r@10_rev_retrieval: 0.8620

Updated run
f253273762

Reviewed By: ytsheng, vedanuj, apsdehal

Differential Revision: D24321055

fbshipit-source-id: 238ce5a98d82fc05cb5f826df535ddfe612b43fe