| d30dad3 | Aapo Kyrola | 30 April 2021, 20:26:35 UTC | [mmf/debuggability] longer timeout on first batch (#919) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/919 Timeout on first batch should be much longer, because workers might start at different times. Reviewed By: madian9 Differential Revision: D28122452 fbshipit-source-id: c076d80f8b75545f67ccc3656445ab159aaf3b46 | 30 April 2021, 20:27:15 UTC |
| 9d8d89c | Sasha Sheng | 30 April 2021, 16:58:26 UTC | [fix] pytorch lightning api change (#916) Summary: Recently, pytorch lightning internal in fbcode got updated to the recent master; My PR https://github.com/facebookresearch/mmf/issues/755 also updated OSS pytorch lightning dependency to master; This is to fix the original fix https://github.com/facebookresearch/mmf/issues/880 to use the most recent pytorch lightning api. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/916 Reviewed By: apsdehal Differential Revision: D28112996 Pulled By: ytsheng fbshipit-source-id: de462682ae0fcb98dd40c39ab5e61fe3dc77e338 | 30 April 2021, 16:59:16 UTC |
| 08f062e | Amanpreet Singh | 29 April 2021, 08:16:00 UTC | [feat] Adds audio (resnet18) and video (r2plus1d18) encoders (#879) Summary: This PR adds support for audio and video modality encoders to MMF. These can be used in conjunction with MMFTransformer. An example config has been added to showcase the usage. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/879 Test Plan: Unit tests have been added. Reviewed By: ytsheng Differential Revision: D27804875 Pulled By: apsdehal fbshipit-source-id: 9f276dab2dc711fb8e5868a029f73c16083c1782 | 29 April 2021, 08:16:39 UTC |
| 231fb16 | Sasha Sheng | 29 April 2021, 07:20:28 UTC | [feat] PL mvp1: validation + logging (#771) Summary: - [x] MVP 1. Evaluation and Logging - Eval accuracy parity between mmf_trainer and lightning_trainer - [X] Validation at the right frequency and at the end (also calculate the right metrics) - [X] Do logging (same logging format with slight difference is ok) - [x] Tests - tests only done for pytorch lightning integration - [X] test logging values are the same as `mmf_trainer`'s logging values - [X] test eval values are the same as `mmf_trainer`'s eval values - [x] test tensorboard logging the same as `mmf_trainer`'s - [ ] Blocked: - [ ] One Epoch Evaluation Run after train: [PR4945](https://github.com/PyTorchLightning/pytorch-lightning/pull/4945/files) and [PR4948](https://github.com/PyTorchLightning/pytorch-lightning/pull/4948/files) Pull Request resolved: https://github.com/facebookresearch/mmf/pull/771 Reviewed By: simran2905 Differential Revision: D26394468 Pulled By: ytsheng fbshipit-source-id: 3fdec30efe3890248cbbc65142f02df6354e3b44 | 29 April 2021, 07:21:25 UTC |
| c671214 | Aapo Kyrola | 28 April 2021, 01:28:31 UTC | [mmf] validate batch sizes are same + debugging improvements (#902) Summary: [feat] Pull Request resolved: https://github.com/facebookresearch/mmf/pull/902 Before adding batch to reporter, sync with all workers and validate all have same batch size. If not, skip the batch. Reports in the end how many were skipped. This should add quite negligible overhead. And once the problem itself is fixed (probably by constructing the dataloader with 'drop_incomplete' -- but i am not sure where to add this), this could be removed -- but perhaps better to just keep it. In addition: - remove master's logging to a file and instead log to stderr - add CompleteInTimeOrDie() so that we get better traces than NCCL deadlocks cause. - some tiny changes. Reviewed By: apsdehal Differential Revision: D27954629 fbshipit-source-id: cf000586772f4510b717153ec66e1f9c6c269ac3 | 28 April 2021, 01:29:36 UTC |
| c29b3cd | Xingjian Shi | 26 April 2021, 19:41:50 UTC | [docs] Suggestion to try turning off build isolation for installation (#888) Summary: Issue: https://github.com/facebookresearch/mmf/issues/881 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/888 Reviewed By: ytsheng Differential Revision: D27887967 Pulled By: apsdehal fbshipit-source-id: e8bd82660e438b3b552233545483f74d45934285 | 26 April 2021, 19:42:36 UTC |
| a4e9c22 | Vaibhav Singh | 26 April 2021, 19:40:19 UTC | [fix] Fixed drop_last setting and all_gather for xla device. (#872) Summary: fixed drop_last setting for xla aligned with length computation. fixed all gather (torch.stack only needed for GPU path.) Pull Request resolved: https://github.com/facebookresearch/mmf/pull/872 Reviewed By: ytsheng Differential Revision: D27798021 Pulled By: apsdehal fbshipit-source-id: 0a80ab0fe946f3885ade9220300f6df8417b6d21 | 26 April 2021, 19:40:56 UTC |
| 62ec6cd | Jun Chen | 21 April 2021, 01:09:24 UTC | [feat][MMF] Add multi-head loss Summary: * Add multi-head loss and multi-head MLP head in MMF * Fix fblearner output metrics in train workflow * Add multi-head metric and processor for new taxonomy in MMF Commerce Reviewed By: lichengunc Differential Revision: D27071740 fbshipit-source-id: c0771bdda57f577ce2cadcd05578497132b3336c | 21 April 2021, 01:10:14 UTC |
| 1a5c2b9 | Licheng Yu | 16 April 2021, 23:45:56 UTC | [mmf_ads][feat] Multi-task Learning (#882) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/882 We fixed a small bug in `mmf/modules/losses.py`. When there are multiple loss, the returned losses from MMFLoss is a dictionary. We need to unwrap its values for summing the total loss. Internally, we add multi-task learning for mmf_ads. Specifically, now it support two tasks: 1) megataxon classification 2) img cluster classification Reviewed By: apsdehal Differential Revision: D27730656 fbshipit-source-id: bd78c8af49d7cc0a9286ee0d7dfc48ff9bade3d7 | 16 April 2021, 23:46:36 UTC |
| 5bb6f50 | Amanpreet Singh | 16 April 2021, 07:50:29 UTC | [fix] regression in tests for lightning with recent change (#880) Summary: Regression caused by https://github.com/facebookresearch/mmf/commit/868c5892cf7607cdeca3df386fa7810ca1ba9b67 Fixes https://github.com/facebookresearch/mmf/issues/876 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/880 Test Plan: pytest Reviewed By: BruceChaun Differential Revision: D27812170 Pulled By: apsdehal fbshipit-source-id: 09f1fcf9d7297c5b02bfb984ffcd330b3ebdc5e0 | 16 April 2021, 07:51:11 UTC |
| 3947693 | Vedanuj Goswami | 14 April 2021, 22:02:44 UTC | [feat] Adding movie mcan e2e model to model zoo (#867) Summary: Movie MCAN e2e model for zoo VQA2.0 TestDev Acc : 73.6 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/867 Reviewed By: ytsheng Differential Revision: D27693464 Pulled By: vedanuj fbshipit-source-id: 7e435176d0330d126ce705f91ca4277644ee7f3c | 14 April 2021, 22:03:45 UTC |
| 8855495 | Amanpreet Singh | 14 April 2021, 01:45:36 UTC | [feat,fix] CPU evaluation and report memory leak (#846) Summary: This PR has three purposes: - Allow CPU evaluation for the cases where evaluation set is big can OOM on calculation for global metrics. Enable this via evaluation.use_cpu - Fix a memory leak which is caused as first report in training loop is not detached while generating a new one. - It also fixes an issues with shallow report copy made prediction report purposes. This would cause issues in multiple metrics calculation where some metrics use prediction report and some don't. Since, it is shallow copy, first report copy shares tensors with original report which when gathered in reporter would lead to average number from all GPUs and hence that report would like into original report. This is fixed by creating a deep copy of report rather than shallow copy by introducing a copy method The PR also fixes - the incorrect config settings for mmf transformer on VQA2.0 - relaunching of a run using cmd api was adding multiple callbacks to trainer as callbacks are a class variable. This PR also reset callbacks in each setup so that multiple logs or checkpoints are seen. - remove lru caching of logging functionality as they are only called once and cause issue with caching when called in cmd api - add evaluation configuration options to trainer configs in tests Pull Request resolved: https://github.com/facebookresearch/mmf/pull/846 Test Plan: Tested with UniT, VisualBERT on Hateful Memes, MMFT on VQA2. Added extensive tests as well. Reviewed By: vedanuj, lichengunc Differential Revision: D27496441 Pulled By: apsdehal fbshipit-source-id: 3fc3243117a2068228b0a3626a029b75c5cc7d00 | 14 April 2021, 01:46:33 UTC |
| 868c589 | Ananth Subramaniam | 14 April 2021, 00:51:52 UTC | Synchronize PyTorchLightning/pytorch-lightning (revision aaccbeea@master) to github/third-party/PyTorchLightning/pytorch-lightning Reviewed By: shuyingsunshine21 Differential Revision: D27674288 fbshipit-source-id: 21af95adf952d632d10289a45fba6dd7cba6b2d9 | 14 April 2021, 00:52:48 UTC |
| 401d337 | Sasha Sheng | 13 April 2021, 02:59:39 UTC | [feat][3/N] clip text processor (#164) Summary: * Added text processor Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/164 Reviewed By: apsdehal Differential Revision: D27457434 Pulled By: ytsheng fbshipit-source-id: 85116028df1866bf0021157df6be9e8ecd69f021 | 13 April 2021, 03:00:41 UTC |
| f024b7b | Sasha Sheng | 08 April 2021, 19:39:46 UTC | [test] lightning trainer to use the new datamodule api (#860) Summary: * onboard the lightning trainer to use the new datamodule api * to test: run: - `pytest tests` Pull Request resolved: https://github.com/facebookresearch/mmf/pull/860 Reviewed By: apsdehal Differential Revision: D27617952 Pulled By: ytsheng fbshipit-source-id: 8881ed40840b85049735bf1fdf33a3f57b0d1987 | 08 April 2021, 19:40:56 UTC |
| b093095 | dependabot[bot] | 07 April 2021, 23:01:49 UTC | Bump y18n from 3.2.1 to 3.2.2 in /website (#845) Summary: Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2. <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/yargs/y18n/commits">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~oss-bot">oss-bot</a>, a new releaser for y18n since your current version.</p> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/facebookresearch/mmf/network/alerts). </details> Pull Request resolved: https://github.com/facebookresearch/mmf/pull/845 Reviewed By: ytsheng, apsdehal Differential Revision: D27495488 Pulled By: vedanuj fbshipit-source-id: b9ea33ea2e6d23d9f2c877860da5f5ce4c952516 | 07 April 2021, 23:02:47 UTC |
| 66a988a | Brett Zachary Allen | 06 April 2021, 21:44:54 UTC | [fix] Download functionality work for frcnn (#858) Summary: Fixed the downloading functionality for the FRCNN feature extraction script. This resolves the TODO of using the mmf download functionality to download the configs and models. Also updated the script to directly use the huggingface urls Resolves: https://github.com/facebookresearch/mmf/issues/836 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/858 Reviewed By: vedanuj Differential Revision: D27571075 Pulled By: brettallenyo fbshipit-source-id: ab7ff27fd2b6156946c26c619a7e67b440d8af11 | 06 April 2021, 21:45:32 UTC |
| 332f60e | Sasha Sheng | 06 April 2021, 10:58:18 UTC | [feat][2/N] clip text/image encoder (#163) Summary: * Added text/image encoders * Added the pretrained checkpoint loading for those encoders We should make it composable in the future - ie., make the encoders configurable with params Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/163 Reviewed By: vedanuj Differential Revision: D27447444 Pulled By: ytsheng fbshipit-source-id: e2de1752df497cbbe9f689ad08b482565d883e29 | 06 April 2021, 10:59:06 UTC |
| d9812a6 | Sasha Sheng | 06 April 2021, 10:52:29 UTC | [fix] test error when using the new datamodule api (#855) Summary: * Updated the TrainerTrainingLoop object to use the new datamodules API + iteration_strategy API * Fixed bug related to API change for the per_gpu_batch_size test * Fixed the test_eval test to use the new API * Fixed the default iteration strategy (constant) to use `should_exhaust_all_iterators`=True. This is when multi-task condition is false. This was the default behavior before the iteration strategy API update. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/855 Test Plan: run `pytest tests` Reviewed By: vedanuj Differential Revision: D27562857 Pulled By: ytsheng fbshipit-source-id: a9baea12b046ea2affd5f5f205224131941a40ab | 06 April 2021, 10:53:48 UTC |
| 8be4a2f | Vedanuj Goswami | 05 April 2021, 23:24:08 UTC | [chores] Upgrade to support Pytorch 1.8 (#849) Summary: Upgrading max supported version to PT 1.8.1 and torchvision 0.9.1 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/849 Reviewed By: ytsheng Differential Revision: D27543579 Pulled By: vedanuj fbshipit-source-id: 603199331a3bbf91b99ff110a1145ab3e164ba02 | 05 April 2021, 23:24:57 UTC |
| 07680c4 | Brett Zachary Allen | 05 April 2021, 22:16:51 UTC | [fix] FRCNN Feature extraction for single image (#850) Summary: Fixed single image bug that had incorrect function arguments for extracting features on a single image. Fix: https://github.com/facebookresearch/mmf/issues/848 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/850 Reviewed By: vedanuj Differential Revision: D27547598 Pulled By: brettallenyo fbshipit-source-id: 7bac9aff5521c684a048441cfee619ecb6b84b5e | 05 April 2021, 22:17:34 UTC |
| 9a7b75c | Brett Zachary Allen | 05 April 2021, 21:14:48 UTC | [feat] Inference CLI (#830) Summary: Created script to allow users to run inference using various models, text inputs, and image inputs. All that is required is using mmf_inference and passing some command line args. Here is how you use it: ``` mmf_interactive checkpoint_path=/checkpoint/brettallen/mmft/model ``` This will initiate an interactive inference script that will ask for image URL and text input to run on. It will keep prompting the user for more images and text until they say `exit`. I also added the ability to just say `same` to use the same image because I figured it was inconvenient for the user to constantly have to copy over the image URL every time they want to ask a new question. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/830 Reviewed By: apsdehal Differential Revision: D27339291 Pulled By: brettallenyo fbshipit-source-id: db190c40626229900a3530da75b61d6e6b63cdd0 | 05 April 2021, 21:15:34 UTC |
| 99b1f74 | Sasha Sheng | 05 April 2021, 00:31:02 UTC | [refactor] reporting + meter (#844) Summary: * Clean the API up to prep for moving some of the report/meter updating inside the base_model for pytorch lightning early stopping/checkpointing. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/844 Reviewed By: vedanuj Differential Revision: D27486844 Pulled By: ytsheng fbshipit-source-id: 759fbaccdc1ce2ef2a8e736fa179afeec48b89dc | 05 April 2021, 00:31:50 UTC |
| 2379d0c | Ronghang Hu | 03 April 2021, 08:52:39 UTC | [fixup] add update_registry_for_model to datamodules (#851) Summary: Call `update_registry_for_model` to datamodules to register e.g. answer num. Fix https://github.com/facebookresearch/mmf/issues/847. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/851 Test Plan: tested locally with M4C and M4C-Captioner Reviewed By: apsdehal Differential Revision: D27553712 Pulled By: ronghanghu fbshipit-source-id: 7e5a4092658e8240ad615b0601cdf40d393453fb | 03 April 2021, 08:53:24 UTC |
| 7828606 | Erjia Guan | 02 April 2021, 22:19:06 UTC | Change MagicMock from return_type to spec Summary: Unblock OSS PyTorch Diff D27327232 ``` class Dataset: pass class IterableDataset(Dataset): pass class NumbersDataset(Dataset): pass d = NumbersDataset() # Prev foo = MagicMock(return_type=c) isinstance(foo, Dataset) # False isinstance(foo, IterableDataset) # False # Now foo = MagicMock(spec=c) isinstance(foo, Dataset) # True isinstance(foo, IterableDataset) # False ``` Reviewed By: ytsheng, vedanuj, apsdehal Differential Revision: D27529078 fbshipit-source-id: 0c6fec0a7379d58dbcd6c4583dc3aed1482f28ab | 02 April 2021, 22:19:56 UTC |
| 9d7696f | Sasha Sheng | 02 April 2021, 10:32:52 UTC | [fix] Use caption only files for localized narratives (#839) Summary: vedanuj. This PR uses smaller caption files. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/839 Reviewed By: vedanuj Differential Revision: D27424834 Pulled By: ytsheng fbshipit-source-id: 3ff4a252fffa700de9e03e880dfa8cf7fadd321b | 02 April 2021, 10:33:33 UTC |
| 6f89879 | Sasha Sheng | 02 April 2021, 08:58:43 UTC | [feat] add tests for eval loop (#835) Summary: * Adds test for evaluation loop for mmf trainer * Makes it so that test folder quality is more explicit in terms of errors. I wrote this test to understand deeper on how the evaluation_loop.py works. This needs to land before the next pytorch lightning update. To test that it works run: `pytest tests` Pull Request resolved: https://github.com/facebookresearch/mmf/pull/835 Reviewed By: vedanuj Differential Revision: D27405061 Pulled By: ytsheng fbshipit-source-id: c1c27fe824ac282d387ba37bb726cdb562f428ef | 02 April 2021, 08:59:58 UTC |
| 71554e3 | Sasha Sheng | 02 April 2021, 08:56:52 UTC | [test] reduce mem leak in tests (#837) Summary: * Fix memory leak in pytest Pull Request resolved: https://github.com/facebookresearch/mmf/pull/837 Test Plan: run `pytest tests` Reviewed By: apsdehal Differential Revision: D27421597 Pulled By: ytsheng fbshipit-source-id: 5edf023ef46fc77c82d0924c9ceb48624fd8c4e3 | 02 April 2021, 08:57:45 UTC |
| 10e3f45 | Ronghang Hu | 02 April 2021, 06:19:02 UTC | [feat] add model checkpoints on UniT w/o task embeddings (#831) Summary: Add new pretrained checkpoints of UniT w/o task embedding, which can be evaluated as follows: ``` # on a 8-GPU machine python mmf_cli/run.py \ config=projects/unit/configs/all_8_datasets/shared_dec_without_task_embed.yaml \ datasets=detection_coco,detection_visual_genome,vqa2,visual_entailment,glue_qnli,glue_sst2,glue_mnli_mismatched,glue_qqp \ model=unit \ run_type=val \ training.batch_size=8 \ checkpoint.resume_zoo=unit.all_8_datasets.shared_dec_with_coco_init_without_task_embedding ``` Results: ``` val/detection_coco/detection_mean_ap: 0.3847, val/detection_visual_genome/detection_mean_ap: 0.0331, val/vqa2/vqa_accuracy: 0.6825, val/visual_entailment/accuracy: 0.7407, val/glue_qnli/accuracy: 0.8815, val/glue_sst2/accuracy: 0.8850, val/glue_mnli_mismatched/accuracy: 0.8061, val/glue_qqp/accuracy: 0.9061 ``` Also added pretrained COCO single-task checkpoints to perform COCO initialization experiments. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/831 Test Plan: Tested locally with UniT model Reviewed By: vedanuj Differential Revision: D27528247 Pulled By: ronghanghu fbshipit-source-id: d7e4c8ec337f502b58f4c71f4b29fb6b996df03d | 02 April 2021, 06:19:49 UTC |
| f3d6f6c | Licheng Yu | 31 March 2021, 22:33:14 UTC | [mmf_ads] models trained on ads+pages (#843) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/843 MT image+text tables: - `ads_and_pages_for_xlm_processed_v5_with_xray_hash_feature:aml` - `ads_and_pages_for_xlm_processed_v5_with_xray_2048_feature:aml` with same splits as `ads_and_pages_for_xlm_processed_v5:ad_delivery`. We fix tokenizer bug in `bert_processors.py`, where the special tokens for Roberta are `<s>`, `</s>`, `<mask>`, etc, instead of Bert's `[CLS]`, `[SEP]`, `[MASK]`, etc. Besides, the padding index for roberta and bert is 1 and 0 respectively. Add `ads+pages` training and MMFT (hugginface, pytext) supports. Reviewed By: ytsheng Differential Revision: D26440611 fbshipit-source-id: 21e97162aa9f5b2363d2c68a71a20f185b7a8db6 | 31 March 2021, 22:34:09 UTC |
| 6395711 | Amanpreet Singh | 31 March 2021, 20:47:25 UTC | [fix] Reallow num_workers and pin_memory overrides from training config (#842) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/842 Recent changes for datamodules support moved the options num_workers and pin_memory along with others inside the dataset_config. This diff fixes backwards compatibility and allows specifying the num_workers and pin_memory from training config. Reviewed By: ytsheng, vedanuj, rayhou0710 Differential Revision: D27421166 fbshipit-source-id: 1fff516dcbf6a0f8bb5ab81ca6c477b53b0e4937 | 31 March 2021, 20:48:30 UTC |
| 03a0527 | Sasha Sheng | 30 March 2021, 22:24:22 UTC | [feat][1/N]: clip contrastive loss (#162) Summary: Add Contrastive Loss Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/162 Reviewed By: vedanuj Differential Revision: D27389621 Pulled By: ytsheng fbshipit-source-id: 60424e6c4cd568bdbef55d8f49cb729dc078af7b | 30 March 2021, 22:25:17 UTC |
| 29f57ea | Amanpreet Singh | 30 March 2021, 07:54:53 UTC | [refactor] Move test_reporter_config to evaluation.reporter (#838) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/838 Test reporter config as opposed to original design should be a top level config node as it is not specific to one dataset but encompasses all datasets in one go. This diff refactors that to be a top level config node so that it is not per dataset config which conflicts with overall definition of test reporter. Reviewed By: madian9 Differential Revision: D27301920 fbshipit-source-id: 6a990cd23c837fb915398a5930b9e348ff4cd626 | 30 March 2021, 07:55:36 UTC |
| a094ad7 | Amanpreet Singh | 30 March 2021, 03:39:45 UTC | [feat] Iteration strategies for multitasking (#822) Summary: This PR adds support for iteration strategies that can be used with multitasking. IterationStrategies as a concept just returns next dataset index to be sampled in each turn. The diff implements multiple of them and makes some sensible assumptions on the defaults. - Always fallback to size proportional sampling for val and test sets - Epoch based strategy only allowed in roundrobin, constant and size proportional. - Adds constant, roundrobin, constant, size proportional and ratios schemes. - constant strategy for single task runs Pull Request resolved: https://github.com/facebookresearch/mmf/pull/822 Reviewed By: vedanuj Differential Revision: D27211249 fbshipit-source-id: bce14a44730bdd830a72f4e72c350211e088c8e8 | 30 March 2021, 03:40:32 UTC |
| 542694f | Amanpreet Singh | 29 March 2021, 17:03:21 UTC | [feat] Add teardown support for datamodules (#834) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/834 Without proper teardown, the job might fail as errored and restart. This diff adds proper teardown support for datamodules which inherently will include onbox dataloader as well. Reviewed By: ytsheng, vedanuj Differential Revision: D26687357 fbshipit-source-id: 8cebc7f49055cea6906cb367b539fc4073666493 | 29 March 2021, 17:04:37 UTC |
| 12650f5 | Amanpreet Singh | 27 March 2021, 09:01:26 UTC | [feat,refactor] Support datamodules directly in dataset loader (#782) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/782 This diff refactors MMF to support datamodules directly. Reviewed By: ytsheng, vedanuj Differential Revision: D26481038 fbshipit-source-id: 6a3e36fe50a30c44f473098152e604962c2f4f8e | 27 March 2021, 09:02:20 UTC |
| 2d29089 | Jun Chen | 26 March 2021, 22:17:52 UTC | [MMF][feat] Add label smoothing cross entropy loss Summary: Add label smoothing cross entropy in loss function Differential Revision: D27092731 fbshipit-source-id: 66b6cf2bf04a2324b7d6f36d90f4a1ebb1243a35 | 26 March 2021, 22:18:55 UTC |
| e12e03d | Amanpreet Singh | 26 March 2021, 17:49:33 UTC | [fix] QoL changes to support onbox (#155) Summary: Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/155 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/827 Some of the original monkey patching doesn't work with onbox dataloader inside processors. Also, sets some env variables to support distributed onbox setting. Reviewed By: ytsheng, vedanuj Differential Revision: D27306251 fbshipit-source-id: 135b320cf3a585413a8902ee6cd0f67a58ec64d7 | 26 March 2021, 17:51:02 UTC |
| 5c5c356 | Amanpreet Singh | 26 March 2021, 17:34:50 UTC | [chores] Add hateful memes config for MMF Transformer (#828) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/828 This is being used in our getting started example but is missing, so adding. Reviewed By: ytsheng, vedanuj Differential Revision: D27306253 fbshipit-source-id: 40407235e0c6feddc556efeb5af4ecda916339a8 | 26 March 2021, 17:35:52 UTC |
| e7ad822 | Shaoliang Nie | 25 March 2021, 22:49:15 UTC | [feat] Hateful Memes fine grained (#156) Summary: Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/156 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/829 hateful memes fine grained Reviewed By: ytsheng, apsdehal Differential Revision: D26490349 fbshipit-source-id: 670698aefd13fa0fae4d61174fe669e86ac5f7fe | 25 March 2021, 22:50:12 UTC |
| 3f1bcf4 | Amanpreet Singh | 25 March 2021, 05:11:07 UTC | [refactor] Collator samplelist conversion logic for more control (#826) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/826 This refactor allows us to use sample list conversion logic used in BatchCollator elsewhere as well. Reviewed By: vedanuj Differential Revision: D27306252 fbshipit-source-id: a9a89388e760d0b23c729f3ac43399df20fc7088 | 25 March 2021, 05:11:56 UTC |
| e88d059 | Madian Khabsa | 25 March 2021, 03:35:36 UTC | [postray] Add fblearner flows (#153) Summary: Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/153 - Create flows for postray to avoid remaining dependant on faim flows - Extend path finder to search for user dir Reviewed By: rayhou0710 Differential Revision: D27040702 fbshipit-source-id: c1b166cb3b8775f25879cbe405435550478b81b7 | 25 March 2021, 03:36:40 UTC |
| 9588752 | Amanpreet Singh | 25 March 2021, 03:28:33 UTC | [feat] Inherit from PL datamodule for dataset builders (#825) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/825 After this change dataset builder will inherit from PL datamodule as at the high level both concepts are same. BaseDatasetBuilder has been adjusted to also support datamodule functionality directly. Reviewed By: ytsheng, vedanuj Differential Revision: D26505873 fbshipit-source-id: f3e1393f65bf25a2ec7161a5ef2e450c98a192e0 | 25 March 2021, 03:29:25 UTC |
| 19f3211 | Sasha Sheng | 24 March 2021, 22:07:02 UTC | [feat] add images to localized narratives (#823) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/823 Reviewed By: vedanuj Differential Revision: D27306005 Pulled By: ytsheng fbshipit-source-id: 1626f2e8762138744130fdc343d3261095293100 | 24 March 2021, 22:07:58 UTC |
| 7699ff0 | Brett Zachary Allen | 24 March 2021, 20:17:08 UTC | [feat] Inference API (#798) Summary: Added an API to run inference in these use cases: - region features -> mmft - region features -> visual_bert Based on configs, some are provided here, users can run inference on various trained models with something as simple as an image URL and a text query. Would love for people to take a look and provide feedback about my coding style and design choices. If you want to run it on your own you can pull and run ``` from mmf.utils.inference import Inference if __name__ == "__main__": inference = Inference(checkpoint_path="/checkpoint/brettallen/visualbert") answer = inference.forward( "http://i.imgur.com/1IWZX69.jpg", {"text": "what type of shoes is the woman in pink wearing"}, image_format="url", ) print(answer) ``` Pull Request resolved: https://github.com/facebookresearch/mmf/pull/798 Reviewed By: vedanuj Differential Revision: D27171794 Pulled By: brettallenyo fbshipit-source-id: 25b51e5da8cdae944b53970ce1a30b253dee339f | 24 March 2021, 20:18:15 UTC |
| 800a086 | Madian Khabsa | 24 March 2021, 03:21:27 UTC | <feat> Refactor MMF Test Reporter and add Hive Exporter (#150) Summary: Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/150 - Modifies test reporter to be fetched/registered in registry - Add hive test reporter for internal support Reviewed By: apsdehal Differential Revision: D26954690 fbshipit-source-id: b8ad088d829a5353c713cb493c38f27099b43fcd | 24 March 2021, 03:23:02 UTC |
| e5edc48 | Vedanuj Goswami | 22 March 2021, 22:58:41 UTC | [fix] Fix iopath get_local_path related checkpoint issues (#817) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/817 Adds force=True for downloading manifold checkpoints so that we do not load cahced versions of the checkpoints like best.ckpt or current.ckpt Reviewed By: apsdehal Differential Revision: D27130415 fbshipit-source-id: 21546b6be43cf87bc37ecb0842fc6321b678bcbf | 22 March 2021, 23:00:03 UTC |
| b752db6 | Vedanuj Goswami | 19 March 2021, 23:50:52 UTC | [fix] Do not run validation if final val not in run_type (#816) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/816 Adding a check to final validation to run only if val is in run_type Reviewed By: ytsheng, apsdehal Differential Revision: D27130474 fbshipit-source-id: 2793a1dd3f77d4d56d737781a68973634d271157 | 19 March 2021, 23:52:20 UTC |
| 62a50f1 | Sasha Sheng | 19 March 2021, 22:29:23 UTC | [fix] tests (pytest + black) (#149) Summary: skip fb tasks on devfair pytest Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/149 Reviewed By: apsdehal Differential Revision: D27185785 Pulled By: ytsheng fbshipit-source-id: 1e88b76607fbf86a67a87ea601ce8a2c35197499 | 19 March 2021, 22:31:00 UTC |
| 3ba4725 | Amanpreet Singh | 19 March 2021, 22:17:27 UTC | [fix] length for multi dataset loader; max_epochs; eta (#820) Summary: The calculation of length for multidataloader used get_batch_size to divide total dataset length which is incorrect as get_batch_size is per device rather than global batch size. This PR fixes it and fixes snapshot interval calculation as well. This in turn: - Fixes max_epochs support - ETA - All other things dependent on this Pull Request resolved: https://github.com/facebookresearch/mmf/pull/820 Test Plan: Tested locally with MVLT runs Reviewed By: ytsheng, madian9 Differential Revision: D27184804 Pulled By: apsdehal fbshipit-source-id: d18aeb52b7e280f33c3b9e7509dafa123ed1c629 | 19 March 2021, 22:18:53 UTC |
| 08fb370 | Amanpreet Singh | 18 March 2021, 22:58:39 UTC | [refactor] Fix typings used from mmf common/typings Summary: Currently, typings used in multiple places in MMF from common/typings are incorrect. This diff fixes them. Reviewed By: ytsheng Differential Revision: D27127754 fbshipit-source-id: fccbb5df0a7ffac2f12e73a9d47e369fb6b7a4d0 | 18 March 2021, 23:00:10 UTC |
| 7fb6bdc | Amanpreet Singh | 18 March 2021, 21:24:54 UTC | [feat] Allow returning sequences in transformer encoder (#814) Summary: - For some cases such as MLM we need full sequences. It would be great if we can reuse existing transformer encoder and support that. This PR adds support for the same along with tests Pull Request resolved: https://github.com/facebookresearch/mmf/pull/814 Test Plan: Tests have been added. Tested also with MVLT Reviewed By: ytsheng, vedanuj Differential Revision: D27075345 Pulled By: apsdehal fbshipit-source-id: c8d475405d4950494831846f2eef795975e5ff6c | 18 March 2021, 21:31:54 UTC |
| bbc8035 | Amanpreet Singh | 18 March 2021, 20:15:41 UTC | [feat,fix] Make some transformer head params optional; label key for MLM (#813) Summary: This PR makes some QoL changes for transformer heads so that they can be used at places other than MMFTransformer. - Make encoder_layers and processed_sample_list options optional by default - Add label key option for mlm to allow direct fetching of maskedlabels from sample list Pull Request resolved: https://github.com/facebookresearch/mmf/pull/813 Test Plan: Tested with GUniT Reviewed By: ytsheng, vedanuj Differential Revision: D27062330 Pulled By: apsdehal fbshipit-source-id: 3f7aed573b65154960a18dffb4ca6600c0d4bb34 | 18 March 2021, 20:17:11 UTC |
| 8f9150b | Amanpreet Singh | 18 March 2021, 00:05:31 UTC | [fix] Allow running slurm scripts from directory other than mmf (#818) Summary: - Currently, hardcoded to be run only from the same directory as mmf - This PR fixes it so that it can be launched from anywhere Pull Request resolved: https://github.com/facebookresearch/mmf/pull/818 Test Plan: Tested with MVLT Reviewed By: ytsheng Differential Revision: D27136433 Pulled By: apsdehal fbshipit-source-id: cfa0cb2696a7547c1a299ee9790eaa8fd9dbd307 | 18 March 2021, 00:06:58 UTC |
| 7cc0e4e | Ananth Subramaniam | 17 March 2021, 17:47:46 UTC | [mmf] Update gradient clipping test to use almost equals for float comparison (#147) Summary: Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/147 Reviewed By: ytsheng Differential Revision: D26783077 fbshipit-source-id: 65c278e68d25736ed89c9a1669aa5f401e0548f6 | 17 March 2021, 17:49:54 UTC |
| c672d08 | Ronghang Hu | 17 March 2021, 01:30:26 UTC | [feat] add the UniT model and supporting utils (#741) Summary: This PR is built upon https://github.com/facebookresearch/mmf/pull/740 - add the UniT model - add `AdamWSkipParamsWithZeroGrad` optimizer (skips unused params) - provide dataset zoo download for VQAv2 splits w.r.t. COCO 2017. - provide model zoo download for UniT. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/741 Test Plan: Tested locally and verified the outputs --- Evaluating the UniT model zoo entry ``` # on a 8-GPU machine python mmf_cli/run.py \ config=projects/unit/configs/all_8_datasets/shared_dec.yaml \ datasets=detection_coco,detection_visual_genome,vqa2,visual_entailment,glue_qnli,glue_sst2,glue_mnli_mismatched,glue_qqp \ model=unit \ run_type=val \ training.batch_size=8 \ checkpoint.resume_zoo=unit.defaults ``` Evaluation outputs on the 8-dataset model: ``` val/detection_coco/detection_mean_ap: 0.3899, val/detection_visual_genome/detection_mean_ap: 0.0329, val/vqa2/vqa_accuracy: 0.6696, val/visual_entailment/accuracy: 0.7316, val/glue_qnli/accuracy: 0.8790, val/glue_sst2/accuracy: 0.8922, val/glue_mnli_mismatched/accuracy: 0.8090, val/glue_qqp/accuracy: 0.9065 ``` Reviewed By: vedanuj, apsdehal Differential Revision: D26894259 Pulled By: ronghanghu fbshipit-source-id: b548313f724d61c2c6e38c2cba1ef19cd735dfa9 | 17 March 2021, 01:31:59 UTC |
| f2e45f7 | Amanpreet Singh | 12 March 2021, 00:17:39 UTC | [fix,feat] Support more wider variety of user dirs (#809) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/809 This PR handles some edge cases around user dirs which weren't handle before especially in cases where python files are compiled and moved to a different place. This different place can also have compile time generated modules which are not available directly to the original folder. The diff also handles support for specifying a python file as user dir input. Lastly, this diff adds extensive test to cover and test various user dir configurations. Reviewed By: ytsheng, vedanuj Differential Revision: D26838780 fbshipit-source-id: 38e4ffaf55053df0b29ff682b26bb4ae5b6638ac | 12 March 2021, 00:18:24 UTC |
| 60c9138 | vedanuj | 11 March 2021, 11:40:07 UTC | [fix] Fix memory leak in tests (#806) Summary: Fixing some memory leak in tests. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/806 Test Plan: Multiple re-runs of tests show no failure. Reviewed By: ytsheng, apsdehal Differential Revision: D26952871 Pulled By: vedanuj fbshipit-source-id: 0dc91b5f497dd234b25671c105afce788797dcb3 | 11 March 2021, 11:41:54 UTC |
| 6210494 | Madian Khabsa | 11 March 2021, 03:13:27 UTC | <fix> support providing configs that contain equal character (#808) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/808 The current parser would fail if the opt value has `=` which is needed when providing filters. This diff takes the first occurrence of `=` as the separator Reviewed By: apsdehal Differential Revision: D26955390 fbshipit-source-id: cedfb358de254a0c6d2a88744a0660984b045f47 | 11 March 2021, 03:15:10 UTC |
| c78a408 | Vedanuj Goswami | 10 March 2021, 23:41:25 UTC | [fix] Fix MMFT for loading older checkpoints, add tests and assertions (#805) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/805 Fix MMFT for loading older checkpoints after head module refactor. Also adds a test to catch these errors in future. Reviewed By: ytsheng Differential Revision: D26924574 fbshipit-source-id: 09fa22b1b922a6b3d1698f0fe545d826a1bfb5c6 | 10 March 2021, 23:42:54 UTC |
| 9a93bc4 | dependabot[bot] | 10 March 2021, 02:44:11 UTC | Bump elliptic from 6.5.3 to 6.5.4 in /website (#800) Summary: Bumps [elliptic](https://github.com/indutny/elliptic) from 6.5.3 to 6.5.4. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/indutny/elliptic/commit/43ac7f230069bd1575e1e4a58394a512303ba803"><code>43ac7f2</code></a> 6.5.4</li> <li><a href="https://github.com/indutny/elliptic/commit/f4bc72be11b0a508fb790f445c43534307c9255b"><code>f4bc72b</code></a> package: bump deps</li> <li><a href="https://github.com/indutny/elliptic/commit/441b7428b0e8f6636c42118ad2aaa186d3c34c3f"><code>441b742</code></a> ec: validate that a point before deriving keys</li> <li><a href="https://github.com/indutny/elliptic/commit/e71b2d9359c5fe9437fbf46f1f05096de447de57"><code>e71b2d9</code></a> lib: relint using eslint</li> <li><a href="https://github.com/indutny/elliptic/commit/8421a01aa3ff789c79f91eaf8845558a7be2b9fa"><code>8421a01</code></a> build(deps): bump elliptic from 6.4.1 to 6.5.3 (<a href="https://github-redirect.dependabot.com/indutny/elliptic/issues/231">https://github.com/facebookresearch/mmf/issues/231</a>)</li> <li>See full diff in <a href="https://github.com/indutny/elliptic/compare/v6.5.3...v6.5.4">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/facebookresearch/mmf/network/alerts). </details> Pull Request resolved: https://github.com/facebookresearch/mmf/pull/800 Reviewed By: vedanuj Differential Revision: D26890681 Pulled By: apsdehal fbshipit-source-id: 063737b8ef957f3e67edcc749b83660bd386a01b | 10 March 2021, 02:45:46 UTC |
| ff40099 | Amanpreet Singh | 09 March 2021, 18:47:16 UTC | [feat] First e2e test with emphasis on user dir (#609) Summary: This PR introduces first e2e test which uses `mmf_cli.run` programmatic API and log parsing to check for log file. Specifically in this PR we test for user_dir import by creating our own simple model and a dataset that always generate one. For a more friendly `user_dir` support, now we override default config in configuration with user_cmd_opts as well so that MMF can discover configs relative to user dir if needed. We run the usual run command and expect the model to overfit and always return 1. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/609 Test Plan: Tested locally on devfair using `pytest -v ./tests`. Reviewed By: vedanuj Differential Revision: D24130662 Pulled By: apsdehal fbshipit-source-id: cde12cc5f8cdbb317797adf374f9cd1c4637d4ab | 09 March 2021, 18:48:56 UTC |
| 413510f | Ronghang Hu | 08 March 2021, 19:17:48 UTC | [feat] detection datasets (COCO, Visual Genome) and mAP metric (#740) Summary: This PR is built upon https://github.com/facebookresearch/mmf/pull/739 - add `detection_coco` dataset - add `detection_visual_genome` datasets - add supporting detection utilities - add `detection_mean_ap` metric - add `DETRImageAndTargetProcessor` to load DETR image and box transforms Pull Request resolved: https://github.com/facebookresearch/mmf/pull/740 Test Plan: Tested locally and verified the outputs Reviewed By: vedanuj, apsdehal Differential Revision: D26765951 Pulled By: ronghanghu fbshipit-source-id: c908e4764656158c688f210c3c0a62add7977417 | 08 March 2021, 19:19:32 UTC |
| 82a158e | Rui Hou | 06 March 2021, 23:31:46 UTC | [feat] Make converting avgpool to Identity optional (#799) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/799 **Create a new diff to fix export to GitHub issue in D26781527** This diff makes convert avgpooling to Identity optional such that an image can be encoded as single embedding to be used as a token instead of feature maps for multiple tokens. Reviewed By: madian9 Differential Revision: D26870914 fbshipit-source-id: 07fc8228a203ec1a532b521df53d005c0e61589b | 06 March 2021, 23:33:36 UTC |
| 788e810 | Sasha Sheng | 06 March 2021, 10:00:52 UTC | [fix] pytorch 1.8 updates with rfft and irfft (#797) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/797 Reviewed By: vedanuj Differential Revision: D26840004 Pulled By: ytsheng fbshipit-source-id: 15ff847a5a0b13537d24fdddcb5e5d33166e6eb7 | 06 March 2021, 10:02:46 UTC |
| 853ac08 | Ning Zhang | 06 March 2021, 00:51:10 UTC | [feat] Image text alignment/retrieval model+dataset on flickr30k (#796) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/796 Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/143 Refer to Shir's diff D23716331 It is a subset of his diff, just the alignment part without RAG. Run f225045496, r@1_retrieval: 0.6310, r@5_retrieval: 0.8700, r@10_retrieval: 0.9250 r@1_rev_retrieval: 0.5022, r@5_rev_retrieval: 0.7870, r@10_rev_retrieval: 0.8632 Result reported in the paper r@1_retrieval: 0.651, r@5_retrieval: 0.873, r@10_retrieval: 0.926 r@1_rev_retrieval: 0.506, r@5_rev_retrieval: 0.788, r@10_rev_retrieval: 0.867 Running shir's code on FAIR dev machine r@1_retrieval: 0.6290, r@5_retrieval: 0.8620, r@10_retrieval: 0.9210, r@1_rev_retrieval: 0.4944, r@5_rev_retrieval: 0.7852, r@10_rev_retrieval: 0.8620 Updated run f253273762 Reviewed By: ytsheng, vedanuj, apsdehal Differential Revision: D24321055 fbshipit-source-id: 238ce5a98d82fc05cb5f826df535ddfe612b43fe | 06 March 2021, 00:53:09 UTC |
| 5f9bf6e | Vedanuj Goswami | 05 March 2021, 20:51:13 UTC | [feat] Decouple transformer head module (#778) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/778 This PR separate heads from dependencies with the trunk, allowing us to add any number of heads/pretraining objectives without modifying the model code and purely through configs. - No model code need to be modified to add a new head - Several heads can be added to a single model. Useful when we want to try several pretraining objectives or several losses based on different heads - Since the heads are decoupled from the model trunk, these can be utilized with other models as well, bringing in more modularity and flexibility. - All head specific parameters and logic will be confined to the head nn.Module - Loss calculation can happen in the head as well as at the model level, no change in behavior here - Added tests for different heads Performance Improvements: - MLM head is now made memory efficient by projecting only the masked tokens Reviewed By: apsdehal Differential Revision: D26310387 fbshipit-source-id: 0699795b266f1c2301f63a929160b58133b9d943 | 05 March 2021, 20:52:56 UTC |
| ef8b0f9 | Sasha Sheng | 04 March 2021, 18:29:40 UTC | [fix] move batch to current device in eval (#795) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/795 Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/142 to address an issue seen by AICommerce. https://fb.workplace.com/groups/574291956570984/permalink/700657133934465/ Reviewed By: apsdehal Differential Revision: D26808859 fbshipit-source-id: 25acdf7a9677c769490e80be48921ca4df6777c9 | 04 March 2021, 18:31:12 UTC |
| 163c202 | Ronghang Hu | 02 March 2021, 17:52:45 UTC | [feat] multitask support: per-dataset evaluation loop, metrics, and sampling ratios (#739) Summary: This PR is built upon https://github.com/facebookresearch/mmf/pull/736 - change the current evaluation loop (that runs on the union of all datasets) to per-dataset evaluation. - generate prediction JSON object during evaluation loop, which can be consumed by metrics that evaluate the entire dataset (e.g. mAP for object detection or CIDEr for image captioning) and cannot be expressed as averaging over per-batch metrics - run metrics on a subset of datasets with (optional) `datasets` config - allow specifying per-dataset sampling ratio for multi-task training Pull Request resolved: https://github.com/facebookresearch/mmf/pull/739 Test Plan: Tested locally and verified the outputs --- Example on specifying per-dataset metric with `datasets` (here `vqa_accuracy` will only run on vqa2 while `detection_mean_ap` will only run on detection_coco and detection_visual_genome). If `datasets` is not specified, the default behavior is to run on all dataset: ``` evaluation: metrics: - type: vqa_accuracy datasets: - vqa2 - type: detection_mean_ap datasets: - detection_coco - detection_visual_genome ``` --- Examples specifying per-dataset sampling ratio during training through `multitasking.enabled` and `multitasking.sampling_ratios`: ``` multitasking: enabled: true sampling_ratios: detection_coco: 0.2 detection_visual_genome: 0.07 visual_entailment: 0.12 vqa2: 0.26 glue_qnli: 0.1 glue_mnli_mismatched: 0.1 glue_qqp: 0.1 glue_sst2: 0.05 ``` Reviewed By: apsdehal Differential Revision: D26709926 Pulled By: ronghanghu fbshipit-source-id: c6c0ebcdda5750890ffd4f93bcc909d9a39e257e | 02 March 2021, 17:55:11 UTC |
| 5246609 | Brett Zachary Allen | 01 March 2021, 22:51:56 UTC | [feat] add visualization API and notebook (#766) Summary: Add a visualization API for the frcnn features, which takes in image and feature paths and uses the huggingface visualization tools to show the image. Also added a jupyter notebook that will display the images accordingly if the features and images are in the correct place. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/766 Reviewed By: vedanuj Differential Revision: D26621835 Pulled By: brettallenyo fbshipit-source-id: a31e1a3da499c4044d6cf275b281ab5e4b0b720d | 01 March 2021, 22:53:16 UTC |
| 52a63d3 | Vedanuj Goswami | 28 February 2021, 01:43:40 UTC | [feat] Sharded DDP (#759) Summary: Adding ShardedDDP implementation - Activates ShardedDDP when Optimizer State Sharding(OSS) is enabled. - Works with both FP16 and FP32 - Requires fairscale to be installed Pull Request resolved: https://github.com/facebookresearch/mmf/pull/759 Test Plan: MMF Transformer VQA accuracy 44k iters (512 batch size): FP16 + OSS ShardedDDP + ShadedGradScaler : 65.87, mem: 17573.0 MB FP16 : 65.56, mem : 19032.0 MB OSS ShardedDDP: 65.90 , mem : 21073.0 MB Reviewed By: ytsheng, min-xu-ai Differential Revision: D26177429 Pulled By: vedanuj fbshipit-source-id: abce18d88e7ac4125a2dcd7ee40153e163835b34 | 28 February 2021, 01:46:10 UTC |
| 5c75d69 | Amanpreet Singh | 27 February 2021, 03:55:08 UTC | [feat] GLUE datasets from Hugginface datasets library (#736) Summary: - Uses single interface to load 4 GLUE Datasets - Currently, only adds 4 datasets needed by UniT, can be extended easily to more - Specify as `glue_qnli`, `glue_mnli_mismatched` etc. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/736 Test Plan: Tested locally and verified the outputs Reviewed By: ytsheng, ronghanghu Differential Revision: D25942348 Pulled By: apsdehal fbshipit-source-id: 83b86f0946df162a6253726daa075630b8533a2a | 27 February 2021, 03:56:26 UTC |
| 33bac8f | Rui Hou | 25 February 2021, 19:19:42 UTC | [perf] use PathManager.get_local_path for loading checkpoints (#789) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/789 Switching to use PathManager.get_local_path instead of directly loading from Manifold to improve stability Reviewed By: vedanuj Differential Revision: D26668375 fbshipit-source-id: 828bd5a49cc03ca2c96c3ca5db93c3b144903eb8 | 25 February 2021, 19:21:12 UTC |
| 2006e90 | John Knox | 25 February 2021, 11:39:57 UTC | [sdoc] Update static docs plugin on all sites Summary: Just a codemod to update everyone to the latest version. Created by running on devserver: ``` cd fbsource/xplat/staticdocs ./update-all-users 0.7.0 ``` Notable features: * No longer need to import FbInternalOnly or OssOnly, theyre already in scope and ready for use in markdown files. * Can now specify a baseDir for all code references to be evaluated relative to. Reviewed By: mweststrate Differential Revision: D26632914 fbshipit-source-id: 6f5ac7316ee23bc8f3b791782cd0509afe909afc | 25 February 2021, 11:41:38 UTC |
| f22814f | Amanpreet Singh | 25 February 2021, 00:30:56 UTC | [feat] Add PathManager support for lmdb_conversion script (#787) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/787 Adds support for PathManager to lmdb_conversion script so lmdbs can be extracted to PathManager registered paths as well. Reviewed By: vedanuj Differential Revision: D26642096 fbshipit-source-id: 3fa12bf1299750e89e1f4f24daf1c471c7ef3012 | 25 February 2021, 00:32:32 UTC |
| 867916a | Vedanuj Goswami | 24 February 2021, 18:10:53 UTC | [fix] Fix imports for FB specific files (#138) Summary: Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/138 Fix internal imports so that we can the code from mmf-internal in FAIR cluster. Fix all tests to run on mmf-internal Fix black/isort mismatches Add missing init files to FB folders Reviewed By: apsdehal Differential Revision: D26478270 fbshipit-source-id: 291f61b25bb5ba1755e5a5f211231471e86ea8b4 | 24 February 2021, 18:12:20 UTC |
| 8409361 | Brett Zachary Allen | 23 February 2021, 23:53:35 UTC | [feat] Add feature extraction script (#764) Summary: Add a feature extraction script that uses Huggingface's Fast RCNN structure and code to discover features in an image. `extract_features_frcnn.py` is my script, the other files were taken from Huggingface, other than super minor changes by me, including rotating the RGB/BGR values in the preprocessing script. The script takes a number of flags that alter it's behavior like exclude files, input files, output files, batch size, etc. Run `python3 mmf/tools/scripts/features/frcnn/extract_features_frcnn.py --model_file model.bin --config_file config.yaml --image_dir ./example_images --output_folder ./output_features` Pull Request resolved: https://github.com/facebookresearch/mmf/pull/764 Reviewed By: vedanuj Differential Revision: D26612531 Pulled By: brettallenyo fbshipit-source-id: aed9e127d9e102b2704cadd131e80bd594bce28e | 23 February 2021, 23:54:53 UTC |
| f5ff2c8 | Madian Khabsa | 23 February 2021, 20:08:40 UTC | [fix] Improvement to VocabDict loader (#786) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/786 Upstream callers to VocabDict tend to pass `data_dir` which leads to inadverently concating it with the vocab_file path. We now skip this step if the path already exists Reviewed By: apsdehal Differential Revision: D26601097 fbshipit-source-id: 6d9006c66d2e6ebef503ba0f8fc43b16a4d21550 | 23 February 2021, 20:10:05 UTC |
| 0837776 | Rui Hou | 23 February 2021, 17:35:24 UTC | [mmf][PostRay] Make capability as an arg Summary: To make capability as an arg so its easier to switch between V100-16GB and 32GB cards. Reviewed By: madian9, vedanuj, apsdehal Differential Revision: D26600874 fbshipit-source-id: b4f95c09a611fb46101cbd2043f132b1d53eb67e | 23 February 2021, 17:36:44 UTC |
| 6db9048 | Vedanuj Goswami | 23 February 2021, 04:55:02 UTC | [refactor, fix] Fix fbl flow to use iopath, some cleanup (#785) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/785 Some cleanup of file_io file Reviewed By: apsdehal, BruceChaun Differential Revision: D26599901 fbshipit-source-id: 1979248b54ec0d5b2566d158cae4a72028b1f116 | 23 February 2021, 04:56:33 UTC |
| b1fd2a9 | Vedanuj Goswami | 19 February 2021, 05:05:59 UTC | [refactor] Refactor output type of MMFT preprocess_output (#781) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/781 - Refactors the output type to be Dict[str, Dict[str, Tensor]] - Namedtuples cannot be inherited to include new fields easily - Dataclasses are not scriptable Reviewed By: apsdehal, xuefeicao Differential Revision: D26479261 fbshipit-source-id: d64b116fadc1cdb314e0cd66189bde54d11dc2f6 | 19 February 2021, 05:08:02 UTC |
| bb6a9b7 | John Knox | 18 February 2021, 19:31:32 UTC | [mmf][site] Upgrade Docusaurus Summary: To use FbInternalOnly, you need to be on Docusaurus 61+, this upgrades it. It also fixes a typo FBInternalOnly -> FbInternalOnly. MDX currently silently (with a warning) accepts components that haven't been imported, so this wasn't raising an error but wasn't doing anything either. Also fixes a few broken links (the new version of docusaurus checks this for you and enforce them). Reviewed By: apsdehal Differential Revision: D26510193 fbshipit-source-id: 81871d3151edce27a2281cbcaaa7e694309205a2 | 18 February 2021, 19:33:46 UTC |
| 99f8010 | Amanpreet Singh | 17 February 2021, 22:52:49 UTC | [feat] Add batch_size_per_device training option (#780) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/780 This will allow specifying batch size per device instead of default global batch size. If specified using `training.batch_size_per_device`, this will supersede the value specified in `training.batch_size` and information will be logged. Reviewed By: madian9, vedanuj Differential Revision: D26480897 fbshipit-source-id: 9ffb3d9b79a84c39db4000e6a3d051e3bd1b6eb5 | 17 February 2021, 22:54:16 UTC |
| daeaa95 | Amanpreet Singh | 17 February 2021, 18:03:07 UTC | [fix] Don't show warning for missing is_xla in registry (#779) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/779 The warning is plaguing the logs and should not be explicitly registered. This diff fixes that. Reviewed By: ytsheng Differential Revision: D26480670 fbshipit-source-id: 9c7d0a67a1cd8fa71d8c7537eeff75c107c1117b | 17 February 2021, 18:04:42 UTC |
| ea71345 | Vedanuj Goswami | 16 February 2021, 17:49:07 UTC | [fix] Fix SPM Tokenizer for masked labels (#775) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/775 Fix masked SPM tokenizer inherited from MaskedBertTokenizer Differential Revision: D26432162 fbshipit-source-id: f0acb2388c4030c48573ec68b8ff6ab4e9ecd99c | 16 February 2021, 17:51:22 UTC |
| 53ee8d8 | Vaibhav Singh | 12 February 2021, 22:14:22 UTC | [feat] Changes to support training on TPUs (#752) Summary: Creating this new PR since pytorch-xla branch is in use for an ongoing project. I have incorporated most of the changes understood from the last comments on https://github.com/facebookresearch/mmf/pull/693 Happy to follow up on the open comments here. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/752 Reviewed By: vedanuj Differential Revision: D26218782 Pulled By: apsdehal fbshipit-source-id: 6ce6e48134153cea9e9969acac021cf2bf955fe6 | 12 February 2021, 22:15:46 UTC |
| 18d685d | Amanpreet Singh | 12 February 2021, 20:29:05 UTC | [refactor,tests,fix] Clean up and fix a bug in MMFT; add extensive tests (#774) Summary: The diff address a bug that was introduced in text position ids by a recent change in https://github.com/facebookresearch/mmf/issues/760. The fix is to actually make single feature dim modalities B X 1 X D before passing to backend instead of making multiple assumption in backend layer which tend to be conflicting with text input ids. This also adds some comments to better explain stack version of text. The PR also refactors MMFT to be more readble and reduce cyclomatic complexity. Finally, this PR adds extensive tests to make sure a change like https://github.com/facebookresearch/mmf/issues/760 doesn't break MMFT again. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/774 Test Plan: Adds extensive tests for MMFT Tested disney's model. Reviewed By: ytsheng Differential Revision: D26410408 Pulled By: apsdehal fbshipit-source-id: 059af500739b3463219b8594b3bb72001326a0a2 | 12 February 2021, 20:30:49 UTC |
| 95a2225 | Amanpreet Singh | 11 February 2021, 22:02:20 UTC | [docs] Update internal docs to latest Summary: Updates fblearner and devserver docs to point to latest content. Reviewed By: ytsheng Differential Revision: D25511541 fbshipit-source-id: 58ba9ff19a294c5c419baad50e18d2b3ed1013d7 | 11 February 2021, 22:04:01 UTC |
| 2ff0062 | Vedanuj Goswami | 09 February 2021, 18:37:46 UTC | [feat] Add MLM pretraining head for MMF Transformer (#769) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/769 1. Adds a MLM pretraining head for MMF Transformer model 1. Fixes `MultiSentenceBertTokenizer` to also support multisentence masking 1. Adds `mlm_labels_list` preprocessing 1. Adds pretraining head specific postprocessing Note : This is not the best way we can add pretraining losses and heads, but we should tackle it properly when we start working on Pretraining Schemes support for MMFT and other models. Facebook : 1. Fixes dataset filtering to support the case when no `set_name_mapping` is available for a dataset 1. Fixes `FeaturesWithTextBatchProcessor` when no labels are available for a dataset 1. Fixes `MultiSentenceSPMTokenizer` to also support multisentence masking Reviewed By: apsdehal Differential Revision: D26270908 fbshipit-source-id: 84a1ae462aea8fda6e6b0c791fad7722a69dbfb2 | 09 February 2021, 18:39:19 UTC |
| 0ee1127 | Sasha Sheng | 08 February 2021, 10:50:33 UTC | [feat] PL mvp0: training (#748) Summary: * pytorch lighting stub mostly involving training * Tests for lightning trainer included * built on top of the mmf grad accumulation fix: https://github.com/facebookresearch/mmf/pull/747 - [X] MVP 0. Training: Goal - Train a model from scratch and reach similar accuracy as using mmf_trainer - [X] Setup the training pipeline: done - [X] Training on the right device: done - [X] Clip gradients: done - [X] Optimizer: done - [X] FP16 Support: done - [X] LR scheduler (incl. warmup etc): done - [X] testcase: train visual_bert on vqa from scratch fo 10 iterations, compare the value: done - [x] tests included in this PR (tests are only done for pytorch lightning integration): - [X] Vanilla Training w/o grad accumulate, make sure loss for 5 iters are the same between mmf and pl - [X] Optimizer working as intended as a part of this PR. - [X] `max_updates` and `max_epochs` calculation - [x] Training with grad accumulate - [x] Training with LR schedule achieves a different value compared to without LR schedule - [x] Training with LR schedule for PL is the same as training with LR schedule for `mmf_tranier` - [x] Training with gradient clipping make sure all grads are within the `grad_clipping` threshold - [x] Training with gradient clipping is the same as training with gradient clipping for `mmf_trainer` Pull Request resolved: https://github.com/facebookresearch/mmf/pull/748 Reviewed By: apsdehal, simran2905 Differential Revision: D26192869 Pulled By: ytsheng fbshipit-source-id: 203a91e893d6b878bbed80ed84960dd059cfc90c | 08 February 2021, 10:52:20 UTC |
| fc72ef0 | Sasha Sheng | 06 February 2021, 04:15:36 UTC | [fix] fix typo in model zoo config (#715) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/715 Reviewed By: vedanuj Differential Revision: D25553078 Pulled By: ytsheng fbshipit-source-id: e24989d7118b0de3e92a381182a1a7fe4b540bc6 | 06 February 2021, 04:17:31 UTC |
| d49abdf | John Knox | 04 February 2021, 19:46:47 UTC | [sdoc][codemod] Fix cert dialog popup on open-source sites Summary: Upgrades the FB-internal documentation plugin, to no longer call the internal api that requires auth, for external viewers. Context: https://fb.workplace.com/groups/654274975334601/permalink/1288672008171584/ Reviewed By: justintrudell Differential Revision: D26252469 fbshipit-source-id: 31068534ce79a7959c38c6e66e6a9cf12371e228 | 04 February 2021, 19:48:55 UTC |
| bfacca4 | Amanpreet Singh | 02 February 2021, 09:31:07 UTC | [feat] Changes to support bento notebook tutorial (#761) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/761 This diff aims to fix various things in MMF to provide an optimal bento notebook experience. - Fix some hasattr checks to "in" checks - Allow empty config_path for datasets and models - Also, allow missing model key or dataset key in yaml configs - Adds dataclass for LossConfig so as Losses can be initialized in the model through structured configs - Fix the usage of next(model.parameters()) to get the device. This doesn't work with dataparallel - Also, using arbitrary keys with text and image modality type. MMFT will now expect to find these in SampleList. - Allow passing `input_mask` as `modality_mask` for text type Reviewed By: vedanuj Differential Revision: D26170013 fbshipit-source-id: 7af3766ffcf50acc9d5d69dd1eb91da224e0783c | 02 February 2021, 09:32:36 UTC |
| 5b01c32 | Amanpreet Singh | 02 February 2021, 07:09:37 UTC | [feat] Support direct feature inputs with single position dim in MMFT (#760) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/760 This diff allows MMFT to support direct features which don't use any encoders are of position dim==1. Also, uses BertModelJit to initialize the transformer base so that it is torchscriptable. Reviewed By: vedanuj Differential Revision: D24555077 fbshipit-source-id: ee9be44fed10ba30f1a88437ec42a01f38952347 | 02 February 2021, 07:11:17 UTC |
| 28f1a81 | Amanpreet Singh | 02 February 2021, 07:09:37 UTC | [feat] Add config dataclasses for MMFT (#658) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/658 - Adds base dataclasses for MMFT and related encoders if something is missing - Fix usage of EncoderFactory with enums in build_model - Adds extensive test for building the model with the configs - Adds defaults to the configs as well based on the original configs - In second set of changes, need to isolate backend configs. Reviewed By: vedanuj Differential Revision: D24469282 fbshipit-source-id: c2e752e163c23b9d1142bbe2fd0b45553f33dc07 | 02 February 2021, 07:11:17 UTC |
| b8f4591 | Vedanuj Goswami | 01 February 2021, 09:50:35 UTC | [fix] Fix accumulate tensor fields in evaluation (#757) Summary: Fixes evaluation loop after changes in https://github.com/facebookresearch/mmf/issues/747 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/757 Reviewed By: ytsheng Differential Revision: D26171041 Pulled By: vedanuj fbshipit-source-id: f0fd24ef96ef54dd7ea17af5968e05f0301e64a3 | 01 February 2021, 09:52:13 UTC |
| 8237ed2 | Vedanuj Goswami | 01 February 2021, 08:13:39 UTC | [fix] Fix windows build for iopath dependency (#758) Summary: Fixes Windows build after it was broken due to iopath dependency inclusion. Latest version of pywin is not compatible with python3.8, so we need to fix it at v225. Reference : [issue](https://github.com/mhammond/pywin32/issues/1431). Pull Request resolved: https://github.com/facebookresearch/mmf/pull/758 Reviewed By: ytsheng Differential Revision: D26171059 Pulled By: vedanuj fbshipit-source-id: 332281c7987a8ba9cdb4cc3ec38f23476a2af146 | 01 February 2021, 08:15:45 UTC |
| 11b531e | Sasha Sheng | 30 January 2021, 03:11:45 UTC | [fix] fix gradient accumulation when update_frequency != 1 (#747) Summary: * fix gradient accumulation when update_frequency is not equal to 1 * The issue is caused by "start_update" being called at an irregular time so the gradients are actually not being accumulated in between update_frequency. When optimizer's `zero_grad` is called, grad is manually set to zero and therefore the gradients are not being accumulated. This fix made it so that `zero_grad` is called in the correct interval so that grad are being accumulated in between update_frequency. * fix `combined_report` not taking into account the new losses causing metrics calculation to be incorrect. * add test (including logging) * add test to make sure there is a final update even if the num batches from the current update < update frequency. * address issue: https://github.com/facebookresearch/mmf/issues/626 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/747 Reviewed By: apsdehal Differential Revision: D26034213 Pulled By: ytsheng fbshipit-source-id: fa0c7ad3566ce30f89b67996ac476a3d2dab779a | 30 January 2021, 03:13:11 UTC |
| 77a931e | Brett Zachary Allen | 27 January 2021, 23:49:03 UTC | [enhancement] replace usage of PathManager to use ioPath (#749) Summary: Replace code in file_io to use the new ioPath Path Manager instead of fvcore. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/749 Test Plan: - use existing `test_file_io.py` test suite Reviewed By: vedanuj Differential Revision: D26058703 Pulled By: brettallenyo fbshipit-source-id: 148aa502a2daed9c8ea44f38235dd19b168b8c54 | 27 January 2021, 23:50:40 UTC |
| b0e6c4a | Karan Desai | 26 January 2021, 00:41:22 UTC | [feat] Add generic TorchvisionResnetImageEncoder in ImageEncoderFactory. (#728) Summary: This PR adds a generic image encoder named `TorchvisionResnetImageEncoder` in `mmf.modules.ImageEncoderFactory` — this encoder can instantiate any ResNet-like model from [`torchvision.models`](https://pytorch.org/docs/stable/torchvision/models.html) based on its name. The encoder returns spatial grid features same as torchvision. It can be instantiated using a minimal config as: ```yaml type: "torchvision_resnet" params: name: <name> pretrained: false ``` - Set `pretrained: true` for loading ImageNet-pretrained weights from torchvision. - Supported `<name>`s are: - `resnet18`, `resnet34`, `resnet50`, `resnet101`, `resnet152` - `resnext50_32x4d`, `resnext101_32x8d` - `wide_resnet50_2`, `wide_resnet101_2` ### Example ```python >>> import torch >>> from omegaconf import OmegaConf >>> from mmf.utils.build import build_encoder >>> >>> config = OmegaConf.load("encoder_config.yaml") >>> encoder = build_encoder(config) >>> images = torch.randn(32, 3, 224, 224) # shape: (b, c, h, w) >>> grid_feats = encoder(images) # shape: (b, 2048, 7, 7) ``` ### Backward Compatibility This PR maintains backward compatibility as it adds a new encoder without changing existing ones. However, it subsumes the functionality of [ResNet152ImageEncoder](https://github.com/facebookresearch/mmf/blob/d04914a7bbae28f33356e8b17113a81d6291a5dc/mmf/modules/encoders.py#L215), hence the latter could be later replaced and deprecated in favor of this addition. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/728 Reviewed By: ytsheng Differential Revision: D25991085 Pulled By: vedanuj fbshipit-source-id: 1c8f732ac43c0c81e6cf9cf41deca9dca3c39efe | 26 January 2021, 00:43:15 UTC |
