| 62a50f1 | Sasha Sheng | 19 March 2021, 22:29:23 UTC | [fix] tests (pytest + black) (#149) Summary: skip fb tasks on devfair pytest Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/149 Reviewed By: apsdehal Differential Revision: D27185785 Pulled By: ytsheng fbshipit-source-id: 1e88b76607fbf86a67a87ea601ce8a2c35197499 | 19 March 2021, 22:31:00 UTC |
| 3ba4725 | Amanpreet Singh | 19 March 2021, 22:17:27 UTC | [fix] length for multi dataset loader; max_epochs; eta (#820) Summary: The calculation of length for multidataloader used get_batch_size to divide total dataset length which is incorrect as get_batch_size is per device rather than global batch size. This PR fixes it and fixes snapshot interval calculation as well. This in turn: - Fixes max_epochs support - ETA - All other things dependent on this Pull Request resolved: https://github.com/facebookresearch/mmf/pull/820 Test Plan: Tested locally with MVLT runs Reviewed By: ytsheng, madian9 Differential Revision: D27184804 Pulled By: apsdehal fbshipit-source-id: d18aeb52b7e280f33c3b9e7509dafa123ed1c629 | 19 March 2021, 22:18:53 UTC |
| 08fb370 | Amanpreet Singh | 18 March 2021, 22:58:39 UTC | [refactor] Fix typings used from mmf common/typings Summary: Currently, typings used in multiple places in MMF from common/typings are incorrect. This diff fixes them. Reviewed By: ytsheng Differential Revision: D27127754 fbshipit-source-id: fccbb5df0a7ffac2f12e73a9d47e369fb6b7a4d0 | 18 March 2021, 23:00:10 UTC |
| 7fb6bdc | Amanpreet Singh | 18 March 2021, 21:24:54 UTC | [feat] Allow returning sequences in transformer encoder (#814) Summary: - For some cases such as MLM we need full sequences. It would be great if we can reuse existing transformer encoder and support that. This PR adds support for the same along with tests Pull Request resolved: https://github.com/facebookresearch/mmf/pull/814 Test Plan: Tests have been added. Tested also with MVLT Reviewed By: ytsheng, vedanuj Differential Revision: D27075345 Pulled By: apsdehal fbshipit-source-id: c8d475405d4950494831846f2eef795975e5ff6c | 18 March 2021, 21:31:54 UTC |
| bbc8035 | Amanpreet Singh | 18 March 2021, 20:15:41 UTC | [feat,fix] Make some transformer head params optional; label key for MLM (#813) Summary: This PR makes some QoL changes for transformer heads so that they can be used at places other than MMFTransformer. - Make encoder_layers and processed_sample_list options optional by default - Add label key option for mlm to allow direct fetching of maskedlabels from sample list Pull Request resolved: https://github.com/facebookresearch/mmf/pull/813 Test Plan: Tested with GUniT Reviewed By: ytsheng, vedanuj Differential Revision: D27062330 Pulled By: apsdehal fbshipit-source-id: 3f7aed573b65154960a18dffb4ca6600c0d4bb34 | 18 March 2021, 20:17:11 UTC |
| 8f9150b | Amanpreet Singh | 18 March 2021, 00:05:31 UTC | [fix] Allow running slurm scripts from directory other than mmf (#818) Summary: - Currently, hardcoded to be run only from the same directory as mmf - This PR fixes it so that it can be launched from anywhere Pull Request resolved: https://github.com/facebookresearch/mmf/pull/818 Test Plan: Tested with MVLT Reviewed By: ytsheng Differential Revision: D27136433 Pulled By: apsdehal fbshipit-source-id: cfa0cb2696a7547c1a299ee9790eaa8fd9dbd307 | 18 March 2021, 00:06:58 UTC |
| 7cc0e4e | Ananth Subramaniam | 17 March 2021, 17:47:46 UTC | [mmf] Update gradient clipping test to use almost equals for float comparison (#147) Summary: Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/147 Reviewed By: ytsheng Differential Revision: D26783077 fbshipit-source-id: 65c278e68d25736ed89c9a1669aa5f401e0548f6 | 17 March 2021, 17:49:54 UTC |
| c672d08 | Ronghang Hu | 17 March 2021, 01:30:26 UTC | [feat] add the UniT model and supporting utils (#741) Summary: This PR is built upon https://github.com/facebookresearch/mmf/pull/740 - add the UniT model - add `AdamWSkipParamsWithZeroGrad` optimizer (skips unused params) - provide dataset zoo download for VQAv2 splits w.r.t. COCO 2017. - provide model zoo download for UniT. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/741 Test Plan: Tested locally and verified the outputs --- Evaluating the UniT model zoo entry ``` # on a 8-GPU machine python mmf_cli/run.py \ config=projects/unit/configs/all_8_datasets/shared_dec.yaml \ datasets=detection_coco,detection_visual_genome,vqa2,visual_entailment,glue_qnli,glue_sst2,glue_mnli_mismatched,glue_qqp \ model=unit \ run_type=val \ training.batch_size=8 \ checkpoint.resume_zoo=unit.defaults ``` Evaluation outputs on the 8-dataset model: ``` val/detection_coco/detection_mean_ap: 0.3899, val/detection_visual_genome/detection_mean_ap: 0.0329, val/vqa2/vqa_accuracy: 0.6696, val/visual_entailment/accuracy: 0.7316, val/glue_qnli/accuracy: 0.8790, val/glue_sst2/accuracy: 0.8922, val/glue_mnli_mismatched/accuracy: 0.8090, val/glue_qqp/accuracy: 0.9065 ``` Reviewed By: vedanuj, apsdehal Differential Revision: D26894259 Pulled By: ronghanghu fbshipit-source-id: b548313f724d61c2c6e38c2cba1ef19cd735dfa9 | 17 March 2021, 01:31:59 UTC |
| f2e45f7 | Amanpreet Singh | 12 March 2021, 00:17:39 UTC | [fix,feat] Support more wider variety of user dirs (#809) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/809 This PR handles some edge cases around user dirs which weren't handle before especially in cases where python files are compiled and moved to a different place. This different place can also have compile time generated modules which are not available directly to the original folder. The diff also handles support for specifying a python file as user dir input. Lastly, this diff adds extensive test to cover and test various user dir configurations. Reviewed By: ytsheng, vedanuj Differential Revision: D26838780 fbshipit-source-id: 38e4ffaf55053df0b29ff682b26bb4ae5b6638ac | 12 March 2021, 00:18:24 UTC |
| 60c9138 | vedanuj | 11 March 2021, 11:40:07 UTC | [fix] Fix memory leak in tests (#806) Summary: Fixing some memory leak in tests. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/806 Test Plan: Multiple re-runs of tests show no failure. Reviewed By: ytsheng, apsdehal Differential Revision: D26952871 Pulled By: vedanuj fbshipit-source-id: 0dc91b5f497dd234b25671c105afce788797dcb3 | 11 March 2021, 11:41:54 UTC |
| 6210494 | Madian Khabsa | 11 March 2021, 03:13:27 UTC | <fix> support providing configs that contain equal character (#808) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/808 The current parser would fail if the opt value has `=` which is needed when providing filters. This diff takes the first occurrence of `=` as the separator Reviewed By: apsdehal Differential Revision: D26955390 fbshipit-source-id: cedfb358de254a0c6d2a88744a0660984b045f47 | 11 March 2021, 03:15:10 UTC |
| c78a408 | Vedanuj Goswami | 10 March 2021, 23:41:25 UTC | [fix] Fix MMFT for loading older checkpoints, add tests and assertions (#805) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/805 Fix MMFT for loading older checkpoints after head module refactor. Also adds a test to catch these errors in future. Reviewed By: ytsheng Differential Revision: D26924574 fbshipit-source-id: 09fa22b1b922a6b3d1698f0fe545d826a1bfb5c6 | 10 March 2021, 23:42:54 UTC |
| 9a93bc4 | dependabot[bot] | 10 March 2021, 02:44:11 UTC | Bump elliptic from 6.5.3 to 6.5.4 in /website (#800) Summary: Bumps [elliptic](https://github.com/indutny/elliptic) from 6.5.3 to 6.5.4. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/indutny/elliptic/commit/43ac7f230069bd1575e1e4a58394a512303ba803"><code>43ac7f2</code></a> 6.5.4</li> <li><a href="https://github.com/indutny/elliptic/commit/f4bc72be11b0a508fb790f445c43534307c9255b"><code>f4bc72b</code></a> package: bump deps</li> <li><a href="https://github.com/indutny/elliptic/commit/441b7428b0e8f6636c42118ad2aaa186d3c34c3f"><code>441b742</code></a> ec: validate that a point before deriving keys</li> <li><a href="https://github.com/indutny/elliptic/commit/e71b2d9359c5fe9437fbf46f1f05096de447de57"><code>e71b2d9</code></a> lib: relint using eslint</li> <li><a href="https://github.com/indutny/elliptic/commit/8421a01aa3ff789c79f91eaf8845558a7be2b9fa"><code>8421a01</code></a> build(deps): bump elliptic from 6.4.1 to 6.5.3 (<a href="https://github-redirect.dependabot.com/indutny/elliptic/issues/231">https://github.com/facebookresearch/mmf/issues/231</a>)</li> <li>See full diff in <a href="https://github.com/indutny/elliptic/compare/v6.5.3...v6.5.4">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/facebookresearch/mmf/network/alerts). </details> Pull Request resolved: https://github.com/facebookresearch/mmf/pull/800 Reviewed By: vedanuj Differential Revision: D26890681 Pulled By: apsdehal fbshipit-source-id: 063737b8ef957f3e67edcc749b83660bd386a01b | 10 March 2021, 02:45:46 UTC |
| ff40099 | Amanpreet Singh | 09 March 2021, 18:47:16 UTC | [feat] First e2e test with emphasis on user dir (#609) Summary: This PR introduces first e2e test which uses `mmf_cli.run` programmatic API and log parsing to check for log file. Specifically in this PR we test for user_dir import by creating our own simple model and a dataset that always generate one. For a more friendly `user_dir` support, now we override default config in configuration with user_cmd_opts as well so that MMF can discover configs relative to user dir if needed. We run the usual run command and expect the model to overfit and always return 1. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/609 Test Plan: Tested locally on devfair using `pytest -v ./tests`. Reviewed By: vedanuj Differential Revision: D24130662 Pulled By: apsdehal fbshipit-source-id: cde12cc5f8cdbb317797adf374f9cd1c4637d4ab | 09 March 2021, 18:48:56 UTC |
| 413510f | Ronghang Hu | 08 March 2021, 19:17:48 UTC | [feat] detection datasets (COCO, Visual Genome) and mAP metric (#740) Summary: This PR is built upon https://github.com/facebookresearch/mmf/pull/739 - add `detection_coco` dataset - add `detection_visual_genome` datasets - add supporting detection utilities - add `detection_mean_ap` metric - add `DETRImageAndTargetProcessor` to load DETR image and box transforms Pull Request resolved: https://github.com/facebookresearch/mmf/pull/740 Test Plan: Tested locally and verified the outputs Reviewed By: vedanuj, apsdehal Differential Revision: D26765951 Pulled By: ronghanghu fbshipit-source-id: c908e4764656158c688f210c3c0a62add7977417 | 08 March 2021, 19:19:32 UTC |
| 82a158e | Rui Hou | 06 March 2021, 23:31:46 UTC | [feat] Make converting avgpool to Identity optional (#799) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/799 **Create a new diff to fix export to GitHub issue in D26781527** This diff makes convert avgpooling to Identity optional such that an image can be encoded as single embedding to be used as a token instead of feature maps for multiple tokens. Reviewed By: madian9 Differential Revision: D26870914 fbshipit-source-id: 07fc8228a203ec1a532b521df53d005c0e61589b | 06 March 2021, 23:33:36 UTC |
| 788e810 | Sasha Sheng | 06 March 2021, 10:00:52 UTC | [fix] pytorch 1.8 updates with rfft and irfft (#797) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/797 Reviewed By: vedanuj Differential Revision: D26840004 Pulled By: ytsheng fbshipit-source-id: 15ff847a5a0b13537d24fdddcb5e5d33166e6eb7 | 06 March 2021, 10:02:46 UTC |
| 853ac08 | Ning Zhang | 06 March 2021, 00:51:10 UTC | [feat] Image text alignment/retrieval model+dataset on flickr30k (#796) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/796 Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/143 Refer to Shir's diff D23716331 It is a subset of his diff, just the alignment part without RAG. Run f225045496, r@1_retrieval: 0.6310, r@5_retrieval: 0.8700, r@10_retrieval: 0.9250 r@1_rev_retrieval: 0.5022, r@5_rev_retrieval: 0.7870, r@10_rev_retrieval: 0.8632 Result reported in the paper r@1_retrieval: 0.651, r@5_retrieval: 0.873, r@10_retrieval: 0.926 r@1_rev_retrieval: 0.506, r@5_rev_retrieval: 0.788, r@10_rev_retrieval: 0.867 Running shir's code on FAIR dev machine r@1_retrieval: 0.6290, r@5_retrieval: 0.8620, r@10_retrieval: 0.9210, r@1_rev_retrieval: 0.4944, r@5_rev_retrieval: 0.7852, r@10_rev_retrieval: 0.8620 Updated run f253273762 Reviewed By: ytsheng, vedanuj, apsdehal Differential Revision: D24321055 fbshipit-source-id: 238ce5a98d82fc05cb5f826df535ddfe612b43fe | 06 March 2021, 00:53:09 UTC |
| 5f9bf6e | Vedanuj Goswami | 05 March 2021, 20:51:13 UTC | [feat] Decouple transformer head module (#778) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/778 This PR separate heads from dependencies with the trunk, allowing us to add any number of heads/pretraining objectives without modifying the model code and purely through configs. - No model code need to be modified to add a new head - Several heads can be added to a single model. Useful when we want to try several pretraining objectives or several losses based on different heads - Since the heads are decoupled from the model trunk, these can be utilized with other models as well, bringing in more modularity and flexibility. - All head specific parameters and logic will be confined to the head nn.Module - Loss calculation can happen in the head as well as at the model level, no change in behavior here - Added tests for different heads Performance Improvements: - MLM head is now made memory efficient by projecting only the masked tokens Reviewed By: apsdehal Differential Revision: D26310387 fbshipit-source-id: 0699795b266f1c2301f63a929160b58133b9d943 | 05 March 2021, 20:52:56 UTC |
| ef8b0f9 | Sasha Sheng | 04 March 2021, 18:29:40 UTC | [fix] move batch to current device in eval (#795) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/795 Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/142 to address an issue seen by AICommerce. https://fb.workplace.com/groups/574291956570984/permalink/700657133934465/ Reviewed By: apsdehal Differential Revision: D26808859 fbshipit-source-id: 25acdf7a9677c769490e80be48921ca4df6777c9 | 04 March 2021, 18:31:12 UTC |
| 163c202 | Ronghang Hu | 02 March 2021, 17:52:45 UTC | [feat] multitask support: per-dataset evaluation loop, metrics, and sampling ratios (#739) Summary: This PR is built upon https://github.com/facebookresearch/mmf/pull/736 - change the current evaluation loop (that runs on the union of all datasets) to per-dataset evaluation. - generate prediction JSON object during evaluation loop, which can be consumed by metrics that evaluate the entire dataset (e.g. mAP for object detection or CIDEr for image captioning) and cannot be expressed as averaging over per-batch metrics - run metrics on a subset of datasets with (optional) `datasets` config - allow specifying per-dataset sampling ratio for multi-task training Pull Request resolved: https://github.com/facebookresearch/mmf/pull/739 Test Plan: Tested locally and verified the outputs --- Example on specifying per-dataset metric with `datasets` (here `vqa_accuracy` will only run on vqa2 while `detection_mean_ap` will only run on detection_coco and detection_visual_genome). If `datasets` is not specified, the default behavior is to run on all dataset: ``` evaluation: metrics: - type: vqa_accuracy datasets: - vqa2 - type: detection_mean_ap datasets: - detection_coco - detection_visual_genome ``` --- Examples specifying per-dataset sampling ratio during training through `multitasking.enabled` and `multitasking.sampling_ratios`: ``` multitasking: enabled: true sampling_ratios: detection_coco: 0.2 detection_visual_genome: 0.07 visual_entailment: 0.12 vqa2: 0.26 glue_qnli: 0.1 glue_mnli_mismatched: 0.1 glue_qqp: 0.1 glue_sst2: 0.05 ``` Reviewed By: apsdehal Differential Revision: D26709926 Pulled By: ronghanghu fbshipit-source-id: c6c0ebcdda5750890ffd4f93bcc909d9a39e257e | 02 March 2021, 17:55:11 UTC |
| 5246609 | Brett Zachary Allen | 01 March 2021, 22:51:56 UTC | [feat] add visualization API and notebook (#766) Summary: Add a visualization API for the frcnn features, which takes in image and feature paths and uses the huggingface visualization tools to show the image. Also added a jupyter notebook that will display the images accordingly if the features and images are in the correct place. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/766 Reviewed By: vedanuj Differential Revision: D26621835 Pulled By: brettallenyo fbshipit-source-id: a31e1a3da499c4044d6cf275b281ab5e4b0b720d | 01 March 2021, 22:53:16 UTC |
| 52a63d3 | Vedanuj Goswami | 28 February 2021, 01:43:40 UTC | [feat] Sharded DDP (#759) Summary: Adding ShardedDDP implementation - Activates ShardedDDP when Optimizer State Sharding(OSS) is enabled. - Works with both FP16 and FP32 - Requires fairscale to be installed Pull Request resolved: https://github.com/facebookresearch/mmf/pull/759 Test Plan: MMF Transformer VQA accuracy 44k iters (512 batch size): FP16 + OSS ShardedDDP + ShadedGradScaler : 65.87, mem: 17573.0 MB FP16 : 65.56, mem : 19032.0 MB OSS ShardedDDP: 65.90 , mem : 21073.0 MB Reviewed By: ytsheng, min-xu-ai Differential Revision: D26177429 Pulled By: vedanuj fbshipit-source-id: abce18d88e7ac4125a2dcd7ee40153e163835b34 | 28 February 2021, 01:46:10 UTC |
| 5c75d69 | Amanpreet Singh | 27 February 2021, 03:55:08 UTC | [feat] GLUE datasets from Hugginface datasets library (#736) Summary: - Uses single interface to load 4 GLUE Datasets - Currently, only adds 4 datasets needed by UniT, can be extended easily to more - Specify as `glue_qnli`, `glue_mnli_mismatched` etc. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/736 Test Plan: Tested locally and verified the outputs Reviewed By: ytsheng, ronghanghu Differential Revision: D25942348 Pulled By: apsdehal fbshipit-source-id: 83b86f0946df162a6253726daa075630b8533a2a | 27 February 2021, 03:56:26 UTC |
| 33bac8f | Rui Hou | 25 February 2021, 19:19:42 UTC | [perf] use PathManager.get_local_path for loading checkpoints (#789) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/789 Switching to use PathManager.get_local_path instead of directly loading from Manifold to improve stability Reviewed By: vedanuj Differential Revision: D26668375 fbshipit-source-id: 828bd5a49cc03ca2c96c3ca5db93c3b144903eb8 | 25 February 2021, 19:21:12 UTC |
| 2006e90 | John Knox | 25 February 2021, 11:39:57 UTC | [sdoc] Update static docs plugin on all sites Summary: Just a codemod to update everyone to the latest version. Created by running on devserver: ``` cd fbsource/xplat/staticdocs ./update-all-users 0.7.0 ``` Notable features: * No longer need to import FbInternalOnly or OssOnly, theyre already in scope and ready for use in markdown files. * Can now specify a baseDir for all code references to be evaluated relative to. Reviewed By: mweststrate Differential Revision: D26632914 fbshipit-source-id: 6f5ac7316ee23bc8f3b791782cd0509afe909afc | 25 February 2021, 11:41:38 UTC |
| f22814f | Amanpreet Singh | 25 February 2021, 00:30:56 UTC | [feat] Add PathManager support for lmdb_conversion script (#787) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/787 Adds support for PathManager to lmdb_conversion script so lmdbs can be extracted to PathManager registered paths as well. Reviewed By: vedanuj Differential Revision: D26642096 fbshipit-source-id: 3fa12bf1299750e89e1f4f24daf1c471c7ef3012 | 25 February 2021, 00:32:32 UTC |
| 867916a | Vedanuj Goswami | 24 February 2021, 18:10:53 UTC | [fix] Fix imports for FB specific files (#138) Summary: Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/138 Fix internal imports so that we can the code from mmf-internal in FAIR cluster. Fix all tests to run on mmf-internal Fix black/isort mismatches Add missing init files to FB folders Reviewed By: apsdehal Differential Revision: D26478270 fbshipit-source-id: 291f61b25bb5ba1755e5a5f211231471e86ea8b4 | 24 February 2021, 18:12:20 UTC |
| 8409361 | Brett Zachary Allen | 23 February 2021, 23:53:35 UTC | [feat] Add feature extraction script (#764) Summary: Add a feature extraction script that uses Huggingface's Fast RCNN structure and code to discover features in an image. `extract_features_frcnn.py` is my script, the other files were taken from Huggingface, other than super minor changes by me, including rotating the RGB/BGR values in the preprocessing script. The script takes a number of flags that alter it's behavior like exclude files, input files, output files, batch size, etc. Run `python3 mmf/tools/scripts/features/frcnn/extract_features_frcnn.py --model_file model.bin --config_file config.yaml --image_dir ./example_images --output_folder ./output_features` Pull Request resolved: https://github.com/facebookresearch/mmf/pull/764 Reviewed By: vedanuj Differential Revision: D26612531 Pulled By: brettallenyo fbshipit-source-id: aed9e127d9e102b2704cadd131e80bd594bce28e | 23 February 2021, 23:54:53 UTC |
| f5ff2c8 | Madian Khabsa | 23 February 2021, 20:08:40 UTC | [fix] Improvement to VocabDict loader (#786) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/786 Upstream callers to VocabDict tend to pass `data_dir` which leads to inadverently concating it with the vocab_file path. We now skip this step if the path already exists Reviewed By: apsdehal Differential Revision: D26601097 fbshipit-source-id: 6d9006c66d2e6ebef503ba0f8fc43b16a4d21550 | 23 February 2021, 20:10:05 UTC |
| 0837776 | Rui Hou | 23 February 2021, 17:35:24 UTC | [mmf][PostRay] Make capability as an arg Summary: To make capability as an arg so its easier to switch between V100-16GB and 32GB cards. Reviewed By: madian9, vedanuj, apsdehal Differential Revision: D26600874 fbshipit-source-id: b4f95c09a611fb46101cbd2043f132b1d53eb67e | 23 February 2021, 17:36:44 UTC |
| 6db9048 | Vedanuj Goswami | 23 February 2021, 04:55:02 UTC | [refactor, fix] Fix fbl flow to use iopath, some cleanup (#785) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/785 Some cleanup of file_io file Reviewed By: apsdehal, BruceChaun Differential Revision: D26599901 fbshipit-source-id: 1979248b54ec0d5b2566d158cae4a72028b1f116 | 23 February 2021, 04:56:33 UTC |
| b1fd2a9 | Vedanuj Goswami | 19 February 2021, 05:05:59 UTC | [refactor] Refactor output type of MMFT preprocess_output (#781) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/781 - Refactors the output type to be Dict[str, Dict[str, Tensor]] - Namedtuples cannot be inherited to include new fields easily - Dataclasses are not scriptable Reviewed By: apsdehal, xuefeicao Differential Revision: D26479261 fbshipit-source-id: d64b116fadc1cdb314e0cd66189bde54d11dc2f6 | 19 February 2021, 05:08:02 UTC |
| bb6a9b7 | John Knox | 18 February 2021, 19:31:32 UTC | [mmf][site] Upgrade Docusaurus Summary: To use FbInternalOnly, you need to be on Docusaurus 61+, this upgrades it. It also fixes a typo FBInternalOnly -> FbInternalOnly. MDX currently silently (with a warning) accepts components that haven't been imported, so this wasn't raising an error but wasn't doing anything either. Also fixes a few broken links (the new version of docusaurus checks this for you and enforce them). Reviewed By: apsdehal Differential Revision: D26510193 fbshipit-source-id: 81871d3151edce27a2281cbcaaa7e694309205a2 | 18 February 2021, 19:33:46 UTC |
| 99f8010 | Amanpreet Singh | 17 February 2021, 22:52:49 UTC | [feat] Add batch_size_per_device training option (#780) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/780 This will allow specifying batch size per device instead of default global batch size. If specified using `training.batch_size_per_device`, this will supersede the value specified in `training.batch_size` and information will be logged. Reviewed By: madian9, vedanuj Differential Revision: D26480897 fbshipit-source-id: 9ffb3d9b79a84c39db4000e6a3d051e3bd1b6eb5 | 17 February 2021, 22:54:16 UTC |
| daeaa95 | Amanpreet Singh | 17 February 2021, 18:03:07 UTC | [fix] Don't show warning for missing is_xla in registry (#779) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/779 The warning is plaguing the logs and should not be explicitly registered. This diff fixes that. Reviewed By: ytsheng Differential Revision: D26480670 fbshipit-source-id: 9c7d0a67a1cd8fa71d8c7537eeff75c107c1117b | 17 February 2021, 18:04:42 UTC |
| ea71345 | Vedanuj Goswami | 16 February 2021, 17:49:07 UTC | [fix] Fix SPM Tokenizer for masked labels (#775) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/775 Fix masked SPM tokenizer inherited from MaskedBertTokenizer Differential Revision: D26432162 fbshipit-source-id: f0acb2388c4030c48573ec68b8ff6ab4e9ecd99c | 16 February 2021, 17:51:22 UTC |
| 53ee8d8 | Vaibhav Singh | 12 February 2021, 22:14:22 UTC | [feat] Changes to support training on TPUs (#752) Summary: Creating this new PR since pytorch-xla branch is in use for an ongoing project. I have incorporated most of the changes understood from the last comments on https://github.com/facebookresearch/mmf/pull/693 Happy to follow up on the open comments here. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/752 Reviewed By: vedanuj Differential Revision: D26218782 Pulled By: apsdehal fbshipit-source-id: 6ce6e48134153cea9e9969acac021cf2bf955fe6 | 12 February 2021, 22:15:46 UTC |
| 18d685d | Amanpreet Singh | 12 February 2021, 20:29:05 UTC | [refactor,tests,fix] Clean up and fix a bug in MMFT; add extensive tests (#774) Summary: The diff address a bug that was introduced in text position ids by a recent change in https://github.com/facebookresearch/mmf/issues/760. The fix is to actually make single feature dim modalities B X 1 X D before passing to backend instead of making multiple assumption in backend layer which tend to be conflicting with text input ids. This also adds some comments to better explain stack version of text. The PR also refactors MMFT to be more readble and reduce cyclomatic complexity. Finally, this PR adds extensive tests to make sure a change like https://github.com/facebookresearch/mmf/issues/760 doesn't break MMFT again. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/774 Test Plan: Adds extensive tests for MMFT Tested disney's model. Reviewed By: ytsheng Differential Revision: D26410408 Pulled By: apsdehal fbshipit-source-id: 059af500739b3463219b8594b3bb72001326a0a2 | 12 February 2021, 20:30:49 UTC |
| 95a2225 | Amanpreet Singh | 11 February 2021, 22:02:20 UTC | [docs] Update internal docs to latest Summary: Updates fblearner and devserver docs to point to latest content. Reviewed By: ytsheng Differential Revision: D25511541 fbshipit-source-id: 58ba9ff19a294c5c419baad50e18d2b3ed1013d7 | 11 February 2021, 22:04:01 UTC |
| 2ff0062 | Vedanuj Goswami | 09 February 2021, 18:37:46 UTC | [feat] Add MLM pretraining head for MMF Transformer (#769) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/769 1. Adds a MLM pretraining head for MMF Transformer model 1. Fixes `MultiSentenceBertTokenizer` to also support multisentence masking 1. Adds `mlm_labels_list` preprocessing 1. Adds pretraining head specific postprocessing Note : This is not the best way we can add pretraining losses and heads, but we should tackle it properly when we start working on Pretraining Schemes support for MMFT and other models. Facebook : 1. Fixes dataset filtering to support the case when no `set_name_mapping` is available for a dataset 1. Fixes `FeaturesWithTextBatchProcessor` when no labels are available for a dataset 1. Fixes `MultiSentenceSPMTokenizer` to also support multisentence masking Reviewed By: apsdehal Differential Revision: D26270908 fbshipit-source-id: 84a1ae462aea8fda6e6b0c791fad7722a69dbfb2 | 09 February 2021, 18:39:19 UTC |
| 0ee1127 | Sasha Sheng | 08 February 2021, 10:50:33 UTC | [feat] PL mvp0: training (#748) Summary: * pytorch lighting stub mostly involving training * Tests for lightning trainer included * built on top of the mmf grad accumulation fix: https://github.com/facebookresearch/mmf/pull/747 - [X] MVP 0. Training: Goal - Train a model from scratch and reach similar accuracy as using mmf_trainer - [X] Setup the training pipeline: done - [X] Training on the right device: done - [X] Clip gradients: done - [X] Optimizer: done - [X] FP16 Support: done - [X] LR scheduler (incl. warmup etc): done - [X] testcase: train visual_bert on vqa from scratch fo 10 iterations, compare the value: done - [x] tests included in this PR (tests are only done for pytorch lightning integration): - [X] Vanilla Training w/o grad accumulate, make sure loss for 5 iters are the same between mmf and pl - [X] Optimizer working as intended as a part of this PR. - [X] `max_updates` and `max_epochs` calculation - [x] Training with grad accumulate - [x] Training with LR schedule achieves a different value compared to without LR schedule - [x] Training with LR schedule for PL is the same as training with LR schedule for `mmf_tranier` - [x] Training with gradient clipping make sure all grads are within the `grad_clipping` threshold - [x] Training with gradient clipping is the same as training with gradient clipping for `mmf_trainer` Pull Request resolved: https://github.com/facebookresearch/mmf/pull/748 Reviewed By: apsdehal, simran2905 Differential Revision: D26192869 Pulled By: ytsheng fbshipit-source-id: 203a91e893d6b878bbed80ed84960dd059cfc90c | 08 February 2021, 10:52:20 UTC |
| fc72ef0 | Sasha Sheng | 06 February 2021, 04:15:36 UTC | [fix] fix typo in model zoo config (#715) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/715 Reviewed By: vedanuj Differential Revision: D25553078 Pulled By: ytsheng fbshipit-source-id: e24989d7118b0de3e92a381182a1a7fe4b540bc6 | 06 February 2021, 04:17:31 UTC |
| d49abdf | John Knox | 04 February 2021, 19:46:47 UTC | [sdoc][codemod] Fix cert dialog popup on open-source sites Summary: Upgrades the FB-internal documentation plugin, to no longer call the internal api that requires auth, for external viewers. Context: https://fb.workplace.com/groups/654274975334601/permalink/1288672008171584/ Reviewed By: justintrudell Differential Revision: D26252469 fbshipit-source-id: 31068534ce79a7959c38c6e66e6a9cf12371e228 | 04 February 2021, 19:48:55 UTC |
| bfacca4 | Amanpreet Singh | 02 February 2021, 09:31:07 UTC | [feat] Changes to support bento notebook tutorial (#761) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/761 This diff aims to fix various things in MMF to provide an optimal bento notebook experience. - Fix some hasattr checks to "in" checks - Allow empty config_path for datasets and models - Also, allow missing model key or dataset key in yaml configs - Adds dataclass for LossConfig so as Losses can be initialized in the model through structured configs - Fix the usage of next(model.parameters()) to get the device. This doesn't work with dataparallel - Also, using arbitrary keys with text and image modality type. MMFT will now expect to find these in SampleList. - Allow passing `input_mask` as `modality_mask` for text type Reviewed By: vedanuj Differential Revision: D26170013 fbshipit-source-id: 7af3766ffcf50acc9d5d69dd1eb91da224e0783c | 02 February 2021, 09:32:36 UTC |
| 5b01c32 | Amanpreet Singh | 02 February 2021, 07:09:37 UTC | [feat] Support direct feature inputs with single position dim in MMFT (#760) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/760 This diff allows MMFT to support direct features which don't use any encoders are of position dim==1. Also, uses BertModelJit to initialize the transformer base so that it is torchscriptable. Reviewed By: vedanuj Differential Revision: D24555077 fbshipit-source-id: ee9be44fed10ba30f1a88437ec42a01f38952347 | 02 February 2021, 07:11:17 UTC |
| 28f1a81 | Amanpreet Singh | 02 February 2021, 07:09:37 UTC | [feat] Add config dataclasses for MMFT (#658) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/658 - Adds base dataclasses for MMFT and related encoders if something is missing - Fix usage of EncoderFactory with enums in build_model - Adds extensive test for building the model with the configs - Adds defaults to the configs as well based on the original configs - In second set of changes, need to isolate backend configs. Reviewed By: vedanuj Differential Revision: D24469282 fbshipit-source-id: c2e752e163c23b9d1142bbe2fd0b45553f33dc07 | 02 February 2021, 07:11:17 UTC |
| b8f4591 | Vedanuj Goswami | 01 February 2021, 09:50:35 UTC | [fix] Fix accumulate tensor fields in evaluation (#757) Summary: Fixes evaluation loop after changes in https://github.com/facebookresearch/mmf/issues/747 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/757 Reviewed By: ytsheng Differential Revision: D26171041 Pulled By: vedanuj fbshipit-source-id: f0fd24ef96ef54dd7ea17af5968e05f0301e64a3 | 01 February 2021, 09:52:13 UTC |
| 8237ed2 | Vedanuj Goswami | 01 February 2021, 08:13:39 UTC | [fix] Fix windows build for iopath dependency (#758) Summary: Fixes Windows build after it was broken due to iopath dependency inclusion. Latest version of pywin is not compatible with python3.8, so we need to fix it at v225. Reference : [issue](https://github.com/mhammond/pywin32/issues/1431). Pull Request resolved: https://github.com/facebookresearch/mmf/pull/758 Reviewed By: ytsheng Differential Revision: D26171059 Pulled By: vedanuj fbshipit-source-id: 332281c7987a8ba9cdb4cc3ec38f23476a2af146 | 01 February 2021, 08:15:45 UTC |
| 11b531e | Sasha Sheng | 30 January 2021, 03:11:45 UTC | [fix] fix gradient accumulation when update_frequency != 1 (#747) Summary: * fix gradient accumulation when update_frequency is not equal to 1 * The issue is caused by "start_update" being called at an irregular time so the gradients are actually not being accumulated in between update_frequency. When optimizer's `zero_grad` is called, grad is manually set to zero and therefore the gradients are not being accumulated. This fix made it so that `zero_grad` is called in the correct interval so that grad are being accumulated in between update_frequency. * fix `combined_report` not taking into account the new losses causing metrics calculation to be incorrect. * add test (including logging) * add test to make sure there is a final update even if the num batches from the current update < update frequency. * address issue: https://github.com/facebookresearch/mmf/issues/626 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/747 Reviewed By: apsdehal Differential Revision: D26034213 Pulled By: ytsheng fbshipit-source-id: fa0c7ad3566ce30f89b67996ac476a3d2dab779a | 30 January 2021, 03:13:11 UTC |
| 77a931e | Brett Zachary Allen | 27 January 2021, 23:49:03 UTC | [enhancement] replace usage of PathManager to use ioPath (#749) Summary: Replace code in file_io to use the new ioPath Path Manager instead of fvcore. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/749 Test Plan: - use existing `test_file_io.py` test suite Reviewed By: vedanuj Differential Revision: D26058703 Pulled By: brettallenyo fbshipit-source-id: 148aa502a2daed9c8ea44f38235dd19b168b8c54 | 27 January 2021, 23:50:40 UTC |
| b0e6c4a | Karan Desai | 26 January 2021, 00:41:22 UTC | [feat] Add generic TorchvisionResnetImageEncoder in ImageEncoderFactory. (#728) Summary: This PR adds a generic image encoder named `TorchvisionResnetImageEncoder` in `mmf.modules.ImageEncoderFactory` — this encoder can instantiate any ResNet-like model from [`torchvision.models`](https://pytorch.org/docs/stable/torchvision/models.html) based on its name. The encoder returns spatial grid features same as torchvision. It can be instantiated using a minimal config as: ```yaml type: "torchvision_resnet" params: name: <name> pretrained: false ``` - Set `pretrained: true` for loading ImageNet-pretrained weights from torchvision. - Supported `<name>`s are: - `resnet18`, `resnet34`, `resnet50`, `resnet101`, `resnet152` - `resnext50_32x4d`, `resnext101_32x8d` - `wide_resnet50_2`, `wide_resnet101_2` ### Example ```python >>> import torch >>> from omegaconf import OmegaConf >>> from mmf.utils.build import build_encoder >>> >>> config = OmegaConf.load("encoder_config.yaml") >>> encoder = build_encoder(config) >>> images = torch.randn(32, 3, 224, 224) # shape: (b, c, h, w) >>> grid_feats = encoder(images) # shape: (b, 2048, 7, 7) ``` ### Backward Compatibility This PR maintains backward compatibility as it adds a new encoder without changing existing ones. However, it subsumes the functionality of [ResNet152ImageEncoder](https://github.com/facebookresearch/mmf/blob/d04914a7bbae28f33356e8b17113a81d6291a5dc/mmf/modules/encoders.py#L215), hence the latter could be later replaced and deprecated in favor of this addition. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/728 Reviewed By: ytsheng Differential Revision: D25991085 Pulled By: vedanuj fbshipit-source-id: 1c8f732ac43c0c81e6cf9cf41deca9dca3c39efe | 26 January 2021, 00:43:15 UTC |
| cd77a4b | Brett Zachary Allen | 25 January 2021, 20:05:38 UTC | [fix, test] Fix recall metric bug, add test (#746) Summary: - Fix infinite recursion error https://github.com/facebookresearch/mmf/issues/725 - Add back mistakenly removed code - Add test for recall metric at 1, 5, and 10 This is a response to bug https://github.com/facebookresearch/mmf/issues/725 I believe `process_ranks` may have been accidentally deleted in this commit so I added it back: https://github.com/facebookresearch/mmf/commit/6b6f4fc221c46f8c55d55a06e110f3d8f9ca2417 I also added a testing file, which basically ranks all of the expected values in increasing order (so the last candidate is always the chosen one). Both the expected values and predicted values are (10, 100) tensors. As the index of top level array increases, I alter the final value to be decreasing by 2, so the chosen candidates are 198 (maximum value 99*2), 191, 189, 187, ... I decrease it in this way so that each of the candidates become out of the bounds for the increasing Recall Metrics. The rankings are `tensor([ 1, 4, 5, 6, 7, 8, 9, 10, 11, 12])` This makes the results varying for each metric: `RecallAt1` --> 1/10 = .1 `RecallAt5` --> 3/10 = .3 `RecallAt10` --> 8/10 = .8 I would love for people to take a look at `metrics.py` line `462` (`process_ranks`) There are print statements in here from the previous commit, but I do not see print statements anywhere else in the modules, should I delete these, leave them, do we have a different logging style? Thanks! Pull Request resolved: https://github.com/facebookresearch/mmf/pull/746 Reviewed By: vedanuj Differential Revision: D26049706 Pulled By: brettallenyo fbshipit-source-id: 03546c76f9aed3a1c26b318eefb00fbb05d1297a | 25 January 2021, 20:07:14 UTC |
| 079f71d | Amanpreet Singh | 21 January 2021, 01:56:15 UTC | [fix] Allow loading checkpoint from a folder (#735) Summary: - Needed for e2e movie+mcan Pull Request resolved: https://github.com/facebookresearch/mmf/pull/735 Test Plan: - Tested locally with e2e movie+mcan handler - Adds tests for loading from a folder - Combines older file based test to main pretrained as they had sequential dep Reviewed By: ytsheng Differential Revision: D25936068 Pulled By: apsdehal fbshipit-source-id: 2c8989fcfaff2b1a2e56e4b97948809fc50bef49 | 21 January 2021, 01:58:09 UTC |
| 0229af6 | Sasha Sheng | 19 January 2021, 19:33:17 UTC | [enhancement] be more explicit in isort (#738) Summary: * print in ci the difference that isort would make * allow isort to be more explicit. * my pytorch lightning PR is experiencing some local/circleci inconsistencies for isort despite using the same version. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/738 Reviewed By: apsdehal Differential Revision: D25942131 Pulled By: ytsheng fbshipit-source-id: 67d33bbefca34229ee96cc815b52efdad913f68a | 19 January 2021, 19:35:00 UTC |
| 2848ba3 | Vedanuj Goswami | 12 January 2021, 19:15:12 UTC | [feat] D2 backbone intergation, Movie MCAN e2e (#700) Summary: [WIP] E2E with grid feats D2 (https://github.com/facebookresearch/mmf/commit/563a62d7bbbd89c857f711e8ca2670d33c129279) models Pull Request resolved: https://github.com/facebookresearch/mmf/pull/700 Reviewed By: apsdehal Differential Revision: D25886006 Pulled By: vedanuj fbshipit-source-id: 1fe3633dc8101ffc6f28ce48c434533d7a300d98 | 12 January 2021, 19:16:32 UTC |
| d04914a | Sasha Sheng | 07 January 2021, 05:54:20 UTC | [docs] update the terms and concepts doc (#724) Summary: * Updated the terms of concepts to include the latest models and datasets * Fixed the links in this page * Renamed pythia to mmf * Added links to those models/datasets Pull Request resolved: https://github.com/facebookresearch/mmf/pull/724 Reviewed By: apsdehal Differential Revision: D25727819 Pulled By: ytsheng fbshipit-source-id: 5e4fb0d3d98a6222ddea84f536e4b0ae19a67618 | 07 January 2021, 05:55:23 UTC |
| e00ef78 | Sasha Sheng | 05 January 2021, 01:21:43 UTC | [docs] dataset split from train feature docs (#726) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/726 Reviewed By: apsdehal Differential Revision: D25766616 Pulled By: ytsheng fbshipit-source-id: ae621b6a42b64d4387b4eb3ae5b9d2ad937cc8ec | 05 January 2021, 01:23:27 UTC |
| 51df5e5 | Vedanuj Goswami | 18 December 2020, 19:25:01 UTC | [docs] Add LXMERT to model zoo doc (#719) Summary: Adding LXMERT to update model zoo documentation. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/719 Reviewed By: ytsheng Differential Revision: D25618345 Pulled By: vedanuj fbshipit-source-id: f87913a9ca6f52d1ac590ef6af01a2438771314c | 18 December 2020, 19:26:49 UTC |
| 421b3f3 | dependabot[bot] | 18 December 2020, 19:24:49 UTC | Bump ini from 1.3.5 to 1.3.8 in /website (#713) Summary: Bumps [ini](https://github.com/isaacs/ini) from 1.3.5 to 1.3.8. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/npm/ini/commit/a2c5da86604bc2238fe393c5ff083bf23a9910eb"><code>a2c5da8</code></a> 1.3.8</li> <li><a href="https://github.com/npm/ini/commit/af5c6bb5dca6f0248c153aa87e25bddfc515ff6e"><code>af5c6bb</code></a> Do not use Object.create(null)</li> <li><a href="https://github.com/npm/ini/commit/8b648a1ac49e1b3b7686ea957e0b95e544bc6ec1"><code>8b648a1</code></a> don't test where our devdeps don't even work</li> <li><a href="https://github.com/npm/ini/commit/c74c8af35f32b801a7e82a8309eab792a95932f6"><code>c74c8af</code></a> 1.3.7</li> <li><a href="https://github.com/npm/ini/commit/024b8b55ac1c980c6225607b007714c54eb501ba"><code>024b8b5</code></a> update deps, add linting</li> <li><a href="https://github.com/npm/ini/commit/032fbaf5f0b98fce70c8cc380e0d05177a9c9073"><code>032fbaf</code></a> Use Object.create(null) to avoid default object property hazards</li> <li><a href="https://github.com/npm/ini/commit/2da90391ef70db41d10f013e3a87f9a8c5d01a72"><code>2da9039</code></a> 1.3.6</li> <li><a href="https://github.com/npm/ini/commit/cfea636f534b5ca7550d2c28b7d1a95d936d56c6"><code>cfea636</code></a> better git push script, before publish instead of after</li> <li><a href="https://github.com/npm/ini/commit/56d2805e07ccd94e2ba0984ac9240ff02d44b6f1"><code>56d2805</code></a> do not allow invalid hazardous string as section name</li> <li>See full diff in <a href="https://github.com/isaacs/ini/compare/v1.3.5...v1.3.8">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~isaacs">isaacs</a>, a new releaser for ini since your current version.</p> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/facebookresearch/mmf/network/alerts). </details> Pull Request resolved: https://github.com/facebookresearch/mmf/pull/713 Reviewed By: ytsheng Differential Revision: D25615831 Pulled By: vedanuj fbshipit-source-id: b7d5f6e3564cd40c883846698fd0831b081bf616 | 18 December 2020, 19:26:48 UTC |
| a2314dd | John Knox | 10 December 2020, 13:53:51 UTC | [staticdocs][codemod] Enable code snippets in docusaurus sites Summary: Lets you embed code in markdown with the following syntax: ```` ```js file=../../some-file.js start=start_marker end=end_marker ``` ```` Reviewed By: nikoant Differential Revision: D25460060 fbshipit-source-id: 29e242d6d896e7c8256371483d72104889b9daa9 | 10 December 2020, 13:55:36 UTC |
| 14ba005 | John Knox | 08 December 2020, 12:18:10 UTC | [codemod][staticdocs] Enable fine-grained internal content Summary: This preset includes a remark plugin to allow you to import markdown files from `.../fb/...` directories. Previously such imports would cause the OSS site to fail to build, but with this it's safe. This means you can embed internal-only parts of pages, by transcluding other markdown files, e.g: ``` import InternalSection from './fb/internal-section.md'; # Public Section some public stuff <InternalSection /> # Another Public Section ... ``` When building the public site variant, files imported from `../fb/..` will be skipped. More instructions [here](https://www.internalfb.com/intern/wiki/Static_Docs/Internal-Only_Content/) The preset delegates to `preset-classic` for everything else. Reviewed By: snarkmaster Differential Revision: D25370009 fbshipit-source-id: 36ea3000f0342a82749316cc8df19ce30ccf4537 | 08 December 2020, 12:19:50 UTC |
| ec5c8b5 | Sasha Sheng | 08 December 2020, 02:20:13 UTC | [feat] allow passing in model's file path for from_pretrained (#677) Summary: - from_pretrained to allow a filepath - add test Pull Request resolved: https://github.com/facebookresearch/mmf/pull/677 Reviewed By: apsdehal Differential Revision: D25167229 Pulled By: ytsheng fbshipit-source-id: 2bc976e6239b4d6bd23b7e0b9f3d8495e9381d2d | 08 December 2020, 02:21:10 UTC |
| f3d3da4 | Sasha Sheng | 08 December 2020, 00:40:01 UTC | [fix, doc] add image feature extraction tutorial + small fixes (#673) Summary: - fix link errors in the documentation - add image feature extraction tutorials Pull Request resolved: https://github.com/facebookresearch/mmf/pull/673 Reviewed By: apsdehal Differential Revision: D24967971 Pulled By: ytsheng fbshipit-source-id: d7302fded6a3ed1dc2b8dd2fe65efea43c7bc11d | 08 December 2020, 00:41:05 UTC |
| 40b63e7 | rizavelioglu | 04 December 2020, 19:05:34 UTC | [fix] Typo in LMDBConversion (#699) Summary: Fix typo when extracting image features from .mdb file to .npy files. **Fix:** In **Line#95** image height is set to be the image width, which is wrong: https://github.com/facebookresearch/mmf/blob/518a5a675586e4dc1b415a52a8a80c75edfc2960/tools/scripts/features/lmdb_conversion.py#L95-L96 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/699 Reviewed By: apsdehal Differential Revision: D25219613 Pulled By: vedanuj fbshipit-source-id: 3735ae890fcbba27cbd206086d582224866b051e | 04 December 2020, 19:06:37 UTC |
| a221f06 | Sasha Sheng | 02 December 2020, 23:59:58 UTC | [fix,feat] update the state_key based on module (#675) Summary: - Allow for module-wise state dict key update - Make use of the `_register_load_state_dict_pre_hook` to update the key of the state dict - opted for this approach because recursion is already being implemented in the load_state_dict function and therefore I think there is no need to re-implement recursion. Better to make use of the pytorch implementation. - Slightly cleaner fix compared to this fix: [664](https://github.com/facebookresearch/mmf/pull/664) - Some documentation clean up Pull Request resolved: https://github.com/facebookresearch/mmf/pull/675 Reviewed By: vedanuj Differential Revision: D24714619 Pulled By: ytsheng fbshipit-source-id: ccbf85c9aedae4bded3234d9b178e6b34241bbc3 | 03 December 2020, 00:01:06 UTC |
| 6aec3eb | Sasha Sheng | 30 November 2020, 19:25:17 UTC | [fix] Update README.md to fix typo (#698) Summary: * fix typo Pull Request resolved: https://github.com/facebookresearch/mmf/pull/698 Reviewed By: vedanuj Differential Revision: D25180465 Pulled By: ytsheng fbshipit-source-id: d53009df5118841f098316e9af6f4b4ab72ee500 | 30 November 2020, 19:26:40 UTC |
| 518a5a6 | Xiaomeng Yang | 19 November 2020, 19:20:08 UTC | [fix] Fix test_nucleus_sampling after pytorch multinomial fix (#684) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/684 Fix test_nucleus_sampling after pytorch multinomial fix Reviewed By: vedanuj Differential Revision: D24997596 fbshipit-source-id: 27b98d1289a36d151abf60e3a2ad0fb150da4139 | 19 November 2020, 19:21:18 UTC |
| 8ff3f56 | Vedanuj Goswami | 18 November 2020, 20:12:25 UTC | [fix] Upgrade actions workflows to use miniconda@v2 (#690) Summary: Github Actions workflows are broken due to deprecation of `set-env` and `add-path` commands. We need to upgrade to newer version of miniconda. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/690 Test Plan: Example broken tests : https://github.com/facebookresearch/mmf/actions/runs/368665121 Fixed in this PR. Reviewed By: apsdehal Differential Revision: D25052542 Pulled By: vedanuj fbshipit-source-id: 3909adba03669c8e22d7ed45bfe052153b0ea67b | 18 November 2020, 20:13:57 UTC |
| 2a0c567 | Vedanuj Goswami | 11 November 2020, 18:49:59 UTC | [chores] Upgrade transformers dependency for mmf (#666) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/666 - Includes changes for compatibility with transformers 3.4.0 - BertLayerNorm is replaced with LayerNorm from torch.nn - Roberta classes are replicated in the latest version. So we need to monkey patch all of them. - BertEmbedding has a new position_id variable, due to which existing checkpoints have to be loaded with strict=False. This doesn't effect the models, instead of generating position ids it is just an added variable. In most of our models its a no-op. - Additional properties need to be ignored in the `PreTrainedModel` class. Reviewed By: ytsheng Differential Revision: D24599639 fbshipit-source-id: ca915227fc8e52d7624a1caedb09c8c57764a3fb | 11 November 2020, 18:51:12 UTC |
| d9ab064 | John Knox | 05 November 2020, 11:06:31 UTC | [mmf][website] Use static docs helpers Summary: Switch to using the helper lib for better tracking, code modding, and removing redundancy. Docs: https://www.internalfb.com/intern/wiki/Static_Docs/Internal-Only_Content/ Reviewed By: mweststrate Differential Revision: D24704871 fbshipit-source-id: fc37ee18c7517eb3107a97e4d130420bca0d68e7 | 05 November 2020, 11:07:37 UTC |
| 5cbda90 | Amanpreet Singh | 02 November 2020, 22:09:54 UTC | [feat,fix] Stronger checkpoint validation, fix missing keys (#664) Summary: - This PR aims to add stronger checkpoint validations to MMF - It will fail the run if there are missing keys in the checkpoint - It will log unexpected keys as warnings - Using this, we fix multiple missing keys in pythia, m4c Fixes https://github.com/facebookresearch/mmf/issues/661 #652 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/664 Reviewed By: vedanuj Differential Revision: D24636673 Pulled By: apsdehal fbshipit-source-id: d54b2495218984b6273276d4f5c86efba89e4d30 | 02 November 2020, 22:10:56 UTC |
| 3ae71ec | Vedanuj Goswami | 30 October 2020, 03:00:53 UTC | [feat] Add optimizer state sharding (ZeRO) (#636) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/636 Adding Zero optimizer state sharding to MMF from [fairscale](https://github.com/facebookresearch/fairscale) library. Added to tutorials It can used with this option : `optimizer.enable_state_sharding=True` Reviewed By: apsdehal Differential Revision: D24317858 fbshipit-source-id: d3b03dd2cbdfc3be7beaa0ee441a62151813df33 | 30 October 2020, 03:02:03 UTC |
| d2f00bf | Sasha Sheng | 30 October 2020, 00:13:53 UTC | [fix] use torch current device rather than using registry (#659) Summary: * Minor fix so that we remove the registry current device dependency, and instead use torch. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/659 Reviewed By: apsdehal Differential Revision: D24542722 Pulled By: ytsheng fbshipit-source-id: 29bcd4318e319a1bc29b4bf4549830b441a0008b | 30 October 2020, 00:15:24 UTC |
| a82fcef | Vedanuj Goswami | 24 October 2020, 19:52:07 UTC | [enhancement] Add freeze backend option to MMFT (#642) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/642 Add option to freeze backend of MMF Transformer model Reviewed By: apsdehal Differential Revision: D24381850 fbshipit-source-id: be550e25cececa3603208c6bd8df7aeabae252ae | 24 October 2020, 19:53:29 UTC |
| 540bd91 | Vedanuj Goswami | 22 October 2020, 18:09:17 UTC | [chores] Move mmf workflows to use manifold (#656) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/656 - Remove support for Gluster, move all flows to use manifold - Enable proxy in fbl workflows - Remove unused launch params - Fix workflow naming Reviewed By: BruceChaun Differential Revision: D24471398 fbshipit-source-id: c5811ad929af75cf9fca82ba808e6440506280e2 | 22 October 2020, 18:10:57 UTC |
| f658adb | Jun Chen | 22 October 2020, 17:20:18 UTC | [fix] Fix text encoder output if the output is tensor (#655) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/655 Some text encoder outputs return a single tensor (say with dimension `batch_size x out_dim`). Additional condition checks are added to make sure this case is not affected. Reviewed By: vedanuj Differential Revision: D24456401 fbshipit-source-id: 09986304a1bd9cecd7dc743395829e5be7497c9b | 22 October 2020, 17:21:46 UTC |
| 0206f23 | Vedanuj Goswami | 21 October 2020, 06:19:29 UTC | [fix] Fix text encoder output for BertModelJIT (#649) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/649 Output of BertModelJit is a 3 tuple. Fix to pick the pooled output for the fusion models. Behavior changed after we fixed `bert-*` models to use `BertModelJit` in D24038872 (https://github.com/facebookresearch/mmf/commit/5be74f067382272d6f6729156659d7f5a00be187) Reviewed By: n-zhang Differential Revision: D24440904 fbshipit-source-id: a9c1e5dd7f58ef1dd1d569308844a3ab7d1e64c3 | 21 October 2020, 06:21:05 UTC |
| 70f36eb | Amanpreet Singh | 21 October 2020, 06:09:36 UTC | [fix] Allow struct=True configs for MMFT encoders,pretrained Pythia (#644) Summary: - If modality is missing encoder key, MMFT after recent fix won't break down - This allows using configs without specifying encoder key and if they are in struct mode which is what MMF gives when ran from command line Pull Request resolved: https://github.com/facebookresearch/mmf/pull/644 Test Plan: Tested with audio video MMFT Reviewed By: vedanuj Differential Revision: D24405513 Pulled By: apsdehal fbshipit-source-id: 3a6e588264bd39426178a8f16e582262a1d7cb1d | 21 October 2020, 06:11:00 UTC |
| d8f9592 | Jun Chen | 20 October 2020, 04:28:34 UTC | [fix] Change format of concatenating image and text as bert input in MMBT (#622) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/622 THe previous version of concatenation of image and text tokens has format `<cls> image_token <sep> <cls> text_tokens <sep> <pad>`, but there is a bug in code -- the `<sep>` token behind `image_token` might be `<pad>` because it simply used the last token in `input_ids`. We can easily see `bert_processor` convert input text into tokens which has format `<cls> tokens <sep> <pad>`. Apart from resolving the aforementioned issue, this diff also converts to a simpler format `<cls> image_token <sep> text_tokens <sep> <pad>`. So based on previous version, modifications are 1. Extract the last non-masked token, which is `<sep>` as `modal_end_token` 2. Remove `<cls>` token at the beginning of text input ids and append last token at the end to ensure it has the same sequence length 3. Fix `input_mask` by truncating the first token and appending 0 at the end Reviewed By: apsdehal Differential Revision: D24164033 fbshipit-source-id: 04246d64ebd483e6809721245fac9dc0ee20cff8 | 20 October 2020, 04:30:14 UTC |
| 9af5b2c | Amanpreet Singh | 19 October 2020, 19:32:37 UTC | [enhancement] Support more modalities in MMFT (#576) Summary: - Generalizes MMFT to more modalities - Modality agnostic initialization, checks and forward for other modalities Pull Request resolved: https://github.com/facebookresearch/mmf/pull/576 Test Plan: Tested on a video-audio dataset Reviewed By: vedanuj Differential Revision: D23969975 Pulled By: apsdehal fbshipit-source-id: 9d45a2d9c50563433c230b6a9c59f38af7deaa59 | 19 October 2020, 19:33:59 UTC |
| 97f3c7d | Vedanuj Goswami | 19 October 2020, 19:21:54 UTC | [fix] Fix in_dim for movie_mcan model (#638) Summary: `in_dim` needs to be added to the config after the change in PR https://github.com/facebookresearch/mmf/issues/564. Fixing that. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/638 Reviewed By: apsdehal Differential Revision: D24388070 Pulled By: vedanuj fbshipit-source-id: dab1c2ce14ea168b8932193950bdae30ebbbf9d7 | 19 October 2020, 19:23:34 UTC |
| c0c834e | Amanpreet Singh | 19 October 2020, 08:38:00 UTC | [feat] Make top level loss classes torchscriptable (#631) Summary: - We weren't doing init_losses in the tests so we didn't notice this - Changes make top level loss function torchscriptable so that if losses are empty, this doesn't cause any issues Pull Request resolved: https://github.com/facebookresearch/mmf/pull/631 Test Plan: Enable init_losses in mmbt tests Reviewed By: vedanuj Differential Revision: D24328059 Pulled By: apsdehal fbshipit-source-id: 063f4938e7475f91904149771824c77eb2d22509 | 19 October 2020, 08:39:30 UTC |
| ae2c8e1 | Amanpreet Singh | 19 October 2020, 07:38:36 UTC | [fix] Issues with encoder update, add tests (#634) Summary: - I introduced some issues with recent encoder update. This PR aims to fix that. - This PR also adds tests to make sure this doesn't happen in future. - Tests only tests initialization as of now, can be build upon in future. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/634 Test Plan: Tests have been added Reviewed By: vedanuj Differential Revision: D24356702 Pulled By: apsdehal fbshipit-source-id: 815a68096db7c802178fdbe28aa6c22601a2c999 | 19 October 2020, 07:39:51 UTC |
| dcc1a95 | Amanpreet Singh | 19 October 2020, 07:22:18 UTC | [feat] Add fp16 support to MMFTrainer (#571) Summary: This PR adds support for fp16 to MMF. Extensive benchmarking on different models and different datasets shows that fp16 runs are comparable in performance but quite faster on longer runs. ## Task Items - [x] Save and load state_dict of scaler into/from checkpoint - [x] fp16 on validation/other loops as well - [x] grad scaling on clip grad - [x] tests - [x] benchmark on (i) TextVQA/M4C (ii) VisualBERT/VQA2 (iii) MMBT/HM - [x] docs ## Benchmarking numbers ### MMBT on Hateful Memes Reported metrics are ROC-AUC on validation set and time taken to complete the run | fp16 | Enabled | Disabled | |----------------------|----------------|----------------| | bs128.lr5e-5.mu22000 | 69.72/3h24m28s | 70.14/3h54m55s | | bs64.lr5e-5.mu22000 | 68.47/2h34m38s | 66.56/2h22m49s | ### VisualBERT on VQA2 Reported metrics are VQA accuracy on validation set and time taken to complete the run. | fp16 | Enabled | Disabled | |----------------------|-----------------|----------------| | bs128.lr5e-5.mu88000 | 66.08/05h09m22s | 65.89/7h28m46s | | bs64.lr5e-5.mu88000 | 66.82/04h14m10s | 66.69/6h04m30s | ### M4C on TextVQA Reported metrics are TextVQA accuracy on validation set and time taken to complete the run | fp16 | Enabled | Disabled | |---------------|----------------|-----------------| | bs128.mu24000 | 38.7/01h35m10s | 39.01/01h42m06s | Pull Request resolved: https://github.com/facebookresearch/mmf/pull/571 Reviewed By: vedanuj Differential Revision: D24240531 Pulled By: apsdehal fbshipit-source-id: c704602dea7f229ebee129841477ba2add74ed72 | 19 October 2020, 07:24:32 UTC |
| 6980fe0 | Amanpreet Singh | 16 October 2020, 01:52:04 UTC | [fix] Parameter list regression in build_optimizers (#630) Summary: - With recent change of https://github.com/facebookresearch/mmf/issues/580, there is a regression for models which don't have `get_optimizer_parameters` or don't return param groups. https://github.com/facebookresearch/mmf/issues/580 specifically expects param groups to be present. - This PR fixes this regression and add tests for optimizers so that this doesn't happen again Pull Request resolved: https://github.com/facebookresearch/mmf/pull/630 Test Plan: Tested added for two type of parameter groups returned from the model Reviewed By: ronghanghu Differential Revision: D24325727 Pulled By: apsdehal fbshipit-source-id: 9245408b19323ee6cb2adc1f1eed9182841f5dc4 | 16 October 2020, 01:53:43 UTC |
| fa195d2 | Amanpreet Singh | 15 October 2020, 22:19:22 UTC | [feat] Introduce QC tests for missing init and empty folders (#616) Summary: - Missing inits cause problem with user dir and packaging - The tests check for missing init in folders with python files - Also, checks for empty folders - Furthers, checks on all MMF folders can be added easily Pull Request resolved: https://github.com/facebookresearch/mmf/pull/616 Test Plan: Tested locally and fixed the issues found TODO: Need to test internally as well. Reviewed By: vedanuj Differential Revision: D24200765 Pulled By: apsdehal fbshipit-source-id: 40a12f3be0c48a4279f3946c3f008fa4f8dfcb85 | 15 October 2020, 22:20:55 UTC |
| 5be74f0 | Amanpreet Singh | 15 October 2020, 20:02:15 UTC | [fix] Allow Transformer layers to be used without monkey patching Summary: This involves using BertModelJit to initialize the pretrained model instead of normal one and then also adding init functions of those methods so that all layers are jit compatible. Reviewed By: vedanuj Differential Revision: D24038872 fbshipit-source-id: c573f6b98754ce28538fe770477a8679b736e9d4 | 15 October 2020, 20:04:01 UTC |
| bd1822d | Vedanuj Goswami | 15 October 2020, 07:59:52 UTC | [chores, refactor] Enable torchscript tests internally, refactor (#619) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/619 - Enable torchscript tests internally by enabling proxy - Remove code duplication in torchscript tests Reviewed By: apsdehal Differential Revision: D24213541 fbshipit-source-id: 48f6e8265fcccdc7a1c5b4d40c1075a5d22fae0c | 15 October 2020, 08:01:09 UTC |
| 88a836a | Amanpreet Singh | 15 October 2020, 05:58:52 UTC | [feat] Add encoder registry and differentiate encoder factory (#628) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/628 This diff aims to smoothen the process of adding new encoders to MMF. This is also specifically needed for FB internal encoders which are hard to registry while not being part of OSS without registry. This will also help with keeping sanity in MMFTransformer and generic modalities diff. This diff also converts the older feature encoders to the factory which is what they technically are. This backwards compatible change as the build api still looks same. Though if someone is directly importing the encoder, this will break their code. But this refactor is much needed at this time. We also add `build_encoder` function which handles both structured config and normal config patterns. We hope to make it more streamlined in future when we make efforts towards Hydra integration. Reviewed By: vedanuj Differential Revision: D24300517 fbshipit-source-id: 3bc68de398ad397fdf8fa5cf39a261691cba31d2 | 15 October 2020, 06:00:15 UTC |
| f35b266 | Vedanuj Goswami | 15 October 2020, 01:56:20 UTC | [chores] Remove cpu and windows build from CircleCI tests (#618) Summary: - Remove CPU tests and Windows Build test from CircleCI as they are already covered in Actions - Some CircleCI CPU tests were flaky - These tests also require high resources(class `large`) that causes forked repo builds to fail Pull Request resolved: https://github.com/facebookresearch/mmf/pull/618 Test Plan: Check if tests run properly Reviewed By: apsdehal Differential Revision: D24227244 Pulled By: vedanuj fbshipit-source-id: dbac96d4abfc57468b44e211a62523737665d336 | 15 October 2020, 01:57:40 UTC |
| ee9cade | Meng Qi | 14 October 2020, 21:13:39 UTC | refactor vilbert to make it compatible for torchscript (#591) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/591 Refactor to make vilbert compatible to torchscript Similar changes are done in [our implementation of Vilbert](https://www.internalfb.com/intern/diffusion/FBS/browsefile/master/fbcode/assistant/mmu/multimodal_bert/vilbert/model.py). Reference diff: D18892981 Reviewed By: vedanuj Differential Revision: D23923842 fbshipit-source-id: a7c3736408e6da828373b1c7a4db94189169c83b | 14 October 2020, 21:17:09 UTC |
| 058e713 | Anton Nikolaev | 14 October 2020, 11:44:50 UTC | [staticdocs] Update docusaurus plugin for Static Docs projects Summary: Plugin update is required to fix hit counter and auto-redirect from public site on Chrome 85+. It will also enable auto-redirect from staticdocs.thefacebook.com to internalfb.com/intern/staticdocs to ensure intern sidebar is visible when documentation is browsed internally. Reviewed By: dkgi Differential Revision: D24281980 fbshipit-source-id: 2614b4228d2df164981cee437952058684575a23 | 14 October 2020, 11:46:59 UTC |
| 7ce17a5 | Amanpreet Singh | 14 October 2020, 01:10:08 UTC | [feat] Enable torchscript on full VisualBERT model (#624) Summary: - This removes opinions on module getting torchscripted rather than the full model. This will also enable us to keep the signature of Dict[str, Tensor] same for all of the MMF models - Modifies code flow according to scripting model and pretraining model conditions - Add helps function around getattr - Modifies the test to play nice with new API which looks better Pull Request resolved: https://github.com/facebookresearch/mmf/pull/624 Test Plan: Modifies VisualBERT tests for TorchScript to script and test the whole model Reviewed By: vedanuj Differential Revision: D24267966 Pulled By: apsdehal fbshipit-source-id: c3ffa2934d64bf58739df9c4910fddd0a89684c6 | 14 October 2020, 01:11:16 UTC |
| 78d908d | Amanpreet Singh | 13 October 2020, 18:26:12 UTC | [fix] Missing in_dim to pythia image encoder (#625) Summary: This is required and should have been passed in original commit to modify to config. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/625 Test Plan: Tested on pythia model Reviewed By: vedanuj Differential Revision: D24279546 Pulled By: apsdehal fbshipit-source-id: 711a62930f638b3c8b8e2af93aa561f6c8f8e648 | 13 October 2020, 18:27:35 UTC |
| 6f14d31 | Amanpreet Singh | 12 October 2020, 21:28:18 UTC | [feat] Add init dataclasses for mmbt and encoders (#565) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/565 As step one of FAIM integration, we allow building our models from config so that the models are purely decoupled from configuration and users know what the model expects. We first do this for the MMBT model. - Adds configs for MMBT and respective encoders. - Also adds from_params method to the BaseModel class so that args initialization is possible - Updates build method to support passing of direct config object - Add Config class to BaseModel as well and update typings There is an issue with OmegaConf that doesn't let us use Union in structured configs. Take a look at https://github.com/omry/omegaconf/issues/144 Reviewed By: vedanuj Differential Revision: D23699688 fbshipit-source-id: 37020346ce820207eb41b7bd43c8fba579b436d5 | 12 October 2020, 21:29:40 UTC |
| c8264bd | Amanpreet Singh | 12 October 2020, 20:54:56 UTC | [refactor] Use config object for ImageFeatureEncoder init (#564) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/564 There has been a mismatch between ImageFeatureEncoder and ImageEncoder signatures which doesn't allow us to create same config types for them even though that's how they are used. This is confusing and should be same. This diff aims to fix this discrepancy. Reviewed By: vedanuj Differential Revision: D23712213 fbshipit-source-id: b6d3ad56a82727c08f3a53b8ee2b77e6af9e7a17 | 12 October 2020, 20:56:16 UTC |
| a5fc02b | Vedanuj Goswami | 12 October 2020, 02:26:27 UTC | [feat] Add masked visual genome dataset (#602) Summary: - Adds visual genome masked dataset - Fixes https://github.com/facebookresearch/mmf/issues/566, for pretraining on visual genome - Fix for loading samples without `question_tokens` - Fix for loading visual genome feature files Pull Request resolved: https://github.com/facebookresearch/mmf/pull/602 Test Plan: Test run lxmert model with visual genome Reviewed By: apsdehal Differential Revision: D24108712 Pulled By: vedanuj fbshipit-source-id: 5ae041a2ce4dcdf914fdecc19466729d445237da | 12 October 2020, 02:27:50 UTC |
| e83f626 | Vedanuj Goswami | 11 October 2020, 02:28:34 UTC | [fix] Fix optimizer params to include all modules for transformer models (#620) Summary: Recent changes in PR https://github.com/facebookresearch/mmf/issues/568 moved the `pooler` layer outside classifier and was not added to optimizer params group. Check added in https://github.com/facebookresearch/mmf/issues/580 helped to catch this. Thanks ronghanghu ! - PR refactors the code in `get_optimizer_parameters_for_bert` - **All** layers other than classifier will have LR of `(lr * finetune_lr_multiplier)` when model head is not of `pretraining` type. - Remove unnecessary setting LR when `finetune_lr_multiplier == 1` Pull Request resolved: https://github.com/facebookresearch/mmf/pull/620 Test Plan: - Checked with MMF Transformer, MMBT, VisualBERT Reviewed By: apsdehal Differential Revision: D24237064 Pulled By: vedanuj fbshipit-source-id: 70f1321d6865bde2f636c46b9d7c00a7bf6667ab | 11 October 2020, 02:29:53 UTC |
| bad7a4d | Jun Chen | 09 October 2020, 19:26:22 UTC | [feat] Add DoCNN as text encoder in MMF Summary: Add DoCNN in TextEncoder. The embedding matrix uses the one in pytext transformer. And support to customize `conv_filters` and `mlp_sizes` in yaml config. # Performance | concat_bert model with text encoder | test/ap | test/accuracy | | docnn+pytext | **0.8998** | **0.8407** | | docnn | 0.8930 | 0.8346 | | xlm_roberta_base | 0.8843 | 0.8336 | Reviewed By: apsdehal Differential Revision: D23933438 fbshipit-source-id: 1cda7920e737d3342a6abf0afa54ca26660496d3 | 09 October 2020, 19:28:06 UTC |
