Skip to main content

Browse the archive

https://github.com/facebookresearch/pythia
31 March 2023, 09:22:40 UTC
sort by:
Revision Author Date Message Commit Date
8ead66a Amanpreet Singh 10 December 2020, 02:12:16 UTC [feat] Hydra integration Attempt to test out hydra 10 December 2020, 02:12:16 UTC
14ba005 John Knox 08 December 2020, 12:18:10 UTC [codemod][staticdocs] Enable fine-grained internal content Summary: This preset includes a remark plugin to allow you to import markdown files from `.../fb/...` directories. Previously such imports would cause the OSS site to fail to build, but with this it's safe. This means you can embed internal-only parts of pages, by transcluding other markdown files, e.g: ``` import InternalSection from './fb/internal-section.md'; # Public Section some public stuff <InternalSection /> # Another Public Section ... ``` When building the public site variant, files imported from `../fb/..` will be skipped. More instructions [here](https://www.internalfb.com/intern/wiki/Static_Docs/Internal-Only_Content/) The preset delegates to `preset-classic` for everything else. Reviewed By: snarkmaster Differential Revision: D25370009 fbshipit-source-id: 36ea3000f0342a82749316cc8df19ce30ccf4537 08 December 2020, 12:19:50 UTC
ec5c8b5 Sasha Sheng 08 December 2020, 02:20:13 UTC [feat] allow passing in model's file path for from_pretrained (#677) Summary: - from_pretrained to allow a filepath - add test Pull Request resolved: https://github.com/facebookresearch/mmf/pull/677 Reviewed By: apsdehal Differential Revision: D25167229 Pulled By: ytsheng fbshipit-source-id: 2bc976e6239b4d6bd23b7e0b9f3d8495e9381d2d 08 December 2020, 02:21:10 UTC
f3d3da4 Sasha Sheng 08 December 2020, 00:40:01 UTC [fix, doc] add image feature extraction tutorial + small fixes (#673) Summary: - fix link errors in the documentation - add image feature extraction tutorials Pull Request resolved: https://github.com/facebookresearch/mmf/pull/673 Reviewed By: apsdehal Differential Revision: D24967971 Pulled By: ytsheng fbshipit-source-id: d7302fded6a3ed1dc2b8dd2fe65efea43c7bc11d 08 December 2020, 00:41:05 UTC
40b63e7 rizavelioglu 04 December 2020, 19:05:34 UTC [fix] Typo in LMDBConversion (#699) Summary: Fix typo when extracting image features from .mdb file to .npy files. **Fix:** In **Line#95** image height is set to be the image width, which is wrong: https://github.com/facebookresearch/mmf/blob/518a5a675586e4dc1b415a52a8a80c75edfc2960/tools/scripts/features/lmdb_conversion.py#L95-L96 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/699 Reviewed By: apsdehal Differential Revision: D25219613 Pulled By: vedanuj fbshipit-source-id: 3735ae890fcbba27cbd206086d582224866b051e 04 December 2020, 19:06:37 UTC
a221f06 Sasha Sheng 02 December 2020, 23:59:58 UTC [fix,feat] update the state_key based on module (#675) Summary: - Allow for module-wise state dict key update - Make use of the `_register_load_state_dict_pre_hook` to update the key of the state dict - opted for this approach because recursion is already being implemented in the load_state_dict function and therefore I think there is no need to re-implement recursion. Better to make use of the pytorch implementation. - Slightly cleaner fix compared to this fix: [664](https://github.com/facebookresearch/mmf/pull/664) - Some documentation clean up Pull Request resolved: https://github.com/facebookresearch/mmf/pull/675 Reviewed By: vedanuj Differential Revision: D24714619 Pulled By: ytsheng fbshipit-source-id: ccbf85c9aedae4bded3234d9b178e6b34241bbc3 03 December 2020, 00:01:06 UTC
6aec3eb Sasha Sheng 30 November 2020, 19:25:17 UTC [fix] Update README.md to fix typo (#698) Summary: * fix typo Pull Request resolved: https://github.com/facebookresearch/mmf/pull/698 Reviewed By: vedanuj Differential Revision: D25180465 Pulled By: ytsheng fbshipit-source-id: d53009df5118841f098316e9af6f4b4ab72ee500 30 November 2020, 19:26:40 UTC
518a5a6 Xiaomeng Yang 19 November 2020, 19:20:08 UTC [fix] Fix test_nucleus_sampling after pytorch multinomial fix (#684) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/684 Fix test_nucleus_sampling after pytorch multinomial fix Reviewed By: vedanuj Differential Revision: D24997596 fbshipit-source-id: 27b98d1289a36d151abf60e3a2ad0fb150da4139 19 November 2020, 19:21:18 UTC
8ff3f56 Vedanuj Goswami 18 November 2020, 20:12:25 UTC [fix] Upgrade actions workflows to use miniconda@v2 (#690) Summary: Github Actions workflows are broken due to deprecation of `set-env` and `add-path` commands. We need to upgrade to newer version of miniconda. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/690 Test Plan: Example broken tests : https://github.com/facebookresearch/mmf/actions/runs/368665121 Fixed in this PR. Reviewed By: apsdehal Differential Revision: D25052542 Pulled By: vedanuj fbshipit-source-id: 3909adba03669c8e22d7ed45bfe052153b0ea67b 18 November 2020, 20:13:57 UTC
2a0c567 Vedanuj Goswami 11 November 2020, 18:49:59 UTC [chores] Upgrade transformers dependency for mmf (#666) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/666 - Includes changes for compatibility with transformers 3.4.0 - BertLayerNorm is replaced with LayerNorm from torch.nn - Roberta classes are replicated in the latest version. So we need to monkey patch all of them. - BertEmbedding has a new position_id variable, due to which existing checkpoints have to be loaded with strict=False. This doesn't effect the models, instead of generating position ids it is just an added variable. In most of our models its a no-op. - Additional properties need to be ignored in the `PreTrainedModel` class. Reviewed By: ytsheng Differential Revision: D24599639 fbshipit-source-id: ca915227fc8e52d7624a1caedb09c8c57764a3fb 11 November 2020, 18:51:12 UTC
d9ab064 John Knox 05 November 2020, 11:06:31 UTC [mmf][website] Use static docs helpers Summary: Switch to using the helper lib for better tracking, code modding, and removing redundancy. Docs: https://www.internalfb.com/intern/wiki/Static_Docs/Internal-Only_Content/ Reviewed By: mweststrate Differential Revision: D24704871 fbshipit-source-id: fc37ee18c7517eb3107a97e4d130420bca0d68e7 05 November 2020, 11:07:37 UTC
5cbda90 Amanpreet Singh 02 November 2020, 22:09:54 UTC [feat,fix] Stronger checkpoint validation, fix missing keys (#664) Summary: - This PR aims to add stronger checkpoint validations to MMF - It will fail the run if there are missing keys in the checkpoint - It will log unexpected keys as warnings - Using this, we fix multiple missing keys in pythia, m4c Fixes https://github.com/facebookresearch/mmf/issues/661 #652 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/664 Reviewed By: vedanuj Differential Revision: D24636673 Pulled By: apsdehal fbshipit-source-id: d54b2495218984b6273276d4f5c86efba89e4d30 02 November 2020, 22:10:56 UTC
3ae71ec Vedanuj Goswami 30 October 2020, 03:00:53 UTC [feat] Add optimizer state sharding (ZeRO) (#636) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/636 Adding Zero optimizer state sharding to MMF from [fairscale](https://github.com/facebookresearch/fairscale) library. Added to tutorials It can used with this option : `optimizer.enable_state_sharding=True` Reviewed By: apsdehal Differential Revision: D24317858 fbshipit-source-id: d3b03dd2cbdfc3be7beaa0ee441a62151813df33 30 October 2020, 03:02:03 UTC
d2f00bf Sasha Sheng 30 October 2020, 00:13:53 UTC [fix] use torch current device rather than using registry (#659) Summary: * Minor fix so that we remove the registry current device dependency, and instead use torch. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/659 Reviewed By: apsdehal Differential Revision: D24542722 Pulled By: ytsheng fbshipit-source-id: 29bcd4318e319a1bc29b4bf4549830b441a0008b 30 October 2020, 00:15:24 UTC
a82fcef Vedanuj Goswami 24 October 2020, 19:52:07 UTC [enhancement] Add freeze backend option to MMFT (#642) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/642 Add option to freeze backend of MMF Transformer model Reviewed By: apsdehal Differential Revision: D24381850 fbshipit-source-id: be550e25cececa3603208c6bd8df7aeabae252ae 24 October 2020, 19:53:29 UTC
540bd91 Vedanuj Goswami 22 October 2020, 18:09:17 UTC [chores] Move mmf workflows to use manifold (#656) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/656 - Remove support for Gluster, move all flows to use manifold - Enable proxy in fbl workflows - Remove unused launch params - Fix workflow naming Reviewed By: BruceChaun Differential Revision: D24471398 fbshipit-source-id: c5811ad929af75cf9fca82ba808e6440506280e2 22 October 2020, 18:10:57 UTC
f658adb Jun Chen 22 October 2020, 17:20:18 UTC [fix] Fix text encoder output if the output is tensor (#655) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/655 Some text encoder outputs return a single tensor (say with dimension `batch_size x out_dim`). Additional condition checks are added to make sure this case is not affected. Reviewed By: vedanuj Differential Revision: D24456401 fbshipit-source-id: 09986304a1bd9cecd7dc743395829e5be7497c9b 22 October 2020, 17:21:46 UTC
0206f23 Vedanuj Goswami 21 October 2020, 06:19:29 UTC [fix] Fix text encoder output for BertModelJIT (#649) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/649 Output of BertModelJit is a 3 tuple. Fix to pick the pooled output for the fusion models. Behavior changed after we fixed `bert-*` models to use `BertModelJit` in D24038872 (https://github.com/facebookresearch/mmf/commit/5be74f067382272d6f6729156659d7f5a00be187) Reviewed By: n-zhang Differential Revision: D24440904 fbshipit-source-id: a9c1e5dd7f58ef1dd1d569308844a3ab7d1e64c3 21 October 2020, 06:21:05 UTC
70f36eb Amanpreet Singh 21 October 2020, 06:09:36 UTC [fix] Allow struct=True configs for MMFT encoders,pretrained Pythia (#644) Summary: - If modality is missing encoder key, MMFT after recent fix won't break down - This allows using configs without specifying encoder key and if they are in struct mode which is what MMF gives when ran from command line Pull Request resolved: https://github.com/facebookresearch/mmf/pull/644 Test Plan: Tested with audio video MMFT Reviewed By: vedanuj Differential Revision: D24405513 Pulled By: apsdehal fbshipit-source-id: 3a6e588264bd39426178a8f16e582262a1d7cb1d 21 October 2020, 06:11:00 UTC
d8f9592 Jun Chen 20 October 2020, 04:28:34 UTC [fix] Change format of concatenating image and text as bert input in MMBT (#622) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/622 THe previous version of concatenation of image and text tokens has format `<cls> image_token <sep> <cls> text_tokens <sep> <pad>`, but there is a bug in code -- the `<sep>` token behind `image_token` might be `<pad>` because it simply used the last token in `input_ids`. We can easily see `bert_processor` convert input text into tokens which has format `<cls> tokens <sep> <pad>`. Apart from resolving the aforementioned issue, this diff also converts to a simpler format `<cls> image_token <sep> text_tokens <sep> <pad>`. So based on previous version, modifications are 1. Extract the last non-masked token, which is `<sep>` as `modal_end_token` 2. Remove `<cls>` token at the beginning of text input ids and append last token at the end to ensure it has the same sequence length 3. Fix `input_mask` by truncating the first token and appending 0 at the end Reviewed By: apsdehal Differential Revision: D24164033 fbshipit-source-id: 04246d64ebd483e6809721245fac9dc0ee20cff8 20 October 2020, 04:30:14 UTC
9af5b2c Amanpreet Singh 19 October 2020, 19:32:37 UTC [enhancement] Support more modalities in MMFT (#576) Summary: - Generalizes MMFT to more modalities - Modality agnostic initialization, checks and forward for other modalities Pull Request resolved: https://github.com/facebookresearch/mmf/pull/576 Test Plan: Tested on a video-audio dataset Reviewed By: vedanuj Differential Revision: D23969975 Pulled By: apsdehal fbshipit-source-id: 9d45a2d9c50563433c230b6a9c59f38af7deaa59 19 October 2020, 19:33:59 UTC
97f3c7d Vedanuj Goswami 19 October 2020, 19:21:54 UTC [fix] Fix in_dim for movie_mcan model (#638) Summary: `in_dim` needs to be added to the config after the change in PR https://github.com/facebookresearch/mmf/issues/564. Fixing that. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/638 Reviewed By: apsdehal Differential Revision: D24388070 Pulled By: vedanuj fbshipit-source-id: dab1c2ce14ea168b8932193950bdae30ebbbf9d7 19 October 2020, 19:23:34 UTC
c0c834e Amanpreet Singh 19 October 2020, 08:38:00 UTC [feat] Make top level loss classes torchscriptable (#631) Summary: - We weren't doing init_losses in the tests so we didn't notice this - Changes make top level loss function torchscriptable so that if losses are empty, this doesn't cause any issues Pull Request resolved: https://github.com/facebookresearch/mmf/pull/631 Test Plan: Enable init_losses in mmbt tests Reviewed By: vedanuj Differential Revision: D24328059 Pulled By: apsdehal fbshipit-source-id: 063f4938e7475f91904149771824c77eb2d22509 19 October 2020, 08:39:30 UTC
ae2c8e1 Amanpreet Singh 19 October 2020, 07:38:36 UTC [fix] Issues with encoder update, add tests (#634) Summary: - I introduced some issues with recent encoder update. This PR aims to fix that. - This PR also adds tests to make sure this doesn't happen in future. - Tests only tests initialization as of now, can be build upon in future. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/634 Test Plan: Tests have been added Reviewed By: vedanuj Differential Revision: D24356702 Pulled By: apsdehal fbshipit-source-id: 815a68096db7c802178fdbe28aa6c22601a2c999 19 October 2020, 07:39:51 UTC
dcc1a95 Amanpreet Singh 19 October 2020, 07:22:18 UTC [feat] Add fp16 support to MMFTrainer (#571) Summary: This PR adds support for fp16 to MMF. Extensive benchmarking on different models and different datasets shows that fp16 runs are comparable in performance but quite faster on longer runs. ## Task Items - [x] Save and load state_dict of scaler into/from checkpoint - [x] fp16 on validation/other loops as well - [x] grad scaling on clip grad - [x] tests - [x] benchmark on (i) TextVQA/M4C (ii) VisualBERT/VQA2 (iii) MMBT/HM - [x] docs ## Benchmarking numbers ### MMBT on Hateful Memes Reported metrics are ROC-AUC on validation set and time taken to complete the run | fp16 | Enabled | Disabled | |----------------------|----------------|----------------| | bs128.lr5e-5.mu22000 | 69.72/3h24m28s | 70.14/3h54m55s | | bs64.lr5e-5.mu22000 | 68.47/2h34m38s | 66.56/2h22m49s | ### VisualBERT on VQA2 Reported metrics are VQA accuracy on validation set and time taken to complete the run. | fp16 | Enabled | Disabled | |----------------------|-----------------|----------------| | bs128.lr5e-5.mu88000 | 66.08/05h09m22s | 65.89/7h28m46s | | bs64.lr5e-5.mu88000 | 66.82/04h14m10s | 66.69/6h04m30s | ### M4C on TextVQA Reported metrics are TextVQA accuracy on validation set and time taken to complete the run | fp16 | Enabled | Disabled | |---------------|----------------|-----------------| | bs128.mu24000 | 38.7/01h35m10s | 39.01/01h42m06s | Pull Request resolved: https://github.com/facebookresearch/mmf/pull/571 Reviewed By: vedanuj Differential Revision: D24240531 Pulled By: apsdehal fbshipit-source-id: c704602dea7f229ebee129841477ba2add74ed72 19 October 2020, 07:24:32 UTC
6980fe0 Amanpreet Singh 16 October 2020, 01:52:04 UTC [fix] Parameter list regression in build_optimizers (#630) Summary: - With recent change of https://github.com/facebookresearch/mmf/issues/580, there is a regression for models which don't have `get_optimizer_parameters` or don't return param groups. https://github.com/facebookresearch/mmf/issues/580 specifically expects param groups to be present. - This PR fixes this regression and add tests for optimizers so that this doesn't happen again Pull Request resolved: https://github.com/facebookresearch/mmf/pull/630 Test Plan: Tested added for two type of parameter groups returned from the model Reviewed By: ronghanghu Differential Revision: D24325727 Pulled By: apsdehal fbshipit-source-id: 9245408b19323ee6cb2adc1f1eed9182841f5dc4 16 October 2020, 01:53:43 UTC
fa195d2 Amanpreet Singh 15 October 2020, 22:19:22 UTC [feat] Introduce QC tests for missing init and empty folders (#616) Summary: - Missing inits cause problem with user dir and packaging - The tests check for missing init in folders with python files - Also, checks for empty folders - Furthers, checks on all MMF folders can be added easily Pull Request resolved: https://github.com/facebookresearch/mmf/pull/616 Test Plan: Tested locally and fixed the issues found TODO: Need to test internally as well. Reviewed By: vedanuj Differential Revision: D24200765 Pulled By: apsdehal fbshipit-source-id: 40a12f3be0c48a4279f3946c3f008fa4f8dfcb85 15 October 2020, 22:20:55 UTC
5be74f0 Amanpreet Singh 15 October 2020, 20:02:15 UTC [fix] Allow Transformer layers to be used without monkey patching Summary: This involves using BertModelJit to initialize the pretrained model instead of normal one and then also adding init functions of those methods so that all layers are jit compatible. Reviewed By: vedanuj Differential Revision: D24038872 fbshipit-source-id: c573f6b98754ce28538fe770477a8679b736e9d4 15 October 2020, 20:04:01 UTC
bd1822d Vedanuj Goswami 15 October 2020, 07:59:52 UTC [chores, refactor] Enable torchscript tests internally, refactor (#619) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/619 - Enable torchscript tests internally by enabling proxy - Remove code duplication in torchscript tests Reviewed By: apsdehal Differential Revision: D24213541 fbshipit-source-id: 48f6e8265fcccdc7a1c5b4d40c1075a5d22fae0c 15 October 2020, 08:01:09 UTC
88a836a Amanpreet Singh 15 October 2020, 05:58:52 UTC [feat] Add encoder registry and differentiate encoder factory (#628) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/628 This diff aims to smoothen the process of adding new encoders to MMF. This is also specifically needed for FB internal encoders which are hard to registry while not being part of OSS without registry. This will also help with keeping sanity in MMFTransformer and generic modalities diff. This diff also converts the older feature encoders to the factory which is what they technically are. This backwards compatible change as the build api still looks same. Though if someone is directly importing the encoder, this will break their code. But this refactor is much needed at this time. We also add `build_encoder` function which handles both structured config and normal config patterns. We hope to make it more streamlined in future when we make efforts towards Hydra integration. Reviewed By: vedanuj Differential Revision: D24300517 fbshipit-source-id: 3bc68de398ad397fdf8fa5cf39a261691cba31d2 15 October 2020, 06:00:15 UTC
f35b266 Vedanuj Goswami 15 October 2020, 01:56:20 UTC [chores] Remove cpu and windows build from CircleCI tests (#618) Summary: - Remove CPU tests and Windows Build test from CircleCI as they are already covered in Actions - Some CircleCI CPU tests were flaky - These tests also require high resources(class `large`) that causes forked repo builds to fail Pull Request resolved: https://github.com/facebookresearch/mmf/pull/618 Test Plan: Check if tests run properly Reviewed By: apsdehal Differential Revision: D24227244 Pulled By: vedanuj fbshipit-source-id: dbac96d4abfc57468b44e211a62523737665d336 15 October 2020, 01:57:40 UTC
ee9cade Meng Qi 14 October 2020, 21:13:39 UTC refactor vilbert to make it compatible for torchscript (#591) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/591 Refactor to make vilbert compatible to torchscript Similar changes are done in [our implementation of Vilbert](https://www.internalfb.com/intern/diffusion/FBS/browsefile/master/fbcode/assistant/mmu/multimodal_bert/vilbert/model.py). Reference diff: D18892981 Reviewed By: vedanuj Differential Revision: D23923842 fbshipit-source-id: a7c3736408e6da828373b1c7a4db94189169c83b 14 October 2020, 21:17:09 UTC
058e713 Anton Nikolaev 14 October 2020, 11:44:50 UTC [staticdocs] Update docusaurus plugin for Static Docs projects Summary: Plugin update is required to fix hit counter and auto-redirect from public site on Chrome 85+. It will also enable auto-redirect from staticdocs.thefacebook.com to internalfb.com/intern/staticdocs to ensure intern sidebar is visible when documentation is browsed internally. Reviewed By: dkgi Differential Revision: D24281980 fbshipit-source-id: 2614b4228d2df164981cee437952058684575a23 14 October 2020, 11:46:59 UTC
7ce17a5 Amanpreet Singh 14 October 2020, 01:10:08 UTC [feat] Enable torchscript on full VisualBERT model (#624) Summary: - This removes opinions on module getting torchscripted rather than the full model. This will also enable us to keep the signature of Dict[str, Tensor] same for all of the MMF models - Modifies code flow according to scripting model and pretraining model conditions - Add helps function around getattr - Modifies the test to play nice with new API which looks better Pull Request resolved: https://github.com/facebookresearch/mmf/pull/624 Test Plan: Modifies VisualBERT tests for TorchScript to script and test the whole model Reviewed By: vedanuj Differential Revision: D24267966 Pulled By: apsdehal fbshipit-source-id: c3ffa2934d64bf58739df9c4910fddd0a89684c6 14 October 2020, 01:11:16 UTC
78d908d Amanpreet Singh 13 October 2020, 18:26:12 UTC [fix] Missing in_dim to pythia image encoder (#625) Summary: This is required and should have been passed in original commit to modify to config. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/625 Test Plan: Tested on pythia model Reviewed By: vedanuj Differential Revision: D24279546 Pulled By: apsdehal fbshipit-source-id: 711a62930f638b3c8b8e2af93aa561f6c8f8e648 13 October 2020, 18:27:35 UTC
6f14d31 Amanpreet Singh 12 October 2020, 21:28:18 UTC [feat] Add init dataclasses for mmbt and encoders (#565) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/565 As step one of FAIM integration, we allow building our models from config so that the models are purely decoupled from configuration and users know what the model expects. We first do this for the MMBT model. - Adds configs for MMBT and respective encoders. - Also adds from_params method to the BaseModel class so that args initialization is possible - Updates build method to support passing of direct config object - Add Config class to BaseModel as well and update typings There is an issue with OmegaConf that doesn't let us use Union in structured configs. Take a look at https://github.com/omry/omegaconf/issues/144 Reviewed By: vedanuj Differential Revision: D23699688 fbshipit-source-id: 37020346ce820207eb41b7bd43c8fba579b436d5 12 October 2020, 21:29:40 UTC
c8264bd Amanpreet Singh 12 October 2020, 20:54:56 UTC [refactor] Use config object for ImageFeatureEncoder init (#564) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/564 There has been a mismatch between ImageFeatureEncoder and ImageEncoder signatures which doesn't allow us to create same config types for them even though that's how they are used. This is confusing and should be same. This diff aims to fix this discrepancy. Reviewed By: vedanuj Differential Revision: D23712213 fbshipit-source-id: b6d3ad56a82727c08f3a53b8ee2b77e6af9e7a17 12 October 2020, 20:56:16 UTC
a5fc02b Vedanuj Goswami 12 October 2020, 02:26:27 UTC [feat] Add masked visual genome dataset (#602) Summary: - Adds visual genome masked dataset - Fixes https://github.com/facebookresearch/mmf/issues/566, for pretraining on visual genome - Fix for loading samples without `question_tokens` - Fix for loading visual genome feature files Pull Request resolved: https://github.com/facebookresearch/mmf/pull/602 Test Plan: Test run lxmert model with visual genome Reviewed By: apsdehal Differential Revision: D24108712 Pulled By: vedanuj fbshipit-source-id: 5ae041a2ce4dcdf914fdecc19466729d445237da 12 October 2020, 02:27:50 UTC
e83f626 Vedanuj Goswami 11 October 2020, 02:28:34 UTC [fix] Fix optimizer params to include all modules for transformer models (#620) Summary: Recent changes in PR https://github.com/facebookresearch/mmf/issues/568 moved the `pooler` layer outside classifier and was not added to optimizer params group. Check added in https://github.com/facebookresearch/mmf/issues/580 helped to catch this. Thanks ronghanghu ! - PR refactors the code in `get_optimizer_parameters_for_bert` - **All** layers other than classifier will have LR of `(lr * finetune_lr_multiplier)` when model head is not of `pretraining` type. - Remove unnecessary setting LR when `finetune_lr_multiplier == 1` Pull Request resolved: https://github.com/facebookresearch/mmf/pull/620 Test Plan: - Checked with MMF Transformer, MMBT, VisualBERT Reviewed By: apsdehal Differential Revision: D24237064 Pulled By: vedanuj fbshipit-source-id: 70f1321d6865bde2f636c46b9d7c00a7bf6667ab 11 October 2020, 02:29:53 UTC
bad7a4d Jun Chen 09 October 2020, 19:26:22 UTC [feat] Add DoCNN as text encoder in MMF Summary: Add DoCNN in TextEncoder. The embedding matrix uses the one in pytext transformer. And support to customize `conv_filters` and `mlp_sizes` in yaml config. # Performance | concat_bert model with text encoder | test/ap | test/accuracy | | docnn+pytext | **0.8998** | **0.8407** | | docnn | 0.8930 | 0.8346 | | xlm_roberta_base | 0.8843 | 0.8336 | Reviewed By: apsdehal Differential Revision: D23933438 fbshipit-source-id: 1cda7920e737d3342a6abf0afa54ca26660496d3 09 October 2020, 19:28:06 UTC
14c0e2f Ronghang Hu 09 October 2020, 02:43:43 UTC [feat] check unused model parameters in optimizer (#580) Summary: This PR adds optimizer option on whether to allow some of the model's parameters not to be used by the optimizer. Default is false to guard against missing parameters due to implementation errors in a model's `get_optimizer_parameters`. Previously `get_optimizer_parameters` is error-prong as the users need to remember to update `get_optimizer_parameters` whenever adding a new submodule, and the error is hard to spot when the users forget to do so. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/580 Test Plan: tested locally w/ VisualBERT, against correctly and incorrectly implemented `get_optimizer_parameters`. Reviewed By: vedanuj Differential Revision: D24201583 Pulled By: ronghanghu fbshipit-source-id: 139f9837e7bc8f5276e33c12bee7f3061f3fa6a5 09 October 2020, 02:45:14 UTC
6074c77 Vedanuj Goswami 09 October 2020, 02:36:13 UTC [feat] Add SentencePiece pytext tokenizer processor (#573) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/573 - Adds pytext SPM tokenizers to be used along with pytext models - Minor modifications to bert_processors to reuse similar parts in other tokenizers Reviewed By: apsdehal Differential Revision: D23782402 fbshipit-source-id: c1c6750cc55a8ab4eb477e9a463990ac9d075014 09 October 2020, 02:37:24 UTC
a769bac Ronghang Hu 09 October 2020, 02:30:07 UTC [fixup] don't init early_stopping from ckpt when resetting counts (#605) Summary: Since the docs in mmf/configs/defaults.yaml on `reset.count` mention that "All counts such as best_update, current_iteration etc will be reset", the early stopping criteria (`best_iteration` and `best_metric_value`) should not be initialized from the ckpt when `reset.count` is on. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/605 Test Plan: tested internally on a DETR-related model when initializing from an earlier model's checkpoint. Reviewed By: vedanuj, apsdehal Differential Revision: D24194277 Pulled By: ronghanghu fbshipit-source-id: 472b5b1b1c17b94639809026f9f3d087285986d7 09 October 2020, 02:31:35 UTC
0d0eaad Amanpreet Singh 08 October 2020, 19:02:17 UTC [fix] Missing init in backends folder (#615) Summary: - Fixes missing imports problem for user dir Pull Request resolved: https://github.com/facebookresearch/mmf/pull/615 Test Plan: Tested locally Reviewed By: vedanuj Differential Revision: D24184482 Pulled By: apsdehal fbshipit-source-id: 52353753ec28147f640f7092a89c359928494daa 08 October 2020, 19:03:41 UTC
132116e Vedanuj Goswami 08 October 2020, 07:57:02 UTC [fix] Fix movie mcan feature reader (#604) Summary: - Fixes https://github.com/facebookresearch/mmf/issues/594 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/604 Reviewed By: apsdehal Differential Revision: D24113348 Pulled By: vedanuj fbshipit-source-id: 1c29c816e697da02370128a40a4a20b25786b25c 08 October 2020, 07:58:31 UTC
33f16ba Amanpreet Singh 07 October 2020, 22:26:09 UTC [fix] Discrepancy in tests isort b/w oss and fbcode (#614) Summary: - fbcode treats tests as third party - This fixes the sync Pull Request resolved: https://github.com/facebookresearch/mmf/pull/614 Test Plan: Tested and compared sorting in the two Reviewed By: vedanuj Differential Revision: D24171915 Pulled By: apsdehal fbshipit-source-id: 08fda10b1f7c4ca11925b70e89f70cc5fdbe1236 07 October 2020, 22:27:54 UTC
4877374 Santiago Castro 06 October 2020, 05:42:48 UTC [chores] Add highlighting to the bibtex entry in README (#603) Summary: Add highlighting to the BibTeX entry in the README. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/603 Reviewed By: vedanuj Differential Revision: D24107511 Pulled By: apsdehal fbshipit-source-id: 6d5750876a2b441bf3dc5d860b4715f734ba1a3b 06 October 2020, 05:43:58 UTC
0d80bc8 Amanpreet Singh 06 October 2020, 03:59:15 UTC [fix,enhancement] Fix CB tutorial and fail on empty loss dict (#585) Summary: - Fix a typo in ConcatBERT tutorial. Modify it to use key "concat_bert_tutorial" to avoid conflicts with existing concat bert inside MMF. - Fail if loss dict is empty in backward with an informative error message - Fixes https://github.com/facebookresearch/mmf/issues/582 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/585 Test Plan: Tested locally Reviewed By: vedanuj Differential Revision: D24068883 Pulled By: apsdehal fbshipit-source-id: 325ada0b558e83e0c9e10051c00fa8750fc2e0cb 06 October 2020, 04:00:36 UTC
f6bb22d Amanpreet Singh 06 October 2020, 03:53:02 UTC [fix] Issue with worker spawn under user dir (#608) Summary: - importlib should always import the module regardless of whether it is in sys.modules or not - This should fix the missing import error that happens in user dir setting for num_workers > 0. - Fixes https://github.com/facebookresearch/mmf/issues/600 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/608 Test Plan: Tested on user dir with disney and e2e tests planned in separate PR Reviewed By: vedanuj Differential Revision: D24125181 Pulled By: apsdehal fbshipit-source-id: 924f98da9206ccec71b5abbc29b3ec2074d08f68 06 October 2020, 03:54:23 UTC
785607a Sasha Sheng 05 October 2020, 17:40:48 UTC [feat] Added the localized narratives dataset (#563) Summary: - Add the masked language modeling task for localized narratives using just caption data. - Create Flickr30k and Coco2017 dataset builder that shares the same mixin as localizednarratives dataset Pull Request resolved: https://github.com/facebookresearch/mmf/pull/563 Reviewed By: vedanuj Differential Revision: D24018522 Pulled By: ytsheng fbshipit-source-id: 862d82475a089b9a4e1069f41e51459f816c4f99 05 October 2020, 17:42:28 UTC
2bc58b5 Vedanuj Goswami 05 October 2020, 17:33:50 UTC [feat] Add torchscriptable version for MMF Transformer (#568) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/568 - Modifications to MMF transformer `preprocess_sample` to be scriptable friendly. - Refactor out backends : Added a BaseBackend abstract class. - Added a new scriptable backend for MMF Transformer that supports only Huggingface Bert, Roberta and XLMR transformer models. Here we are implementing a version of MMF Transformer that supports torchscriptable layers and works only for BERT, RoBERTa and XLMR models - Add scriptable `RobertaEmbeddingsJIT` - Add tests for MMF Transformer Reviewed By: apsdehal Differential Revision: D23569352 fbshipit-source-id: c6527beb1f8b344eb14650d0560a71584c828192 05 October 2020, 17:35:07 UTC
07b786d Sasha Sheng 02 October 2020, 20:21:43 UTC [fix] correct editor_config url (#592) Summary: Thanks for your contribution! If you're sending a large PR (e.g., >50 lines), please open an issue first about the feature/bug, and indicate how you want to contribute. Use [contributing guidelines](https://github.com/facebookresearch/mmf/tree/master/.github/CONTRIBUTING.md) before opening up the PR to follow MMF style guidelines. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/592 Reviewed By: vedanuj Differential Revision: D24083600 Pulled By: ytsheng fbshipit-source-id: 65ed98019e7a232f74e1bb9f92dd73ba9d5f24b4 02 October 2020, 20:23:13 UTC
0cb5bcf Amanpreet Singh 02 October 2020, 16:38:12 UTC [chores] Update Hateful Memes stuff for Phase 2 (#595) Summary: - Updates convert command - Updates features - Updates annotations to point to unseen - Updates docs to mention how to run on seen set - Updates version number of MMF (Time to release a new version, yay) Pull Request resolved: https://github.com/facebookresearch/mmf/pull/595 Test Plan: Tested the runs and everything locally Fixes https://github.com/facebookresearch/mmf/issues/593 Reviewed By: vedanuj Differential Revision: D24069499 Pulled By: apsdehal fbshipit-source-id: 61e8dc4ea3c08327b9492651a242984556b2ea07 02 October 2020, 16:39:26 UTC
1f14f8c John Knox 02 October 2020, 15:51:54 UTC [codemod][staticdocs] Upgrade internaldocs plugin Summary: Codemod generated using: ``` cd fbsource/xplat/staticdocs ./update-all-users.sh '0.4.0' ``` Reviewed By: colriot Differential Revision: D24077905 fbshipit-source-id: 54e322dcb2078c65547f212c723aa227fa1b6014 02 October 2020, 15:53:12 UTC
7bd0a8f Meghan Lele 29 September 2020, 17:20:00 UTC [JIT] Enable @unused syntax for ignoring properties (#45261) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/45261 **Summary** This commit enables `unused` syntax for ignoring properties. Inoring properties is more intuitive with this feature enabled. `ignore` is not supported because class type properties cannot be executed in Python (because they exist only as TorchScript types) like an `ignored` function and module properties that cannot be scripted are not added to the `ScriptModule` wrapper so that they may execute in Python. **Test Plan** This commit updates the existing unit tests for class type and module properties to test properties ignored using `unused`. Test Plan: Imported from OSS Reviewed By: navahgar, Krovatkin, mannatsingh Differential Revision: D23971881 Pulled By: SplitInfinity fbshipit-source-id: 8d3cc1bbede7753d6b6f416619e4660c56311d33 29 September 2020, 17:21:34 UTC
709a696 suzyahyah 28 September 2020, 17:43:43 UTC [fix] memory leak in loss reporting (#560) Summary: Previously `meter_update_dict.update(reduced_loss_dict)` was storing the loss with the gradient and computation graph. This PR should fix various GPU and CPU oom problems. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/560 Reviewed By: apsdehal Differential Revision: D23683638 Pulled By: vedanuj fbshipit-source-id: 733842436129426861fbab5bde3ce3b649509570 28 September 2020, 17:45:00 UTC
ba437e6 Vedanuj Goswami 25 September 2020, 23:20:06 UTC [feat, fix] Modify MMBT torchscript and other fixes for PT 1.7 torchscript (#581) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/581 - Modify MMBT to torchscript full model - Move `replace_with_jit()` to MMBTBase - Changes to `property` methods to make them scriptable. - Changes to base model Reviewed By: apsdehal Differential Revision: D23927970 fbshipit-source-id: b5da316c61597fd8ec9dde5e86ba628b6f1c65a0 25 September 2020, 23:21:32 UTC
259b81b Amanpreet Singh 24 September 2020, 16:42:07 UTC [fix] Update CLEVR urls to latest for now to fix internal task (#579) Summary: Update URL. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/579 Reviewed By: vedanuj Differential Revision: D23907427 Pulled By: apsdehal fbshipit-source-id: 40367c4d628700c975563c07d22b8dee151bd980 24 September 2020, 16:43:29 UTC
50c325a Sasha Sheng 23 September 2020, 07:00:01 UTC [refactor] point to the right maskrcnn_benchmark (#555) Summary: * Update the comment to point to the correct maskrcnn-benchmark. There are many on github that is available. This one is the correct one to use. * Added the X152 model * Tested that both models can be loaded with the updated repo. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/555 Reviewed By: vedanuj Differential Revision: D23835588 Pulled By: ytsheng fbshipit-source-id: d500588f6a14ff45f55bf12937f7f5e69c35067d 23 September 2020, 07:01:16 UTC
7882228 Amanpreet Singh 18 September 2020, 23:57:25 UTC [fix] Black formatting in sample (#572) Summary: - Accidentally forgot to run formatter on sample.py - This fixes it Pull Request resolved: https://github.com/facebookresearch/mmf/pull/572 Reviewed By: vedanuj Differential Revision: D23790238 Pulled By: apsdehal fbshipit-source-id: 3fd4314e7f6a451dba2ac6dbd464570c2c03674e 18 September 2020, 23:58:45 UTC
ee133a0 Amanpreet Singh 18 September 2020, 17:30:12 UTC [feat] Add prediction support for internal datasets Summary: Custom prediction processors can now be specified for the internal datasets. Specify your custom processor that takes in a report and return back a list to be dumped into json. Default processor has been set to ArgMax prediction processor Reviewed By: vedanuj Differential Revision: D23535841 fbshipit-source-id: 43d9265b821b9a191c480a42d7040ff4afafd7e8 18 September 2020, 17:32:14 UTC
404905d Vedanuj Goswami 18 September 2020, 04:20:32 UTC [fix] Add missing if statement to check visual bert bypass_transformer (#570) Summary: Fixes https://github.com/facebookresearch/mmf/issues/569 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/570 Reviewed By: apsdehal Differential Revision: D23777134 Pulled By: vedanuj fbshipit-source-id: b59be010e53be689c4eb9e0283e2d4ea51ec9189 18 September 2020, 04:21:41 UTC
2f33738 Vedanuj Goswami 16 September 2020, 06:09:44 UTC [refactor] Refactor MMF Transformer embeddings to support scriptability (#551) Summary: - Changes to support torchscript for mmf transformer derivatives - Changes to `MMFTransformerEmbeddings` by modifying the different modality embeddings to `nn.ModuleList` to make it scriptable. - Some changes to the forward function Note: It is difficult to make MMF transformer scriptable as a whole since its base can be any type of transformer and making that scriptable is dependent on other libraries. The idea is to have derived models from MMF transformer that can be scripted. For example: We will have a version of MMF transformer that works with Bert, Roberta and XLMR that will be scriptable. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/551 Test Plan: - Test with hateful memes dataset Reviewed By: apsdehal Differential Revision: D23569270 Pulled By: vedanuj fbshipit-source-id: 21d9ac80d912fcc65810a13f4facd0e828fcdd39 16 September 2020, 06:11:16 UTC
00fa16f Vedanuj Goswami 16 September 2020, 05:11:14 UTC [chores] Add linter tests to Actions (#559) Summary: Add linter tests to Actions. We can then remove CircleCI CPU tests completely. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/559 Reviewed By: apsdehal Differential Revision: D23687038 Pulled By: vedanuj fbshipit-source-id: fb6ab9e627394fcf7b47bf60b07f90740f456359 16 September 2020, 05:12:22 UTC
f24c849 Vedanuj Goswami 15 September 2020, 02:37:31 UTC [fix] Fix tests with larger CPU resource, fail-fast set to false (#558) Summary: - Increase resource size for CircleCI CPU tests, change to machine executor which have larger RAM - set fail-fast to false for Actions fto be able to pinpoint which matrix combinations cause failures Pull Request resolved: https://github.com/facebookresearch/mmf/pull/558 Test Plan: - Run the tests and check if they pass Reviewed By: apsdehal Differential Revision: D23668047 Pulled By: vedanuj fbshipit-source-id: 47d135e784e62bb645de0227d75488478d7761b2 15 September 2020, 02:38:56 UTC
cdf1bcc Vedanuj Goswami 14 September 2020, 20:35:40 UTC [mmf] [feat] Batch Beam Search (#557) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/557 Implement beam search for batch_size > 1. Current Implementation of Beam Search works only for batch size = 1. Here is the implementation : https://github.com/facebookresearch/mmf/blob/08e5416db2ffa86508af4161505213b976df2cfd/mmf/utils/text.py#L271 Reviewed By: howardhsu Differential Revision: D22860178 fbshipit-source-id: 95b70147ab8fad8ddd43dfef54b7be7c45841f3f 14 September 2020, 20:37:14 UTC
c766d82 Vedanuj Goswami 12 September 2020, 08:41:35 UTC [feat] Add torchscript support for MMBT (#550) Summary: - Adds torchscript support for MMBT model - Adds `BertEmbeddingsJit` and `BertModelJit` layers which are torchscriptable versions of `BertEmbeddings` and `BertModel` layers from transformers package - Replaces transformers Bert model layers with scriptable layers by monkey patching, function `replace_with_jit` - Adds tests Pull Request resolved: https://github.com/facebookresearch/mmf/pull/550 Test Plan: - Run tests - Run old pretrained checkpoints for MMBT and verify results Reviewed By: apsdehal Differential Revision: D23569134 Pulled By: vedanuj fbshipit-source-id: ec688aa5686812ae838a95ad203668ae06dfd1fc 12 September 2020, 08:42:42 UTC
ca78547 Vedanuj Goswami 12 September 2020, 05:31:13 UTC [feat] Add torchscript support for VisualBERT (#537) Summary: - Adds scriptable `VisualBERTForClassification` model and `VisualBERTForPretraining` model - Adds scriptable versions of some [transformer](https://github.com/huggingface/transformers/blob/v2.3.0/transformers/modeling_bert.py) package BERT model layers : `BertSelfAttentionJit`, `BertAttentionJit`, `BertLayerJit`, `BertEncoderJit` - Changes to `BertVisioLinguisticEmbeddings` to make it scriptable - Add tests for `VisualBERT` model Pull Request resolved: https://github.com/facebookresearch/mmf/pull/537 Test Plan: - Run tests - Run validation of model on old checkpoints and verify accuracy Reviewed By: apsdehal Differential Revision: D23496932 Pulled By: vedanuj fbshipit-source-id: 07d7c31eba15657e4e922eb70b244078124416c3 12 September 2020, 05:32:41 UTC
cf866c8 suzyahyah 11 September 2020, 20:39:40 UTC [refactor] Decouple visual_bert embeddings forward function (#536) Summary: Summary - decouple `forward` into `encode_text` and `encode_image` functions for readability and potential reusability. Tested with `mmf_run datasets=hateful_memes run_type=train_val_test config=projects/hateful_memes/configs/visual_bert/from_coco.yaml model=visual_bert checkpoint.resume_zoo=visual_bert.finetuned.hateful_memes.from_coco ` Pull Request resolved: https://github.com/facebookresearch/mmf/pull/536 Reviewed By: vedanuj Differential Revision: D23503492 Pulled By: apsdehal fbshipit-source-id: 30982de25d8e1b10ef0b1df2a2fa3e82597ce21a 11 September 2020, 20:42:07 UTC
a67ca62 Ronghang Hu 10 September 2020, 16:32:50 UTC [fixup] fix config and zoo file for m4c_captioner (#553) Summary: - Fix the the mismatch between M4C-Captioner checkpoint and TextVQA features (https://github.com/facebookresearch/mmf/issues/544). Keep two variants of M4C-Captioner config, one trained with newer features extracted with maskrcnn-benchmark (`defaults`), and the other trained with older features extracted with Caffe2 (`with_caffe2_feat`), which is used in our experimentations in the paper and has higher CIDEr. - Create a MMF website page for M4C-Captioner. - Also fix previous M4C website page. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/553 Test Plan: - tested locally with model zoo checkpoints `m4c_captioner.textcaps.defaults` (for `defaults.yaml`) and `m4c_captioner.textcaps.with_caffe2_feat` (for `with_caffe2_feat.yaml`) Reviewed By: apsdehal Differential Revision: D23620758 Pulled By: ronghanghu fbshipit-source-id: a5c7df33d8424da1e3d0bcb1030a3e621487cb27 10 September 2020, 16:34:14 UTC
11194ac suzyahyah 09 September 2020, 06:41:46 UTC [refactor] Minor refactor to ckpt _load function (#549) Summary: Summary * remove unnecessary conditional logic for `"model" in ckpt` * replace `new_dict` and `final_dict` with just `state_dict` No actual logic changed Pull Request resolved: https://github.com/facebookresearch/mmf/pull/549 Reviewed By: vedanuj Differential Revision: D23575778 Pulled By: apsdehal fbshipit-source-id: 064eb021962fa2d4962b95818ff82a53384a3e8b 09 September 2020, 06:43:11 UTC
584e26d Vedanuj Goswami 08 September 2020, 22:59:00 UTC [chores] Upgrade to pytorch 1.6 and torchvision 0.7 (#546) Summary: - Upgrading to pytorch 1.6, which comes with new features required for upcoming torchscript changes - Optimizer keys are no longer non-deterministic in PT1.6, check pytorch [PR](https://github.com/pytorch/pytorch/pull/37347 ). Remove `skip_keys` check from checkpoint optimizer tests. - Fixed nucleus sampling tests - Update docs to reflect support for pytorch 1.6 - Update CI tests, CircleCI and Actions to use pytorch 1.6, torchvision 0.7 Test Plan : - Tested with visual bert and mmbt models on Hateful Memes - Running internally at FB (runs on pytorch master) Pull Request resolved: https://github.com/facebookresearch/mmf/pull/546 Reviewed By: apsdehal Differential Revision: D23558059 Pulled By: vedanuj fbshipit-source-id: 3d82978276462a3f3dea564ac33581240e284d03 08 September 2020, 23:00:27 UTC
9e83320 steph-en-m 08 September 2020, 18:57:05 UTC [feat] Add descriptive error message for missing processor in registry (#548) Summary: This PR resolves https://github.com/facebookresearch/mmf/issues/455 Pull Request resolved: https://github.com/facebookresearch/mmf/pull/548 Reviewed By: vedanuj Differential Revision: D23559767 Pulled By: apsdehal fbshipit-source-id: 59e4e4e5d7dd9eaf88edf81b6906d5d8768f83a2 08 September 2020, 18:58:48 UTC
3b8e26b Amanpreet Singh 06 September 2020, 01:59:53 UTC [fix] Update HM docs for resume_pretrained=False during evaluation (#545) Summary: - Update README and docs Pull Request resolved: https://github.com/facebookresearch/mmf/pull/545 Reviewed By: vedanuj Differential Revision: D23545636 Pulled By: apsdehal fbshipit-source-id: bd23195e39584725b447d7be8ccedd0281a41d76 06 September 2020, 02:00:48 UTC
efe4023 steph-en-m 04 September 2020, 19:56:04 UTC [fix] modify mmf_convert_hm to create a copy of a file (#532) Summary: This PR resolves https://github.com/facebookresearch/mmf/issues/528 to ensure `mmf_convert_hm` preserves the original file Pull Request resolved: https://github.com/facebookresearch/mmf/pull/532 Test Plan: - tested it on the hateful memes dataset on colab and the original file is preserved. Reviewed By: apsdehal Differential Revision: D23545590 Pulled By: vedanuj fbshipit-source-id: 136f6bd9a526d4ed91fc27b4420160fbe9ff02e0 04 September 2020, 19:57:07 UTC
637a5bc Sasha Sheng 04 September 2020, 04:53:35 UTC [feat] added split dataset feature (#470) Summary: * allow user to specify x% of dataset to split from train, example yaml shown below: ``` textvqa: zoo_requirements: - textvqa.defaults - textvqa.ocr_en split_train: val: 0.2 test: 0.1 seed: 42 features: train: - textvqa/defaults/features/open_images/detectron.lmdb,textvqa/ocr_en/features/ocr_en_frcn_features.lmdb processors: ``` * To test: `pytests test` * See logs, 24221 is 0.7 of 34602 which is the total size of the train after calculation, etc. ![Screen Shot 2020-08-10 at 11 47 12 PM](https://user-images.githubusercontent.com/1545487/89866133-e052f600-db63-11ea-80e0-4f2c6b1a7acc.png) Pull Request resolved: https://github.com/facebookresearch/mmf/pull/470 Reviewed By: apsdehal Differential Revision: D23098846 Pulled By: ytsheng fbshipit-source-id: 9498ae0eda8001716f8d8cf77724904a4e88a495 04 September 2020, 04:54:47 UTC
1e64628 rizavelioglu 03 September 2020, 19:43:59 UTC [fix] Typo in pretrain train+val config of visual bert (#541) Summary: fix typo in the name of the features files Pull Request resolved: https://github.com/facebookresearch/mmf/pull/541 Reviewed By: vedanuj Differential Revision: D23504310 Pulled By: apsdehal fbshipit-source-id: 9e3055a5150c98b8535edf3389804f6ecf509357 03 September 2020, 19:45:50 UTC
7a35e17 Amanpreet Singh 03 September 2020, 17:15:14 UTC [fix] Answer processor issues in MaskedVQA2Dataset (#529) Summary: - Adds add_answer option to default config - Set it to default in the dataset - Add default answer processor config Pull Request resolved: https://github.com/facebookresearch/mmf/pull/529 Test Plan: Tested on full masked_vqa2 config under VisualBERT Reviewed By: vedanuj Differential Revision: D23470296 Pulled By: apsdehal fbshipit-source-id: 8b8aacea829f2c4de392ab906de571fe8ec8d93d 03 September 2020, 17:17:03 UTC
98964eb Amanpreet Singh 03 September 2020, 16:58:55 UTC [fix] Add missing init files to new folders (#531) Summary: - Without this, these folders are not included when installed through pip Pull Request resolved: https://github.com/facebookresearch/mmf/pull/531 Test Plan: Tested in separate repo Reviewed By: vedanuj Differential Revision: D23485913 Pulled By: apsdehal fbshipit-source-id: cc49fd7763016aa58ac8aa6dc47cd2e8a40fdf39 03 September 2020, 17:00:05 UTC
24dedb1 Amanpreet Singh 01 September 2020, 23:08:45 UTC [chores] Update contributing guidelines, issue templates, installation (#524) Summary: - Contributing guidelines now mention pre commit, our style guidelines and general PR suggestions - Adds issue template for a catch all category and enforce template always - Adds Pull request template - Updates installation docs to mention conda environment, windows install, installing directly from github with pip, and contributing guidelines along with recommended, optional tags for better clarity Pull Request resolved: https://github.com/facebookresearch/mmf/pull/524 Test Plan: Tested locally by building the website Reviewed By: vedanuj Differential Revision: D23407113 Pulled By: apsdehal fbshipit-source-id: 2da0bfe91838b56c039c8a34bda94140dd5f7c62 01 September 2020, 23:10:03 UTC
d6fc967 Vedanuj Goswami 01 September 2020, 05:43:16 UTC [fix] Fix lxmert model bbox processing (#526) Summary: - Fixes https://github.com/facebookresearch/mmf/issues/520 and use `transformer_bbox_processor` processor - `transformer_bbox_processor` will now work when bbox features already have area norm Pull Request resolved: https://github.com/facebookresearch/mmf/pull/526 Test Plan: Run lxmert model for pretraining and finetuning on vqa2 Reviewed By: apsdehal Differential Revision: D23414310 Pulled By: vedanuj fbshipit-source-id: 0fa1848448ba193e0924966ae840ea225bb718ca 01 September 2020, 05:44:20 UTC
bf148a8 Grzegorz Jacenków 29 August 2020, 17:31:04 UTC [docs] Fix tutorial for the Hateful Memes Challenge (#523) Summary: * Updates the documentation with a working code example. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/523 Reviewed By: vedanuj Differential Revision: D23407535 Pulled By: apsdehal fbshipit-source-id: e431f1a1ef6ee0105101d28701cec9bbdf9881b0 29 August 2020, 17:32:11 UTC
cb3ecd1 Amanpreet Singh 28 August 2020, 22:17:09 UTC [fix] Remove all .cuda calls from mmf so models can run on CPU (#522) Summary: - VisualBERT and VILBERT Pull Request resolved: https://github.com/facebookresearch/mmf/pull/522 Test Plan: Tested locally with and without GPU Reviewed By: vedanuj Differential Revision: D23404310 Pulled By: apsdehal fbshipit-source-id: e1e774e86d3726efd31271de1fe565b449739f2b 28 August 2020, 22:18:47 UTC
57ed846 Suzanna Sia 28 August 2020, 18:07:04 UTC [fix] PathManager in sweep script as part of Manifold Migration (#498) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/498 Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/132 Adopting `PathManager` for various file io utilities as part of Manifold migration. The default checkpoint_dir is still gluster, because I'm assuming not everyone has a manifold directory. To use a manifold_dir, we would sent it as an argument `--checkpoints_dir <manifold_save_dir>` via cli. Reasons for all the if-else statements within `fblearner.py` for manifold vs gluster, * this diff actually slows down submitting jobs to flow (because of the PathManager dependency). * It also requires a custom built kernel with torch instead of a regular bento kernel to run the flow script. Hen Other changes * simplified the logic of `has_started` * better logging of run in progress and what to do * check for `config.yaml` instead of `train.log` to determine if run exists Reviewed By: apsdehal Differential Revision: D23236008 fbshipit-source-id: 0aff4684934c82e0a3cc6f55143d55e5a4b6e719 28 August 2020, 18:08:10 UTC
6d98b6c Vedanuj Goswami 28 August 2020, 15:49:43 UTC [fix] Fix grad accumulation logic for Iterable datasets (#521) Summary: PR https://github.com/facebookresearch/mmf/issues/367 breaks training with `IterableDataset` as `len(self.train_loader) == 0` since `__len__` is not implemented for every `IterableDataset`. This PR mitigates this regression and : - Sets `num_remaining_batches` to be equal to (number of updates remaining x update frequency) for `IterableDataset` case. For non `IterableDataset` the old behavior is maintained (`len(self.train_loader)`) - Modifies the logic to avoid the need for checking if iterator has finished - Fixes another instance where we used to check `len(self.train_loader)` for setting number of updates when `max_epochs` is specified. Now in case of `IterableDataset` it will fall back to using `max_updates`. Added warning. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/521 Test Plan: - Tested internally on FB datasets - Tested on Hateful Memes Reviewed By: apsdehal Differential Revision: D23395374 Pulled By: vedanuj fbshipit-source-id: 415358455d12bdcb1d7ba87dd304c203c6cd990e 28 August 2020, 15:51:08 UTC
3f313b6 Amanpreet Singh 28 August 2020, 00:27:50 UTC [feat] MultiClassFromFile Processor along with Violations (#519) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/519 - Adds violations dataset - Add processor for MultiClassFromFile - Adds batch processor for MutliClassFeaturesWithText Reviewed By: vedanuj Differential Revision: D23273579 fbshipit-source-id: 6f474f312014d41823689de2f7736b3563ff2b3f 28 August 2020, 00:28:49 UTC
ec1aa0f Vedanuj Goswami 27 August 2020, 18:02:02 UTC [docs] FB internal documentation (#518) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/518 Reviewed By: apsdehal Differential Revision: D23365885 fbshipit-source-id: ffbbe65ee6e8e9ae4dd05eb054cdf38df945f5e1 27 August 2020, 18:03:44 UTC
f5d2a88 Vedanuj Goswami 27 August 2020, 17:54:20 UTC [feat] Open image features with PathManager (#468) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/468 - Adds ability to read features from different URIs (Manifold, Everstore URIs) in addition to file paths. - Adds target for lmdb conversion/extraction Reviewed By: apsdehal Differential Revision: D23021466 fbshipit-source-id: 01cc4811730a02c98b57896e122e4a044f4546a7 27 August 2020, 17:55:55 UTC
78c5bfa Vedanuj Goswami 27 August 2020, 03:57:34 UTC [docs] Update BUTD documentation (#516) Summary: Update BUTD documentation - Move to website docs - Add more instructions about inference - Added table with updated best model Pull Request resolved: https://github.com/facebookresearch/mmf/pull/516 Test Plan: NA Reviewed By: apsdehal Differential Revision: D23351839 Pulled By: vedanuj fbshipit-source-id: 58a79b4bea6484f10e93425dc6032a4163082461 27 August 2020, 03:58:38 UTC
cf12af6 Vedanuj Goswami 27 August 2020, 03:49:24 UTC [fix] Freeze CircleCI black version (#517) Summary: Freeze black and flake8 versions in CircleCI to be in sync with the pre-commit hooks Pull Request resolved: https://github.com/facebookresearch/mmf/pull/517 Test Plan: Pass CircleCI tests Reviewed By: apsdehal Differential Revision: D23359588 Pulled By: vedanuj fbshipit-source-id: bdf7eb75b26ef26996bbb5ebb207841d099a04b6 27 August 2020, 03:50:37 UTC
31f0cc1 Vedanuj Goswami 27 August 2020, 02:50:05 UTC [feat] Generalized MM transformer (#422) Summary: Introduces a generalized version of multimodal transformer. - Extends `BaseTransformer` and builds a modular multimodal transformer model - Base transformer model can be any model from [transformers](https://github.com/huggingface/transformers) - Supports n number of modalities - Adds ability to add separate input, position and segment embeddings for each modality - text modalities share weight with base transformer model embeddings - position embeddings use separate embeddings for each modality - token type/segment embeddings are shared/mean of base transformer token/segment embeddings (if available) Pull Request resolved: https://github.com/facebookresearch/mmf/pull/422 Test Plan: - Tested with HM, VQA2, OKVQA Reviewed By: apsdehal Differential Revision: D22636744 Pulled By: vedanuj fbshipit-source-id: 3fa05472f3486a62fb6d3a1ffcd689712af076b4 27 August 2020, 02:51:25 UTC
2db5ad3 Ronghang Hu 27 August 2020, 02:44:52 UTC [fix] load lr scheduler state dict during checkpoint loading (#513) Summary: the current implementation does not load lr_scheduler's states from checkpoints, which is a bug (causing last_epoch to be reset when loading from checkpoints). This commit fixes it by saving and loading lr_scheduler's states. It fall backs to setting lr_scheduler's last_epoch when lr_scheduler's states don't exist in checkpoint. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/513 Test Plan: tested with M4C checkpoint saving and loading. (CPU tests failed due to black formatting error in mmf/datasets/processors/bert_processors.py, unrelated to this PR.) Reviewed By: apsdehal Differential Revision: D23349944 Pulled By: ronghanghu fbshipit-source-id: 5c0802957804d3e169f30f0b944aa3d54ccf956a 27 August 2020, 02:46:02 UTC
aab972e Eric Undersander 27 August 2020, 02:39:13 UTC [feature, fix] added support for update_frequency (v2) (#367) Summary: This PR adds support for config.training.update_frequency: ``` # Run update_frequency=K batches with batch_size=N, accumulate gradients, then do # one update (one optimizer step). The effect is a large effective batch size of # KxN (without incurring the memory overhead of setting batch_size to KxN). update_frequency: 1 ``` https://www.internalfb.com/intern/tasks/?t=66569918 This is a revised approach; see also earlier PR https://github.com/facebookresearch/mmf/pull/292. With this change, we have to be more clear with naming. Amanpreet and I agreed on this: - we have no particular term for a single forward/backward pass for a single batch - "update" == "iteration" == one optimizer step (will be more than one batch when update_frequency > 1) The update logic in `BaseTrainer._backward` has mostly been moved to `_start_update` and `_finish_update`. There's also a fix for `MultiDatasetLoader.__len__`. The existing bug is only present for the case that the train set size is not a multiple of per-worker batch size, which is probably rare in practice. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/367 Test Plan: I did ad-hoc testing of BaseTrainer.train() (`mmf_run config="projects/hateful_memes/configs/late_fusion/defaults.yaml" model="late_fusion" dataset=hateful_memes`) with print statements and varying `[batch_size, update_frequency, train set size]`, including end-of-epoch and completion of the train run. Reviewed By: vedanuj Differential Revision: D23335921 Pulled By: apsdehal fbshipit-source-id: 984ae2706e50679eb0be47e14d872a359de36fd6 27 August 2020, 02:41:45 UTC
a5096b4 John Knox 26 August 2020, 15:40:54 UTC [mmf] Use static docs plugin Summary: I noticed mmf doesn't have the static docs plugin enabled. This will make sure your browser address bar and page titles work as expected. Reviewed By: apsdehal Differential Revision: D23343272 fbshipit-source-id: 1de615c72968722da162bf443d386301cbe60063 26 August 2020, 15:41:55 UTC
f700be8 John Knox 26 August 2020, 15:11:11 UTC [staticdocs] Update plugin version for all consumers Summary: Updates the static docs plugin everywhere. Done by running `./update-all-users.sh 0.3.6` inside `fbsource/xplat/staticdocs/` Reviewed By: mweststrate Differential Revision: D23343132 fbshipit-source-id: 2deaec1f8a4ead43475d77f40b7bf35e6b25f994 26 August 2020, 15:12:17 UTC
53ed195 Amanpreet Singh 25 August 2020, 23:07:18 UTC [fix] Unimodal base in case of batch size 2 (#502) Summary: Current implementation uses `len` to check whether output was from bert encoder or not. Add extra check for `torch.is_tensor` so that image features case for batch size 2 safely works. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/502 Test Plan: Tested on batch size 2. Reviewed By: vedanuj Differential Revision: D23243300 Pulled By: apsdehal fbshipit-source-id: 120469dc4ced94dd8348285223213d4409a3b22b 25 August 2020, 23:08:58 UTC
298718e Vedanuj Goswami 24 August 2020, 22:54:13 UTC [fix] Fix multilabel macro-f1 inheritance (#508) Summary: - Fixes MultiLabel Macro F1 metric. It should inherit from the `MultiLabelF1`. - Fixes an issue with a recent change in pytorch 1.7, where earlier argmax would return the index of ANY min or max, but now they will return the index of the FIRST min or max. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/508 Test Plan: Tests passed Reviewed By: apsdehal Differential Revision: D23283349 Pulled By: vedanuj fbshipit-source-id: e52dac18eed875767f251ef491329d1d5093bbf1 24 August 2020, 22:56:12 UTC
ab8d3f5 Vedanuj Goswami 22 August 2020, 01:50:19 UTC [feat] Add support for BS, NS metric calculation, update butd model etc. (#495) Summary: Fixes https://github.com/facebookresearch/mmf/issues/482 #489 Previously caption bleu4 metric in mmf could not be directly used with Beam Search(BS), Nucleus Sampling(NS). The process to follow was extract the predictions and then compute metrics separately. - Adds support for calculating metrics when using BS, NS - Uploaded new model to zoo trained with latest features. Old model when used with new features with mmf doesn't work. Issue https://github.com/facebookresearch/mmf/issues/489 - Modified `accumulate_tensor_fields` to throw a warning instead of KeyError when some metric field is not available. This is done to support use cases where different model output needs to be accumulated depending on train/val/test stages. - Fix zoo requirement for coco captioning - Fixes the issue of CUDA OOM when running inference with BS, NS on a GPU with less memory. Now only the captions(bs x cap_len) are used instead of scores (bs x cap_len x vocab_sz). Pull Request resolved: https://github.com/facebookresearch/mmf/pull/495 Test Plan: BLEU4 Score on val set : 36.04 ``` mmf_run config=projects/butd/configs/coco/beam_search.yaml run_type=val dataset=coco model=butd checkpoint.resume_zoo=butd ``` Reviewed By: apsdehal Differential Revision: D23214751 Pulled By: vedanuj fbshipit-source-id: 27f0cdc21cf09061d82a580440f4fa8ee13a1016 22 August 2020, 01:51:58 UTC
95e89ee Amanpreet Singh 20 August 2020, 15:56:35 UTC [feat] Support for more than 2 segments in MMBT (#460) Summary: Segment embedding other than first 2 dim are initialized from the average of first 2 embeddings This is required for the case where there are 3 segments such as post, ocr, and features. Pull Request resolved: https://github.com/facebookresearch/mmf/pull/460 Reviewed By: vedanuj Differential Revision: D22970794 Pulled By: apsdehal fbshipit-source-id: 4d5ee86162079ae96d5387b0c08e96d73bb3bbe1 20 August 2020, 15:57:56 UTC
697ca3b Suzanna Sia 20 August 2020, 04:17:32 UTC [fix] Support for reading raw COCO images for VQA2 and COCO to support models like MMBT (#459) Summary: Pull Request resolved: https://github.com/facebookresearch/mmf/pull/459 Pull Request resolved: https://github.com/fairinternal/mmf-internal/pull/127 VQA2 and COCO dataset readers `dataset.py` previously only support reading in from feature file. After this PR, it should now be compatible with a model which reads in raw images. Should not affect any models which previously relied on `use_images: false`, `use_features: true` Changes: * Loads images in `{vqa2,coco}/{masked_,}dataset.py` if `self._use_features` flag is False * Included image transform in `faim/mmf/mmf/datasets/builders/vqa2/dataset.py:def init_processors(self)` * Enabled npy loading for internal `faim/mmf/mmf/datasets/databases/annotation_database.py:def _load_npy()` * Configs -- The main difference for configs is the inclusion of image_processor in this config file: `mmf/configs/datasets/vqa2/with_raw_images.yaml` and the dataset config `use_images` and `use_features` flags. ``` dataset_config: vqa2: use_images: true use_features: false images: train: - coco/defaults/images/train2014 val: - coco/defaults/images/val2014 test: - coco/defaults/images/test2015 ``` Reviewed By: apsdehal Differential Revision: D22979686 fbshipit-source-id: 3cac3d41329c098e0db1b50e4c554b72a194c120 20 August 2020, 04:18:37 UTC
back to top