v1.5.3

Overview

This release offers clarity and ease of use for Model-in-the-Loop tasks. The pattern of communicating with the server over the course of the task (called "remote procedures") was added to FormComposer, and an example called "Interactive Generation of a Look-Alike Image" was provided. Other examples of remote procedure tasks were brought up-to-date as well.

FormComposer gainded a few useful additions:

• dynamic fieldsets (to save multiple instances of the same response fieldset)
• additional flexibility for coding up UI triggers
• buttons as an element of a form fieldset

This is useful when:

• Worker should react to continuously generated content
• Worker should guide your algorithm to achieve a goal

What's Changed

• Renamed React component files to jsx; fixed VideoAnnotator bug by @meta-paul in #1256
• Added example task 'Interactive Image Generation' by @meta-paul in #1257
• Updated remote procedure examples; Refactored remote procedure blueprint by @meta-paul in #1258

Full Changelog: v1.5.2...v1.5.3

v1.5.2

Overview

Mephisto now offers a UI for managing qualifications and qualification assigns. We've nested it under TaskReview app since it already can run a local web server. Some new capabilities were also added:

• Now workers can be granted qualifications directly (without context of a specific Task Unit)
• Qualifications now support not only name, but also description
• Multiple qualifications can now be assigned to a single Task Unit
• Screenshots and all details are described in the docs

Note that this code change includes a database migration. It should port all your existing qualifications data automatically. Though just in case we'd recommend backing up the qualifications part of your database (using mephisto db export --qualification-only command as described here)

Some other changes include bug fixes and reorganization of local Mephisto packages for more transparent naming and versioning.

What's Changed

• Qualifications management UI by @meta-paul in #1254
• Added local builds of Mephisto packages by @meta-paul in #1255

Full Changelog: v1.5.1...v1.5.2

v1.5.1

Overview

Patch with small fixes to VideoAnnotator and docs update.

VideoAnnotator

• Display validation error for overlapping video segments on the same track
• Add support for multiple video formats (not just mp4)
• Fix a few outdated unittests for generators

For docs

• Add "Known Issues" page in the docs
• Add FormComposer screenshots

What's Changed

• v1.5.1 by @meta-paul in #1253

Full Changelog: v1.5.0...v1.5.1

v1.5.0

Overview

This release introduces a couple of key features, and a number of smaller improvements.

Video Annotator

VideoAnnotator is a new Task Generator for convenient and functional video annotation.

Key features:

• Worker can indicate multiple video segments along multiple tracks, and annotate each one
• For each segment, you can specify (via JSON config) desired annotation input (as a list of standard form fields)
• VideoAnnotator JSON config works similar to FormComposer (including pre-signed S3 URLs, and support for custom JS triggers)
• Workers inputs are downloadable in JSON
• When viewing a completed unit, worker inputs are also available in WebVTT format

InHouse provider

The new InHouse provider enables using Mephisto with your own "in-house" crowd - and basically crowdsourcing your task to people you know. (This used to be "mock" provider last year, but now we're using "mock" only for testing).

The initial implementation is quite simple: a worker navigates to your Task page, types in their worker nickname, and proceeds to the next available Task unit (if any). Afterwards you can review their results and take notes or grant qualifications, just like you do that with the 3rd party workers.

Future roadmap for InHouse provider would include: proper worker authorization; allowing workers to view researcher notes; restricting worker acces based on their qualification.

Feature improvements

TaskReview app

• Visualized selection-based response stats in Task Review
• Added ability to view completed task units, without assessing them

DataPorter feature

• Now you can port only granted worker qualifications (without all other data), to easily build pool of known workers collaboratively

FormComposer

• Enabled CSS insertions in FormComposer

Grafana

• Updated Grafana dockerized deployment and docs
• Added access to Grafana dashboard from TaskReview app

General housekeeping

• Updated and added missing architecture diagrams in Mephisto docs
• Refactored increasingly nested CLI commands, and updated their Help strings
• Add example of GoldUnit and Worker Onboarding
• upgraded NodeJS to v20

Included PRs

• Bump eslint-plugin-flowtype from 4.7.0 to 8.0.3 by @dependabot in #1157
• Bump @storybook/addon-actions from 6.5.16 to 8.0.10 by @dependabot in #1158
• Bump peaceiris/actions-gh-pages from 3 to 4 by @dependabot in #1163
• Bump @babel/plugin-proposal-class-properties from 7.16.7 to 7.18.6 by @dependabot in #1161
• Bump pre-commit/action from 2.0.0 to 3.0.1 by @dependabot in #1164
• Bump pytest from 5.4.3 to 8.2.0 by @dependabot in #1166
• Additional unittests for Data Porter feature by @meta-paul in #1178
• Added porting of worker qualifications only for Data Porter by @meta-paul in #1180
• Visualization of selection-based response stats in Task Review by @meta-paul in #1193
• Enable CSS insertions in FormComposer by @meta-paul in #1194
• Bump mdutils from 1.4.0 to 1.6.0 by @dependabot in #1165
• Bump requests from 2.31.0 to 2.32.0 by @dependabot in #1172
• Bump tornado from 6.3.3 to 6.4.1 by @dependabot in #1179
• Bump urllib3 from 1.26.17 to 1.26.19 by @dependabot in #1187
• Bump ws from 7.5.9 to 7.5.10 by @dependabot in #1188
• Bump ws from 5.2.3 to 5.2.4 in /mephisto/abstractions/architects/router/node by @dependabot in #1189
• Bump torch from 1.13.1 to 2.3.1 by @dependabot in #1181
• Bump tabulate from 0.8.10 to 0.9.0 by @dependabot in #1171
• Bump pdoc from 14.4.0 to 14.5.1 by @dependabot in #1192
• Bump braces from 3.0.2 to 3.0.3 in /packages/annotation-toolkit by @dependabot in #1184
• Bump cypress-io/github-action from 3.1.0 to 6.7.1 by @dependabot in #1186
• Bump pillow from 10.2.0 to 10.3.0 by @dependabot in #1138
• Bump @storybook/node-logger from 6.5.16 to 8.1.10 by @dependabot in #1191
• Experimental functions for resuming a task run by @JackUrb in #910
• Update architecture diagrams by @meta-paul in #1206
• Update Grafana dockerized deployment and docs, refactor CLI commands by @meta-paul in #1207
• Fixed Docker testing matrix, removed unneeded image versions by @meta-paul in #1208
• Add example of GoldUnit usage by @meta-paul in #1211
• Add examples of worker onboarding and screening by @meta-paul in #1212
• Update Mephisto manual installation instructions by @meta-paul in #1213
• Added access to Grafana dashboard from TaskReview app by @meta-paul in #1214
• Create mephisto task addons by @meta-paul in #1218
• Update help for mephisto CLI command by @meta-paul in #1219
• Updated NodeJS to v22 by @meta-paul in #1221
• Accommodate lengthy task instructions in Form Composer by @meta-paul in #1222
• Add in house provider by @meta-paul in #1227
• Include token values in downloaded data for FormComposer tasks by @meta-paul in #1230
• Add Video Annotator feature by @meta-paul in #1247

Full Changelog: v1.4.0...v1.5.0

v1.4.0

Overview

In this release we added Data Porter feature that allows to move full/partial data dumps between different isntances of Mephisto. This enables things like:

• backups/restores of Mephisto data
• consolidating task data across different Mephisto instances
• sharing task data with your colleagues

We also introduced a simple (forward-only) workflow for database migrations, enabled custom code snippets in Form Composer, and updated numerous library versions.

Last but not least, mephisto.ai documentation got a major update as well.

Included PRs

• Bump pyarrow from 9.0.0 to 14.0.1 by @dependabot in #1083
• Bump @cypress/request and cypress in /examples/form_composer_demo/webapp by @dependabot in #1106
• Bump @cypress/request and cypress in /packages/react-form-composer by @dependabot in #1105
• Bump ip from 2.0.0 to 2.0.1 by @dependabot in #1103
• Bump aiohttp from 3.8.6 to 3.9.2 by @dependabot in #1101
• Bump follow-redirects from 1.15.2 to 1.15.4 by @dependabot in #1094
• Bump pillow from 10.0.1 to 10.2.0 by @dependabot in #1099
• Fixing agent desync on RemoteProcedure tasks by @JackUrb in #1063
• Mock architect actions recording by @JackUrb in #1064
• Script to write out qualifications by @JackUrb in #1069
• Enabled custom code insertions for FormComposer by @meta-paul in #1108
• Display qualifications granted to a worker in TaskReview app by @meta-paul in #1127
• Update mephisto.ai documentation by @meta-paul in #1128
• Data porter feature by @meta-paul in #1153
• Bump webpack-dev-middleware from 5.3.3 to 5.3.4 by @dependabot in #1130
• Bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /mephisto/review_app/client by @dependabot in #1129
• Bump xmltodict from 0.12.0 to 0.13.0 by @dependabot in #1126
• Bump @babel/core from 7.17.7 to 7.24.0 by @dependabot in #1118
• Bump follow-redirects from 1.15.2 to 1.15.6 in /examples/static_react_task/webapp by @dependabot in #1116
• Bump actions/setup-python from 2 to 5 by @dependabot in #1113
• Bump actions/setup-node from 2 to 4 by @dependabot in #1111
• Bump dorny/paths-filter from 2 to 3 by @dependabot in #1109
• Bump react-shadow from 19.1.0 to 20.4.0 by @dependabot in #1121
• Bump follow-redirects from 1.15.5 to 1.15.6 in /mephisto/review_app/client by @dependabot in #1114
• Bump follow-redirects from 1.15.5 to 1.15.6 in /packages/mephisto-task-multipart by @dependabot in #1112
• Bump react-player from 2.9.0 to 2.16.0 by @dependabot in #1148
• Bump @storybook/addon-links from 6.5.16 to 8.0.10 by @dependabot in #1151
• Bump aiohttp from 3.9.2 to 3.9.4 by @dependabot in #1146
• Bump idna from 3.4 to 3.7 by @dependabot in #1143
• Bump ejs from 3.1.9 to 3.1.10 in /mephisto/review_app/client by @dependabot in #1150
• Bump follow-redirects from 1.15.4 to 1.15.6 by @dependabot in #1154
• Bump @storybook/react from 6.5.16 to 8.0.10 by @dependabot in #1152
• Bump ejs from 3.1.8 to 3.1.10 by @dependabot in #1149
• Bump mypy from 0.971 to 1.10.0 by @dependabot in #1147
• Bump tar from 6.1.13 to 6.2.1 by @dependabot in #1142
• Added unittests for Data Porter command options by @meta-paul in #1156
• Bump codecov/codecov-action from 3 to 4 by @dependabot in #1115
• Bump express from 4.18.2 to 4.19.2 by @dependabot in #1135
• Bump pyyaml from 5.4.1 to 6.0.1 by @dependabot in #1125
• Bump express from 4.18.3 to 4.19.2 in /mephisto/review_app/client by @dependabot in #1136
• Bump pdoc from 10.0.4 to 14.4.0 by @dependabot in #1123

Full Changelog: v1.3.0...v1.4.0

v1.3.0

Overview

This release introduces FormComposer task generator. It aims to simplify creating React front-end for form-based Tasks. Main highlights:

• Task front-end can be defined via JSON configuration (no need for custom code)
• Multiple form versions are supported within a single Task
• Multi-version forms can be generated within the same Task, based on variables' values
• Expiring (presigned) URLs are supported for S3 files (for data safety)
• Task Review app support includes preview of uploaded media files

Included PRs

• Mephisto v1.3 by @meta-paul in #1102

Full Changelog: v1.2.1...v1.3.0

v1.2.1

Overview

This release rounded out TaskReview app functionality. Now you can conveniently give feedback and bonuses to individual workers while performing review of their submissions. This feature works with all currently supported providers.

Included PRs

• V1.2.1 dev by @meta-paul in #1100

Full Changelog: v1.2.0...v1.2.1

Mephisto 1.2.0

Overview

This release is focused on adding a new provider (Prolific) and better data review experience.
It has a larger diff than usual, and we will return to a more frequent release cadence afterwards.

See the list of changes when upgrading from v1.1.0 below. Note that only essential dependency updates are listed.

Big Changes

• Added integration with Prolific provider, as an alternative to Mturk for data collection. Expect some minor additions in the subsequent releases, but the core functionality is already there. #1008
• Added UI-driven review of collected task results, a major upgrade over the previously used command-line script. Support for bonuses and feedback for workers to be added in the subsequent releases. When jobs have had no activity for a set duration, they'll be shut down automatically (suspecting an issue). #1058
• Added dockerization support for running Mephisto. #1008

Included PRs

Feature + UX Improvements

• Getting complete MTurk pay for individual units by @JackUrb in #446
• Adding gold units and functionality by @JackUrb in #923
• including hydra_configs in package (for 1.1.1) by @JackUrb in #942
• Adding missing hook registration for MTurk Sandbox wrap_crowd_source by @JackUrb in #987
• Reorganizing pyproject.toml to use Extras over Groups by @JackUrb in #1002
• Added AWS credentials pre-commit hook by @meta-paul in #1018
• Update heroku_architect.py by @JackUrb in #1025
• ec2 architect uninterruptable shutdown by @JackUrb in #1023
• Permissive socket upload by @JackUrb in #1026
• Adding auth implementation of prolific client by @JackUrb in #1028
• Add Prolific provider (#1008) by @meta-paul in #1012
• Improve logger in Python code by @meta-paul in #1045
• Laying down basic strokes of standard get_data response by @JackUrb in #1065
• Prolific workspace management by @JackUrb in #1067
• Update heroku_architect.py by @JackUrb in #1062
  1076
• Create a reusable app for task review by @meta-paul in #1058
• Prevent oversampling of study submissions by any single worker by @meta-paul in #1080

Bugfixes

• Fixed Static task to not explicitly check for objData.file1 by @gzhihongwei in #966
• Refactoring QA from unit assign to use in onboarding by @JackUrb in #960
• dropped the self from cant_cancel_expirations by @arjunan-k in #971
• Fixing missing events in wrap_crowd_source.js by @JackUrb in #988
• InaccessiblePrometheusServer Exception by @mojtaba-komeili in #1016
• Updated error message to manually delete EC2 server by @gzhihongwei in #1007
• quick fix: change order of checks in operator by @PReithofer in #1039
• Small fixes from local branch by @JackUrb in #1054
• fix race condition in generate_units by @PReithofer in #1056

Docs + Examples

• Corrected documentation inconsistencies and typos by @gzhihongwei in #965
• Adding citation note to readme by @JackUrb in #989
• Fix typo in operations/README.md by @eltociear in #992
• Add additional python versions to test by @pringshia in #925
• fix rendering of assignment url for mock tasks by @edwardguo61 in #1005
• New testing option for ec2 (reuse existing server) by @JackUrb in #1036
• Small typo fix for running the static_test_script by @benmatselby in #1019
• Add unittests for Prolific integration by @meta-paul in #1046
• running black across codebase by @JackUrb in #1043
• Add unittests for Prolific integration (2) by @meta-paul in #1047
• Updated Prolific integrations docs by @meta-paul in #1049
• Clean up TODO comments by @meta-paul in #1050
• Update copyright headers across the codebase by @meta-paul in #1075

Dependencies

• Update to React 18 by @pringshia in #905
• Upgrading codecov version by @JackUrb in #1027

New Contributors

• @gzhihongwei made their first contribution in #965
• @arjunan-k made their first contribution in #971
• @eltociear made their first contribution in #992
• @edwardguo61 made their first contribution in #1005
• @benmatselby made their first contribution in #1019
• @meta-paul made their first contribution in #1018
• @PReithofer made their first contribution in #1039

Full Changelog: v1.1.0...v1.2.0

Mephisto 1.1.0

Overview

This release contains a number of Stability and Quality of Life improvements from v1.0.0 and v1.0.2. The big focus has been on collection success rate, documentation, and various bugfixes.

For those updating from 1.0, see the 1.0.2 changes as well for a complete changelog.

Breaking changes

• Mephisto no longer supports Python 3.7. This is as of #898, wherein to get the ParlAI peer imports working we needed to bump to ParlAI 1.7.0, which no longer supports Python 3.7.

Big Changes

• When launching on MTurk, Mephisto will request that you run the mephisto scripts mturk cleanup script inline before launching. This exercise prevents having long-running tasks that haven't been cleaned up properly. #918
• When jobs have had no activity for a set duration, they'll be shut down automatically (suspecting an issue). #919

Big Fixes

• Lots of documentation, thanks @Etesam913!
• Significant fixes to proper execution and state tracking in Mephisto, which should improve stability across all job types (#875, #878, #902, #922)
• Major fix for Heroku tasks using the useMephistoLiveTask hook, which was causing all messages to be double sent in both directions. (#921)

Included PRs

Feature + UX Improvements

• Enhance logging messages with rich by @Etesam913 in #869
• Added ability to customize html-static-task-example submit button by @Etesam913 in #879
• Adds ability to clear onboarding for a worker by @Etesam913 in #886
• Adding more general function with better name by @mojtaba-komeili in #893
• Making MTurk cleanup run daily per requester by @JackUrb in #918
• 12 hour inactivity shutdown by @JackUrb in #919

Bugfixes

• [Bugfix] Fixing double update bug on heroku by @JackUrb in #921
• Fixing Worker Pool race condition for assignments by @JackUrb in #875
• ParlAI Assignment Fixes pt 2 by @JackUrb in #878
• Resolving static return status bug by @JackUrb in #902
• Make self.metadata backward-compatible by @chiehminwei in #861
• Finding OSTYPE if not exists by @mojtaba-komeili in #892
• fix breaks in create_validation_function for screening units by @macota in #864
• Fix mephisto web cli command by @Etesam913 in #865
• Removing threadsafety as we already lock connections behind threads by @JackUrb in #897
• Extending some 1.1 changes by @JackUrb in #922

Testing & DevX

• Add path filter by @Etesam913 in #868
• Add cypress test to simple static task by @Etesam913 in #881
• fix #874: Upgrade pip and create main_data_directory in Dockerfile by @mjkanji in #917
• pre-commit install by @mojtaba-komeili in #894
• Some fixes made to run a live demo of Mephisto functionality by @JackUrb in #909

Docs + Examples

• Adds cypress section to frontend docs by @Etesam913 in #860
• Adds tips/feedback section to docs by @Etesam913 in #838
• Add tests to tips example by @Etesam913 in #833
• Add searchbar to docs by @Etesam913 in #858
• Update searchbar colors by @Etesam913 in #863
• Auto Generate Blueprint Documentation by @Etesam913 in #870
• Auto Generate Architect Documentation by @Etesam913 in #871
• Auto Generate Requester Documentation by @Etesam913 in #872
• Auto Generate Provider Documentation by @Etesam913 in #873
• Update deploy docs github action to auto-generate docs by @Etesam913 in #877
• Add Screening Example by @Etesam913 in #883
• Creates a readme for simple_static_task by @Etesam913 in #882
• Improve onboarding example by @Etesam913 in #887
• Add info on link_task_source to docs by @Etesam913 in #885
• Update documentation to reflect that make_qualification_dict takes 3 arguments. by @xksteven in #908

Dependencies

• Bump terser from 5.12.1 to 5.14.2 in /docs/web by @dependabot in #853
• Update @blueprintjs/core version by @pringshia in #876
• Bump oauthlib from 3.2.0 to 3.2.1 by @dependabot in #901
• Bump protobuf from 3.19.4 to 3.19.5 by @dependabot in #903
• Bump joblib from 1.1.0 to 1.2.0 by @dependabot in #904
• Upgrading to poetry 1.2, resolving dependency issues by @JackUrb in #898
• Upgrade yarn 3.2.1->3.2.4 by @pringshia in #920

New Contributors

• @chiehminwei made their first contribution in #861
• @mjkanji made their first contribution in #917

Full Changelog: v1.0.3...v1.1.0

v1.0.3: Last version, but ParlAI-stable

Overview

v1.0.2 shipped with a bug that broke ParlAI-chat related tasks. This release only includes the fix for that (#859), but is otherwise equivalent to v1.0.2.