Run model-in-the-loop tasks
Mephisto allows to create interactive Tasks where worker can interact with the server during completion of their Task.
Interactive Image Generation Task
A good example Task demonstrating this is "Interactive Generation of a Look-Alike Image"
Task Flow
At a high level, here's what this Task entails:
- Worker's goal is to compose a Generative AI text prompt to produce an image that closely resembles initially displayed "target" image
- Worker is given up to
max_answer_loops
attempts to achieve the goal- Each attempt consists of judging the generated image (with the resemblance
score
), and updating the text prompt- Worker cannot reuse exactly the same text prompt from any of the previous attempts
- Goal is reached if worker indicates
score=10
in their response (perfect resemblance is achieved)
- Each attempt consists of judging the generated image (with the resemblance
- After receiving each attempt, based on the updated text prompt the server produces a new image and extra instruction
- Here we're picking a random locally stored image, but this could also be a request to a 3rd party API
- Extra instruction shown to worker will vary, depending on provided score
Technical Notes
Some technical aspects are worth pointing out, to understand this example better:
- This Task is based on our FormComposer feature since worker input is form-based
- We make use of custom triggers to send server requests invoking server-side (called "remote") procedures
We recommend examining source code for a deeper look, in particular:
- run_tasklocalinhouse.py - see how task is being prepared and the logic of remote procedure (that simulates the image generation part)
- data - see FormComposer config used for this Task
- custom_triggers.js -
see the UI logic behind clicking "Start" and "Update" buttons in the form:
- preparing request data
- sending server request
- processing server response
- updating form's React state
Run with Docker
To launch this example from within a Docker container follow these steps:
(Optional) You can modify, then prepare and validate your config files as described in FormComposer config files and form_composer config command sections
Launch the example for a local testing scenario:
docker-compose -f docker/docker-compose.dev.yml run \
--build \
--publish 8081:8000 \
--publish 3001:3000 \
--rm mephisto_dc \
python /mephisto/examples/remote_procedure/interactive_image_generation/run_task__local__inhouse.py
- Once your Task launches, your console will display you URLs like this:
http://localhost:3001?worker_id=WORKER_USERNAME&id=1
Further Details
You can learn more in these sections: