send MultiModal chat messages

[FlyWieEinAirplane]

So I would like to implement something that is capable of sending an image and text in one message to ChatGPT.

I had a look at what ChatGPT is doing under the hood - it is uploading the image and then having a pointer to the image inside the /conversation message. Also some values in the request are "multimodal" instead of "text". Not sure if it was necessary but I implemented the upload of the image but now I am sort of stuck sending the message with the file pointer.

Does anyone have an idea on how to enable multimodality with an event or similar so that when sending the message via chagpt.js, it is going to pick up on the file that was uploaded and then send both in one message?
Any input is welcome :)

[FlyWieEinAirplane]

I just found #175 which is asking for file upload which is similar.
But I just need to understand how it is supposed to work, then I can implement it myself (and merge back here)

[adamlui]

You should share your code then me or others could maybe build on that

[FlyWieEinAirplane]

So I have a working version now. I removed all the manual http calls and focussed more on the ChatBox and how to paste the file in there. Works 10/10 times :)

This generally works for files. I am generating the image/blob in memory but it could be adapted to take a file from file system or wherever afaict.

function sendBlobToChatBox(blob) {
      if (!blob) {
          console.error('Invalid blob');
          return;
      }

      // Find the target editable textfield div
      const textField = chatgpt.getChatBox();
      if (!textField) {
          console.error('Target textfield not found');
          alert('Target textfield not found');
          return;
      }

      // Create a DataTransfer object to hold the image data
      const dataTransfer = new DataTransfer();
      const file = new File([blob], 'image.png', { type: 'image/png' });
      dataTransfer.items.add(file);

      // Create and dispatch a paste event with the DataTransfer object
      const pasteEvent = new ClipboardEvent('paste', {
          bubbles: true,
          cancelable: true,
          clipboardData: dataTransfer
      });

      textField.dispatchEvent(pasteEvent);
      chatgpt.send('here is the image');
}