Fix idefics3

[Blaizzy]

This PR addresses two critical bugs that were introduced in PR #124:

• Corrects the image token placement strategy by moving tokens from last position to first position
• Resolves missing image token index (tracked in issue Fix broken stream generate for SmolVLM and others #132)

Changes Made

• Modified token placement algorithm to insert image tokens at the beginning of the sequence instead of appending them at the end
• Implemented proper index tracking and reporting for image tokens
• Added validation to ensure token indices are correctly maintained

Visual Demonstration

Input Image

Sample menu section used for testing:

Before ❌

Token placement and index reporting issues visible in these screenshots:

After ✅

Correct token placement and index reporting:

Testing

• Verified token placement in multiple scenarios
• Confirmed correct index reporting across different image types
• Validated backwards compatibility with existing implementations

Related Issues

• Fixes Fix broken stream generate for SmolVLM and others #132 (Missing image token index reporting)
• Resolves regression from Idefics 3 / SmolVLM support #124

[andimarafioti]

idefics3 and smolvlm have different language models and tokenizers! So you need to get this from the model

[andimarafioti]

model.config.image_token_id for idefics3

[Blaizzy]

Thanks for pointing it out!

The image_token_id and all other arguments will be overriden when a model with different config but same arch is loaded.