How to get Prediction results(prediction class) using tflite model (using custom dataset to train the Yolov11) in tensorflow js (want to use in Nextjs)?

[BiswajitAich]

I trained the Yolov11 model for ROCK-PAPER-SCISSOR dataset(roboflow). It's predicting the image with good accuracy. Then I converted it to tflite to use it as a website (Nextjs framework). To get predicted class I was trying to use tensorflow js ! But using python I am getting the predicted class using loop( as shown in documentation) but I am not getting the same using tf.js ; I also tried onnx format still can't get the predicted class.

( I also tried as python api it's working as intended but can't use it as in free render version it's not returning the response only works in my desktop so no use. )

Please help/guide me!. It will be helpful if a sample could be provided to me.

Thank you

[UltralyticsAssistant]

👋 Hello @BiswajitAich, thank you for your interest in Ultralytics 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom deployment ❓ Question, we recommend providing additional details about your workflow, such as:

• Region of interest and usage constraints
• TFLite model architecture and how you've converted it (e.g., tools, steps)
• TensorFlow.js code snippets and issues encountered
• Any error logs or unexpected behaviors, particularly with ONNX or tf.js
• Your specific Next.js setup or integrations

For maintaining minimal latency in web apps like Next.js, you can also explore ONNXruntime.js or TensorFlow.js for optimized inference pipelines. Details included in the model API routing<here context

[BiswajitAich]

Using tensorflow js or similar to that how to get the predicted class to work directly with Nextjs framework .

I get the results as I need using python as backend api but render not giving response back [code is working as intended in my desktop] so I want to run the model in user device with js . How should I do that!

app.py used asi api ...:

from ultralytics import YOLO
from flask import Flask, request, jsonify
from PIL import Image
import os
from flask_cors import CORS
app = Flask(__name__)
CORS(app)
MODEL_PATH = "model/rock-paper-scissors-best-4-int8.tflite"
model = None
@app.route('/predict', methods=['POST'])
def predict():
    if 'image' not in request.files:
        return jsonify({'err': 'No image file provided'}), 400
    image_file = request.files['image']
    allowed_extensions = {'png', 'jpg', 'jpeg', 'gif'}
    if not image_file.filename.lower().split('.')[-1] in allowed_extensions:
        return jsonify({'error': 'Invalid file type'}), 400
    try:
        img = Image.open(image_file.stream).convert('RGB')
        img = img.resize((64, 64))
    except Exception as e:
        return jsonify({'error': f'Failed to process image---: {str(e)}'}), 400
    try:
        results = model(img, imgsz=64)
        if results and results[0].boxes is not None and len(results[0].boxes) > 0:
            class_idx = int(results[0].boxes[0].cls.item()) 
            predicted_class = results[0].names[class_idx]   
            return jsonify({'result': predicted_class}), 200
        else:
            return jsonify({'result': 'none'}), 200
    except Exception as e:
        return jsonify({'error': f'YOLO inference failed: {str(e)}'}), 500

def load_model():
    global model
    model = YOLO(MODEL_PATH)


load_model()

@app.route('/health', methods=['GET'])
def health_check():
    return jsonify({'status': 'OK'}), 200
    
if __name__ == '__main__':
    port = int(os.environ.get("PORT", 5000))
    app.run(debug=False, host='0.0.0.0', port=port)

[Y-T-G]

You can try LitServe

https://lightning.ai/docs/litserve/features/request-response-format

[BiswajitAich]

Ok, let me check it (lightning. ai) if it works (ie the api) then there is no need for the js(tensorflow js or something else).
I will give you reply if it's works ..!
Thank you ☺️...
Still is it not possible to do it in js ?
Though let me try it(lightning) first 🐱

[glenn-jocher]

You're welcome! Yes, it is possible to run YOLO11 models in JavaScript using TensorFlow.js after exporting the model to the TF.js format. You can refer to our documentation on TF.js integration for guidance. Let us know how it goes with Lightning or if you need further assistance with the JS approach!

[Y-T-G]

You can check this example on how to preprocesss and postprocess.

https://github.com/ultralytics/ultralytics/tree/main/examples/YOLOv8-TFLite-Python

You will have to implement similar pre and post processing.

[BiswajitAich]

I had seen this one (GitHub link) it's in python, I am getting the results (prediction class) in python no problem there but how to do it in tensorflow js or something similar to it to directly run/use with Nextjs framework.

I did run the code using api(deployed in render) but not getting response. [Api working in my desktop].

app.py deployed in render ..:

from ultralytics import YOLO
from flask import Flask, request, jsonify
from PIL import Image
import os
from flask_cors import CORS
app = Flask(__name__)
CORS(app)
MODEL_PATH = "model/rock-paper-scissors-best-4-int8.tflite"
model = None
@app.route('/predict', methods=['POST'])
def predict():
    if 'image' not in request.files:
        return jsonify({'err': 'No image file provided'}), 400
    image_file = request.files['image']
    allowed_extensions = {'png', 'jpg', 'jpeg', 'gif'}
    if not image_file.filename.lower().split('.')[-1] in allowed_extensions:
        return jsonify({'error': 'Invalid file type'}), 400
    try:
        img = Image.open(image_file.stream).convert('RGB')
        img = img.resize((64, 64))
    except Exception as e:
        return jsonify({'error': f'Failed to process image---: {str(e)}'}), 400
    try:
        results = model(img, imgsz=64)
        if results and results[0].boxes is not None and len(results[0].boxes) > 0:
            class_idx = int(results[0].boxes[0].cls.item()) 
            predicted_class = results[0].names[class_idx]   
            return jsonify({'result': predicted_class}), 200
        else:
            return jsonify({'result': 'none'}), 200
    except Exception as e:
        return jsonify({'error': f'YOLO inference failed: {str(e)}'}), 500

def load_model():
    global model
    model = YOLO(MODEL_PATH)


load_model()

@app.route('/health', methods=['GET'])
def health_check():
    return jsonify({'status': 'OK'}), 200
    
if __name__ == '__main__':
    port = int(os.environ.get("PORT", 5000))
    app.run(debug=False, host='0.0.0.0', port=port)