Flutter realtime object detection with Tensorflow Lite

Flutter realtime object detection with Tensorflow Lite

Info

An app made with Flutter and TensorFlow Lite for realtime object detection using model YOLO, SSD, MobileNet, PoseNet.

⭐ Features

  • Realtime object detection on the live camera

  • Using Model: YOLOv2-Tiny, SSDMobileNet, MobileNet, PoseNet

  • Save image has been detected

  • MVVM architecture

🚀  Installation

  1. Install Packages

camera: get the streaming image buffers
https://pub.dev/packages/camera


tflite: run model TensorFlow Lite
https://pub.dev/packages/tflite


provider: state management
https://pub.dev/packages/provider

2. Configure Project

  • Android

android/app/build.gradle

android {
    ...
    aaptOptions {
        noCompress 'tflite'
        noCompress 'lite'
    }
    ...
}


minSdkVersion 21

3. Load model

loadModel() async {
    Tflite.close();
    await Tflite.loadModel(
        model: "assets/models/yolov2_tiny.tflite",  
        //ssd_mobilenet.tflite, mobilenet_v1.tflite, posenet_mv1_checkpoints.tflite
        labels: "assets/models/yolov2_tiny.txt",    
        //ssd_mobilenet.txt, mobilenet_v1.txt
        //numThreads: 1, // defaults to 1
        //isAsset: true, // defaults: true, set to false to load resources outside assets
        //useGpuDelegate: false // defaults: false, use GPU delegate
    );
  }

4. Run model

For Realtime Camera

  //YOLOv2-Tiny
  Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
     var recognitions = await Tflite.detectObjectOnFrame(
          bytesList: image.planes.map((plane) {
            return plane.bytes;
          }).toList(),
          model: "YOLO",
          imageHeight: image.height,
          imageWidth: image.width,
          imageMean: 0,                 // defaults to 127.5
          imageStd: 255.0,              // defaults to 127.5
          threshold: 0.2,               // defaults to 0.1
          numResultsPerClass: 1,
        );   
    return recognitions;
  }

  //SSDMobileNet
  Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
     var recognitions = await Tflite.detectObjectOnFrame(
          bytesList: image.planes.map((plane) {
            return plane.bytes;
          }).toList(),
          model: "SSDMobileNet",
          imageHeight: image.height,
          imageWidth: image.width,
          imageMean: 127.5,
          imageStd: 127.5,
          threshold: 0.4,
          numResultsPerClass: 1,
        );   
    return recognitions;
  }

  //MobileNet
  Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
     var recognitions = await Tflite.runModelOnFrame(
          bytesList: image.planes.map((plane) {
            return plane.bytes;
          }).toList(),
          imageHeight: image.height,
          imageWidth: image.width,
          numResults: 5
        );   
    return recognitions;
  }

  //PoseNet
  Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
     var recognitions = await Tflite.runPoseNetOnFrame(
          bytesList: image.planes.map((plane) {
            return plane.bytes;
          }).toList(),
          imageHeight: image.height,
          imageWidth: image.width,
          numResults: 5
        );   
    return recognitions;
  }

For Image

  Future<List<dynamic>?> runModelOnImage(File image) async {
    var recognitions = await Tflite.detectObjectOnImage(
        path: image.path,
        model: "YOLO",
        threshold: 0.3,
        imageMean: 0.0,
        imageStd: 127.5,
        numResultsPerClass: 1
    );
    return recognitions;
  }


Output format:

YOLO,SSDMobileNet
  [{
    detectedClass: "dog",
    confidenceInClass: 0.989,
    rect: {
        x: 0.0,
        y: 0.0,
        w: 100.0,
        h: 100.0
    }
  },...]

MobileNet
[{
    index: 0,
    label: "WithMask",
    confidence: 0.989
  },...]

PoseNet
[{
    score: 0.5,
    keypoints: {
        0: {
            x: 0.2,
            y: 0.12,
            part: nose,
            score: 0.803
        },
        1: {
            x: 0.2,
            y: 0.1,
            part: leftEye,
            score: 0.8666
        },
        ...
    }
  },...]

5. Issue

* IOS
Downgrading TensorFlowLiteC to 2.2.0

Downgrade your TensorFlowLiteC in /ios/Podfile.lock to 2.2.0
run pod install in your /ios folder

6. Source code

please checkout repo github
https://github.com/hiennguyen92/flutter_realtime_object_detection

💡 Demo

  1. Demo Illustration: https://www.youtube.com/watch?v=__i7PRmz5kY&ab_channel=HienNguyen
  2. Image

GitHub

View Github