Flutter realtime object detection with Tensorflow Lite
Flutter realtime object detection with Tensorflow Lite
Flutter realtime object detection with Tensorflow Lite
Info
An app made with Flutter and TensorFlow Lite for realtime object detection using model YOLO, SSD, MobileNet, PoseNet.
⭐ Features
-
Realtime object detection on the live camera
-
Using Model: YOLOv2-Tiny, SSDMobileNet, MobileNet, PoseNet
-
Save image has been detected
-
MVVM architecture
🚀 Installation
- Install Packages
camera: get the streaming image buffers
https://pub.dev/packages/camera
tflite: run model TensorFlow Lite
https://pub.dev/packages/tflite
provider: state management
https://pub.dev/packages/provider
2. Configure Project
- Android
android/app/build.gradle
android {
...
aaptOptions {
noCompress 'tflite'
noCompress 'lite'
}
...
}
minSdkVersion 21
3. Load model
loadModel() async {
Tflite.close();
await Tflite.loadModel(
model: "assets/models/yolov2_tiny.tflite",
//ssd_mobilenet.tflite, mobilenet_v1.tflite, posenet_mv1_checkpoints.tflite
labels: "assets/models/yolov2_tiny.txt",
//ssd_mobilenet.txt, mobilenet_v1.txt
//numThreads: 1, // defaults to 1
//isAsset: true, // defaults: true, set to false to load resources outside assets
//useGpuDelegate: false // defaults: false, use GPU delegate
);
}
4. Run model
For Realtime Camera
//YOLOv2-Tiny
Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
var recognitions = await Tflite.detectObjectOnFrame(
bytesList: image.planes.map((plane) {
return plane.bytes;
}).toList(),
model: "YOLO",
imageHeight: image.height,
imageWidth: image.width,
imageMean: 0, // defaults to 127.5
imageStd: 255.0, // defaults to 127.5
threshold: 0.2, // defaults to 0.1
numResultsPerClass: 1,
);
return recognitions;
}
//SSDMobileNet
Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
var recognitions = await Tflite.detectObjectOnFrame(
bytesList: image.planes.map((plane) {
return plane.bytes;
}).toList(),
model: "SSDMobileNet",
imageHeight: image.height,
imageWidth: image.width,
imageMean: 127.5,
imageStd: 127.5,
threshold: 0.4,
numResultsPerClass: 1,
);
return recognitions;
}
//MobileNet
Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
var recognitions = await Tflite.runModelOnFrame(
bytesList: image.planes.map((plane) {
return plane.bytes;
}).toList(),
imageHeight: image.height,
imageWidth: image.width,
numResults: 5
);
return recognitions;
}
//PoseNet
Future<List<dynamic>?> runModelOnFrame(CameraImage image) async {
var recognitions = await Tflite.runPoseNetOnFrame(
bytesList: image.planes.map((plane) {
return plane.bytes;
}).toList(),
imageHeight: image.height,
imageWidth: image.width,
numResults: 5
);
return recognitions;
}
For Image
Future<List<dynamic>?> runModelOnImage(File image) async {
var recognitions = await Tflite.detectObjectOnImage(
path: image.path,
model: "YOLO",
threshold: 0.3,
imageMean: 0.0,
imageStd: 127.5,
numResultsPerClass: 1
);
return recognitions;
}
Output format:
YOLO,SSDMobileNet
[{
detectedClass: "dog",
confidenceInClass: 0.989,
rect: {
x: 0.0,
y: 0.0,
w: 100.0,
h: 100.0
}
},...]
MobileNet
[{
index: 0,
label: "WithMask",
confidence: 0.989
},...]
PoseNet
[{
score: 0.5,
keypoints: {
0: {
x: 0.2,
y: 0.12,
part: nose,
score: 0.803
},
1: {
x: 0.2,
y: 0.1,
part: leftEye,
score: 0.8666
},
...
}
},...]
5. Issue
* IOS
Downgrading TensorFlowLiteC to 2.2.0
Downgrade your TensorFlowLiteC in /ios/Podfile.lock to 2.2.0
run pod install in your /ios folder
6. Source code
please checkout repo github
https://github.com/hiennguyen92/flutter_realtime_object_detection
💡 Demo
- Demo Illustration: https://www.youtube.com/watch?v=__i7PRmz5kY&ab_channel=HienNguyen
- Image