Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds mediapipe_core package #11

Merged
merged 17 commits into from
Nov 13, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Runs unit tests for `mediapipe_core`
test_core:
cd packages/mediapipe-core && dart test

# Runs the utility to pull in all header files from `google/mediapipe`
headers:
craiglabenz marked this conversation as resolved.
Show resolved Hide resolved
cd packages/build_cmd && dart bin/main.dart headers

# Runs `ffigen` for `mediapipe_core`
generate_core:
cd packages/mediapipe-core && dart run ffigen --config=ffigen.yaml

# Runs `ffigen` for `mediapipe_text`
generate_text:
cd packages/mediapipe-task-text && dart run ffigen --config=ffigen.yaml

# Runs `ffigen` for all packages
generate: generate_core generate_text

# Compiles the faked C artifacts for testing
compile_fake_text:
# Builds standalone executable
cd packages/mediapipe-task-text/test/c && gcc fake_text_classifier.c -o fake_text_classifier
# Builds what Flutter needs
cd packages/mediapipe-task-text/test/c && gcc -static -c -fPIC *.c -o fake_text_classifier.o
cd packages/mediapipe-task-text/test/c && gcc -shared -o fake_text_classifier.dylib fake_text_classifier.o

# Runs all text tests
test_text_only:
cd packages/mediapipe-task-text && flutter test

# Runs `ffigen` for `mediapipe_text` and all text tests
test_text: compile_fake_text test_text_only

# Runs `ffigen` for all packages, compiles the faked C artifacts, and runs all tests
test: generate_text test_text generate_core test_core

# Runs `ffigen` for all packages and all tests for all packages
test_only: test_core test_text

7 changes: 7 additions & 0 deletions packages/mediapipe-core/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# https://dart.dev/guides/libraries/private-files
# Created by `dart pub`
.dart_tool/

# Avoid committing pubspec.lock for library packages; see
# https://dart.dev/guides/libraries/private-files#pubspeclock.
pubspec.lock
3 changes: 3 additions & 0 deletions packages/mediapipe-core/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## 1.0.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 1.0? Should we wait until 1.0 is a ready for use package?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh for sure. This was auto-generated and I never really looked at the file 😂


- Initial version.
39 changes: 39 additions & 0 deletions packages/mediapipe-core/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
<!--
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: Come back and review this once filled out

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'tis filled out 🎉

This README describes the package. If you publish this package to pub.dev,
this README's contents appear on the landing page for your package.
For information about how to write a good package README, see the guide for
[writing package pages](https://dart.dev/guides/libraries/writing-package-pages).
For general information about developing packages, see the Dart guide for
[creating packages](https://dart.dev/guides/libraries/create-library-packages)
and the Flutter guide for
[developing packages and plugins](https://flutter.dev/developing-packages).
-->

TODO: Put a short description of the package here that helps potential users
know whether this package might be useful for them.

## Features

TODO: List what your package can do. Maybe include images, gifs, or videos.

## Getting started

TODO: List prerequisites and provide or point to information on how to
start using the package.

## Usage

TODO: Include short and useful examples for package users. Add longer examples
to `/example` folder.

```dart
const like = 'sample';
```

## Additional information

TODO: Tell users more about the package: where to find more information, how to
contribute to the package, how to file issues, what response they can expect
from the package authors, and more.
7 changes: 7 additions & 0 deletions packages/mediapipe-core/analysis_options.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Specify analysis options.

include: package:lints/recommended.yaml

analyzer:
exclude:
- "lib/src/mediapipe_common_bindings.dart"
13 changes: 13 additions & 0 deletions packages/mediapipe-core/ffigen.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: "MediaPipeCommonBindings"
description: "Bindings for shared MediaPipe structs common across many tasks"
output:
craiglabenz marked this conversation as resolved.
Show resolved Hide resolved
bindings: "lib/src/mediapipe_common_bindings.dart"
symbol-file:
output: "package:mediapipe_core/generated/core_symbols.yaml"
import-path: "package:mediapipe_core/src/mediapipe_common_bindings.dart"
headers:
entry-points:
- "third_party/mediapipe/tasks/c/core/base_options.h"
craiglabenz marked this conversation as resolved.
Show resolved Hide resolved
- "third_party/mediapipe/tasks/c/components/containers/category.h"
- "third_party/mediapipe/tasks/c/components/containers/classification_result.h"
- "third_party/mediapipe/tasks/c/components/processors/classifier_options.h"
38 changes: 38 additions & 0 deletions packages/mediapipe-core/lib/generated/core_symbols.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
format_version: 1.0.0
files:
package:mediapipe_core/src/mediapipe_common_bindings.dart:
used-config:
ffi-native: false
symbols:
c:@S@BaseOptions:
name: BaseOptions
c:@S@Category:
name: Category
c:@S@ClassificationResult:
name: ClassificationResult
c:@S@Classifications:
name: Classifications
c:@S@ClassifierOptions:
name: ClassifierOptions
c:@S@__darwin_pthread_handler_rec:
name: __darwin_pthread_handler_rec
c:@S@_opaque_pthread_attr_t:
name: _opaque_pthread_attr_t
c:@S@_opaque_pthread_cond_t:
name: _opaque_pthread_cond_t
c:@S@_opaque_pthread_condattr_t:
name: _opaque_pthread_condattr_t
c:@S@_opaque_pthread_mutex_t:
name: _opaque_pthread_mutex_t
c:@S@_opaque_pthread_mutexattr_t:
name: _opaque_pthread_mutexattr_t
c:@S@_opaque_pthread_once_t:
name: _opaque_pthread_once_t
c:@S@_opaque_pthread_rwlock_t:
name: _opaque_pthread_rwlock_t
c:@S@_opaque_pthread_rwlockattr_t:
name: _opaque_pthread_rwlockattr_t
c:@S@_opaque_pthread_t:
name: _opaque_pthread_t
c:@UA@__mbstate_t:
name: __mbstate_t
5 changes: 5 additions & 0 deletions packages/mediapipe-core/lib/mediapipe_core.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
library;
craiglabenz marked this conversation as resolved.
Show resolved Hide resolved

export 'src/containers.dart' show Category, Classifications;
export 'src/ffi_utils.dart';
export 'src/task_options.dart' show BaseOptions, ClassifierOptions;
109 changes: 109 additions & 0 deletions packages/mediapipe-core/lib/src/containers.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
import 'dart:ffi';
import 'package:ffi/ffi.dart';
import 'package:mediapipe_core/mediapipe_core.dart';
import 'mediapipe_common_bindings.dart' as bindings;

class Category {
craiglabenz marked this conversation as resolved.
Show resolved Hide resolved
const Category({
required this.index,
required this.score,
required this.categoryName,
required this.displayName,
});
final int index;
final double score;
final String? categoryName;
final String? displayName;

static List<Category> fromStructs(
Pointer<bindings.Category> structs,
int count,
) {
final categories = <Category>[];
for (int i = 0; i < count; i++) {
categories.add(fromStruct(structs[i]));
}
return categories;
}

static Category fromStruct(bindings.Category struct) {
return Category(
index: struct.index,
score: struct.score,
categoryName: toDartString(struct.category_name),
displayName: toDartString(struct.display_name),
);
}

static void freeStructs(Pointer<bindings.Category> structs, int count) {
int index = 0;
while (index < count) {
bindings.Category obj = structs[index];
calloc.free(obj.category_name);
calloc.free(obj.display_name);
index++;
}
calloc.free(structs);
}

@override
String toString() => 'Category(index=$index, score=$score, '
'categoryName=$categoryName, displayName=$displayName)';
}

class Classifications {
const Classifications({
required this.categories,
required this.headIndex,
required this.headName,
});
final List<Category> categories;
final int headIndex;
final String? headName;

static List<Classifications> fromStructs(
Pointer<bindings.Classifications> structs,
int count,
) {
final classifications = <Classifications>[];
for (int i = 0; i < count; i++) {
classifications.add(fromStruct(structs[i]));
}
return classifications;
}

static Classifications fromStruct(bindings.Classifications struct) {
return Classifications(
categories: Category.fromStructs(
struct.categories,
struct.categories_count,
),
headIndex: struct.head_index,
headName: toDartString(struct.head_name),
);
}

static void freeStructs(
Pointer<bindings.Classifications> structs,
int count,
) {
int index = 0;
while (index < count) {
bindings.Classifications obj = structs[index];
Category.freeStructs(obj.categories, obj.categories_count);
calloc.free(obj.head_name);
index++;
}
calloc.free(structs);
}

Category? get firstCategory =>
categories.isNotEmpty ? categories.first : null;

@override
String toString() {
final categoryStrings = categories.map((cat) => cat.toString()).join(', ');
return 'Classification(categories=[$categoryStrings], '
'headIndex=$headIndex, headName=$headName)';
}
}
52 changes: 52 additions & 0 deletions packages/mediapipe-core/lib/src/ffi_utils.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
import 'dart:ffi';
craiglabenz marked this conversation as resolved.
Show resolved Hide resolved
import 'dart:typed_data';
import 'package:ffi/ffi.dart';

Pointer<Pointer<Char>> prepareListOfStrings(List<String> values) {
final ptrArray = calloc<Pointer<Char>>(values.length);
for (var i = 0; i < values.length; i++) {
ptrArray[i] = values[i].toNativeUtf8().cast<Char>();
}
return ptrArray;
}

Pointer<Char> prepareString(String val) => val.toNativeUtf8().cast<Char>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need both methods to be public since prepareListOfStrings calls this? Maybe prepareString -> _prepareString?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both functions are called from elsewhere - the list format method just makes use of the single-string variant in its loop.


String? toDartString(Pointer<Char> val) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here maybe? toDartString --> _toDartString

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above - this function is also necessarily public in the PRs for other tasks.

if (val == nullptr) return null;
return val.cast<Utf8>().toDartString();
}

List<String?> toDartStrings(Pointer<Pointer<Char>> val, int length) {
final dartStrings = <String?>[];
int counter = 0;
while (counter < length) {
dartStrings.add(toDartString(val[counter]));
counter++;
}
return dartStrings;
}

Pointer<Char> prepareUint8List(Uint8List ints) {
final Pointer<Uint8> ptr = calloc<Uint8>(ints.length);
ptr.asTypedList(ints.length).setAll(0, ints);
return ptr.cast();
}

Uint8List toUint8List(Pointer<Char> val, {int? length}) {
final codeUnits = val.cast<Uint8>();
if (length != null) {
RangeError.checkNotNegative(length, 'length');
} else {
length = _length(codeUnits);
}
return codeUnits.asTypedList(length);
}

int _length(Pointer<Uint8> codeUnits) {
var length = 0;
while (codeUnits[length] != 0) {
length++;
}
return length;
}
Loading