A flutter offline package for a fast BPE tokeniser for OpenAI models
flutter_tiktoken
flutter_tiktoken is a flutter offline package for a fast BPE tokeniser for OpenAI models.
flutter_tiktoken is a copy package in https://github.com/mvitlov/tiktoken and fixed the issue build spent many time (more than one hour) when build in release mode.
| Open Ai Official Calcuate | flutter_tiktoken |
|---|---|
Usage
init flutter_tiktoken
void main() async {
WidgetsFlutterBinding.ensureInitialized();
await TiktokenDataProcessCenter().initata();
runApp(const MyApp());
}
calcuateTokens
static int _calcuateTokens(
{required String text, String modelName = 'gpt-4'}) {
final encoding = encodingForModel(modelName);
final numTokens = encoding.encode(text).length;
return numTokens;
}