FFI: Custom open behavior, isolate API, docs

This commit is contained in:
Simon Binder 2019-09-22 14:21:45 +02:00
parent d6913af380
commit 0d56594933
No known key found for this signature in database
GPG Key ID: 7891917E4147B8C0
14 changed files with 552 additions and 32 deletions

View File

@ -1,38 +1,102 @@
# moor_ffi
Moor backend that uses `dart:ffi`. Note that, while we have integration tests
on this package, it depends on the `dart:ffi` apis, which are in "preview" status at the moment.
Thus, this library is not suited for production use.
Experimental bindings to sqlite by using `dart:ffi`. This library contains utils to make
integration with [moor](https://pub.dev/packages/moor) easier, but it can also be used
as a standalone package.
If you want to use moor on Android or iOS, see the [getting started guide](https://moor.simonbinder.eu/docs/getting-started/)
which recommends to use the [moor_flutter](https://pub.dev/packages/moor_flutter) package.
At the moment, this library is targeted at advanced moor users who want to try out the `ffi`
backend.
## Warnings
At the moment, `dart:ffi` is in preview and there will be breaking changes that this
library has to adapt to. This library has been tested on Dart `2.5.0`.
If you're using a development Dart version (this includes Flutter channels that are not
`stable`), this library might not work.
If you just want to use moor, using the [moor_flutter](https://pub.dev/packages/moor_flutter)
package is the better option at the moment.
## Supported platforms
At the moment, this plugin only supports Android without further work. However, it's also going
to run on all platforms that expose `sqlite3` as a shared native library (macOS and virtually
all Linux distros, I'm not sure about Windows). Native iOS and macOS support is planned.
As Flutter desktop doesn't support plugins on Windows and Linux yet, we can't bundle the
sqlite library on those platforms.
You can make this library work on any platform that let's you obtain a `DynamicLibrary`
from which moor_ffi loads the functions (see below).
Out of the box, this libraries supports all platforms where `sqlite3` is installed:
- iOS: Yes
- macOS: Yes
- Linux: Available on most distros
- Windows: When the user has installed sqlite (they probably have)
- Android: Yes when used with Flutter
This library works with and without Flutter.
If you're using Flutter, this library will bundle `sqlite3` in your Android app. This
requires the Android NDK to be installed (You can get the NDK in the [SDK Manager](https://developer.android.com/studio/intro/update.html#sdk-manager)
of Android Studio). Note that the first `flutter run` is going to take a very long time as
we need to compile sqlite.
### On other platforms
Using this library on platforms that are not supported out of the box is fairly
straightforward. For instance, if you release your own `sqlite3.so` with your application,
you could use
```dart
import 'dart:ffi';
import 'dart:io';
import 'package:moor_ffi/database.dart';
import 'package:moor_ffi/open_helper.dart';
void main() {
open.overrideFor(OperatingSystem.linux, _openOnLinux);
final db = Database.memory();
db.close();
}
DynamicLibrary _openOnLinux() {
final script = File(Platform.script.toFilePath());
final libraryNextToScript = File('${script.path}/sqlite3.so');
return DynamicLibrary.open(libraryNextToScript.path);
}
```
Just be sure to first override the behavior and then opening the database. Further,
if you want to use the isolate api, you can only use a static method or top-level
function to open the library.
### Supported datatypes
This library supports `null`, `int`, other `num`s (converted to double),
`String` and `Uint8List` to bind args. Returned columns from select statements
will have the same types.
## Using without moor
```dart
import 'package:moor_ffi/database.dart';
void main() {
final database = Database.memory();
// run some database operations. See the example for details
database.close();
}
```
You can also use an asynchronous API on a background isolate by using `IsolateDb.openFile`
or `IsolateDb.openMemory`, respectively. be aware that the asynchronous API is much slower,
but it moves work out of the UI isolate.
Be sure to __always__ call `Database.close` to avoid memory leaks!
## Migrating from moor_flutter
__Note__: For production apps, please use `moor_flutter` until this package
reaches a stable version.
Add both `moor` and `moor_ffi` to your pubspec, the `moor_flutter` dependency can be dropped.
```yaml
dependencies:
moor: ^2.0.0
moor: ^1.7.0
moor_ffi: ^0.0.1
dev_dependencies:
moor: ^2.0.0
moor_generator: ^1.7.0
```
In your main database file, replace the `package:moor_flutter/moor_flutter.dart` import with
`package:moor/moor.dart` and `package:moor_ffi/moor_ffi.dart`.
In the file where you created a `FlutterQueryExecutor`, replace the `moor_flutter` import
with both `package:moor/moor.dart` and `package:moor_ffi/moor_ffi.dart`.
In all other project files that use moor apis (e.g. a `Value` class for companions), just import `package:moor/moor.dart`.
Finally, replace usages of `FlutterQueryExecutor` with `VmDatabase`.
## Notes
After importing this library, the first Flutter build is going to take a very long time. The reason is that we're
compiling sqlite to bundle it with your app. Subsequent builds should take an acceptable time to execute.

View File

@ -0,0 +1,27 @@
import 'package:moor_ffi/database.dart';
const _createTable = r'''
CREATE TABLE frameworks (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
name VARCHAR NOT NULL
);
''';
void main() {
final db = Database.memory();
db.execute(_createTable);
final insertStmt = db.prepare('INSERT INTO frameworks(name) VALUES (?)');
insertStmt.execute(['Flutter']);
insertStmt.execute(['AngularDart']);
insertStmt.close();
final selectStmt = db.prepare('SELECT * FROM frameworks ORDER BY name');
final result = selectStmt.select();
for (var row in result) {
print('${row['id']}: ${row['name']}');
}
selectStmt.close();
db.close();
}

View File

@ -0,0 +1,28 @@
import 'package:moor_ffi/database.dart';
const _createTable = r'''
CREATE TABLE frameworks (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
name VARCHAR NOT NULL
);
''';
void main() async {
final db = await IsolateDb.openMemory();
await db.execute(_createTable);
final insertStmt =
await db.prepare('INSERT INTO frameworks(name) VALUES (?)');
await insertStmt.execute(['Flutter']);
await insertStmt.execute(['AngularDart']);
await insertStmt.close();
final selectStmt = await db.prepare('SELECT * FROM frameworks ORDER BY name');
final result = await selectStmt.select();
for (var row in result) {
print('${row['id']}: ${row['name']}');
}
await selectStmt.close();
await db.close();
}

View File

@ -1,6 +1,11 @@
/// Exports the raw []
/// Exports the low-level [Database] and [IsolateDb] classes to run operations
/// on a sqflite database.
library database;
import 'package:moor_ffi/src/bindings/types.dart';
import 'src/impl/isolate/isolate_db.dart';
export 'src/api/database.dart';
export 'src/api/result.dart';
export 'src/impl/database.dart' show SqliteException, Database;
export 'src/impl/isolate/isolate_db.dart';

View File

@ -1,4 +1,3 @@
import 'dart:ffi';
import 'dart:io';
import 'package:moor/backends.dart';
@ -6,4 +5,3 @@ import 'package:moor/moor.dart';
import 'package:moor_ffi/database.dart';
part 'src/vm_database.dart';
part 'src/load_library.dart';

View File

@ -0,0 +1,7 @@
/// Utils to open a [DynamicLibrary] on platforms that aren't supported by
/// `moor_ffi` by default.
library open_helper;
import 'dart:ffi';
export 'src/load_library.dart';

View File

@ -4,7 +4,7 @@
import 'dart:ffi';
import 'package:moor_ffi/moor_ffi.dart';
import 'package:moor_ffi/open_helper.dart';
import '../ffi/blob.dart';

View File

@ -0,0 +1,179 @@
import 'dart:async';
import 'dart:isolate';
import 'package:moor_ffi/database.dart';
import 'package:moor_ffi/src/impl/database.dart';
enum IsolateCommandType {
openDatabase,
closeDatabase,
executeSqlDirectly,
prepareStatement,
getUserVersion,
setUserVersion,
getUpdatedRows,
getLastInsertId,
preparedSelect,
preparedExecute,
preparedClose
}
class IsolateCommand {
final int requestId;
final IsolateCommandType type;
final dynamic data;
/// If this command operates on a prepared statement, contains the id of that
/// statement as sent by the background isolate.
int preparedStatementId;
IsolateCommand(this.requestId, this.type, this.data);
}
class IsolateResponse {
final int requestId;
final dynamic response;
final dynamic error;
IsolateResponse(this.requestId, this.response, this.error);
}
/// Communicates with a background isolate over an RPC-like api.
class DbOperationProxy {
/// Stream of messages received by the background isolate.
final StreamController<dynamic> backgroundMsgs;
final ReceivePort _receivePort;
final Map<int, Completer> _pendingRequests = {};
final SendPort send;
final Isolate isolate;
int _currentRequestId = 0;
DbOperationProxy(
this.backgroundMsgs, this._receivePort, this.send, this.isolate) {
backgroundMsgs.stream.listen(_handleResponse);
}
Future<dynamic> sendRequest(IsolateCommandType type, dynamic data,
{int preparedStmtId}) {
final id = _currentRequestId++;
final cmd = IsolateCommand(id, type, data)
..preparedStatementId = preparedStmtId;
final completer = Completer();
_pendingRequests[id] = completer;
send.send(cmd);
return completer.future;
}
void _handleResponse(dynamic response) {
if (response is IsolateResponse) {
final completer = _pendingRequests.remove(response.requestId);
if (response.error != null) {
completer.completeError(response.error);
} else {
completer.complete(response.response);
}
}
}
void close() {
_receivePort.close();
backgroundMsgs.close();
isolate.kill();
}
static Future<DbOperationProxy> spawn() async {
final foregroundReceive = ReceivePort();
final backgroundSend = foregroundReceive.sendPort;
final isolate = await Isolate.spawn(_entryPoint, backgroundSend,
debugName: 'moor_ffi background isolate');
final controller = StreamController.broadcast();
foregroundReceive.listen(controller.add);
final foregroundSend = await controller.stream
.firstWhere((msg) => msg is SendPort) as SendPort;
return DbOperationProxy(
controller, foregroundReceive, foregroundSend, isolate);
}
static void _entryPoint(SendPort backgroundSend) {
final backgroundReceive = ReceivePort();
final foregroundSend = backgroundReceive.sendPort;
// inform the main isolate about the created send port
backgroundSend.send(foregroundSend);
BackgroundIsolateRunner(backgroundReceive, backgroundSend).start();
}
}
class BackgroundIsolateRunner {
final ReceivePort receive;
final SendPort send;
Database db;
List<PreparedStatement> stmts = [];
BackgroundIsolateRunner(this.receive, this.send);
void start() {
receive.listen((data) {
if (data is IsolateCommand) {
try {
final response = _handleCommand(data);
send.send(IsolateResponse(data.requestId, response, null));
} catch (e) {
send.send(IsolateResponse(data.requestId, null, e));
}
}
});
}
dynamic _handleCommand(IsolateCommand cmd) {
switch (cmd.type) {
case IsolateCommandType.openDatabase:
assert(db == null);
db = Database.open(cmd.data as String);
break;
case IsolateCommandType.closeDatabase:
db?.close();
stmts.clear();
db = null;
break;
case IsolateCommandType.executeSqlDirectly:
db.execute(cmd.data as String);
break;
case IsolateCommandType.prepareStatement:
final stmt = db.prepare(cmd.data as String);
stmts.add(stmt);
return stmts.length - 1;
case IsolateCommandType.getUserVersion:
return db.userVersion();
case IsolateCommandType.setUserVersion:
final version = cmd.data as int;
db.setUserVersion(version);
break;
case IsolateCommandType.getUpdatedRows:
return db.getUpdatedRows();
case IsolateCommandType.getLastInsertId:
return db.getLastInsertId();
case IsolateCommandType.preparedSelect:
final stmt = stmts[cmd.preparedStatementId];
return stmt.select(cmd.data as List);
case IsolateCommandType.preparedExecute:
final stmt = stmts[cmd.preparedStatementId];
stmt.execute(cmd.data as List);
break;
case IsolateCommandType.preparedClose:
final index = cmd.preparedStatementId;
stmts[index].close();
stmts.removeAt(index);
break;
}
}
}

View File

@ -0,0 +1,105 @@
import 'dart:async';
import 'dart:io';
import 'package:moor_ffi/database.dart';
import 'package:moor_ffi/src/impl/database.dart';
import 'package:moor_ffi/src/impl/isolate/background.dart';
class IsolateDb implements BaseDatabase {
/// Spawns a background isolate and opens the [file] on that isolate. The file
/// will be created if it doesn't exist.
static Future<IsolateDb> openFile(File file) => open(file.absolute.path);
/// Opens a in-memory database on a background isolates.
///
/// If you're not using extensive queries, a synchronous [Database] will
/// provide better performance for in-memory databases!
static Future<IsolateDb> openMemory() => open(':memory:');
/// Spawns a background isolate and opens a sqlite3 database from its
/// filename.
static Future<IsolateDb> open(String path) async {
final proxy = await DbOperationProxy.spawn();
final isolate = IsolateDb._(proxy);
await isolate._open(path);
return isolate;
}
final DbOperationProxy _proxy;
IsolateDb._(this._proxy);
Future<int> _sendAndAssumeInt(IsolateCommandType type, [dynamic data]) async {
return await _proxy.sendRequest(type, data) as int;
}
Future<void> _open(String path) {
return _proxy.sendRequest(IsolateCommandType.openDatabase, path);
}
@override
Future<void> close() async {
await _proxy.sendRequest(IsolateCommandType.closeDatabase, null);
_proxy.close();
}
@override
Future<void> execute(String sql) async {
await _proxy.sendRequest(IsolateCommandType.executeSqlDirectly, sql);
}
@override
Future<int> getLastInsertId() async {
return _sendAndAssumeInt(IsolateCommandType.getLastInsertId);
}
@override
Future<int> getUpdatedRows() async {
return _sendAndAssumeInt(IsolateCommandType.getUpdatedRows);
}
@override
FutureOr<BasePreparedStatement> prepare(String sql) async {
final id =
await _sendAndAssumeInt(IsolateCommandType.prepareStatement, sql);
return IsolatePreparedStatement(this, id);
}
@override
Future<void> setUserVersion(int version) async {
await _proxy.sendRequest(IsolateCommandType.setUserVersion, version);
}
@override
Future<int> userVersion() async {
return _sendAndAssumeInt(IsolateCommandType.getUserVersion);
}
}
class IsolatePreparedStatement implements BasePreparedStatement {
final IsolateDb _db;
final int _id;
IsolatePreparedStatement(this._db, this._id);
@override
Future<void> close() async {
await _db._proxy.sendRequest(IsolateCommandType.preparedClose, null,
preparedStmtId: _id);
}
@override
Future<void> execute([List params]) async {
await _db._proxy.sendRequest(IsolateCommandType.preparedExecute, params,
preparedStmtId: _id);
}
@override
Future<Result> select([List params]) async {
final response = await _db._proxy.sendRequest(
IsolateCommandType.preparedSelect, params,
preparedStmtId: _id);
return response as Result;
}
}

View File

@ -1,4 +1,5 @@
part of 'package:moor_ffi/moor_ffi.dart';
import 'dart:ffi';
import 'dart:io';
/// Signature responsible for loading the dynamic sqlite3 library that moor will
/// use.

View File

@ -8,19 +8,16 @@ environment:
sdk: ">=2.5.0-dev <2.6.0"
dependencies:
moor: ^2.0.0
moor: ">=1.7.0 <2.1.0"
# flutter:
# sdk: flutter
dev_dependencies:
test: ^1.6.0
path: ^1.6.0
flutter:
plugin:
# the flutter.plugin key needs to exists so that this project gets recognized as a plugin when imported. We need to
# get recognized as a plugin so that our build scripts are executed.
foo: bar
dependency_overrides:
moor:
path: ../moor

View File

@ -0,0 +1,64 @@
import 'dart:async';
import 'dart:io';
import 'package:path/path.dart' as p;
import 'package:test/test.dart';
import 'package:moor_ffi/database.dart';
import 'suite/select.dart' as select;
import 'suite/user_version.dart' as user_version;
var _tempFileCounter = 0;
List<File> _createdFiles = [];
File temporaryFile() {
final count = _tempFileCounter++;
final path =
p.join(Directory.systemTemp.absolute.path, 'moor_ffi_test_$count.db');
final file = File(path);
_createdFiles.add(file);
return file;
}
abstract class TestedDatabase {
FutureOr<BaseDatabase> openFile(File file);
FutureOr<BaseDatabase> openMemory();
}
class TestRegularDatabase implements TestedDatabase {
@override
BaseDatabase openFile(File file) => Database.openFile(file);
@override
BaseDatabase openMemory() => Database.memory();
}
class TestIsolateDatabase implements TestedDatabase {
@override
Future<BaseDatabase> openFile(File file) => IsolateDb.openFile(file);
@override
FutureOr<BaseDatabase> openMemory() => IsolateDb.openMemory();
}
void main() {
group('regular database', () {
_declareAll(TestRegularDatabase());
});
group('isolate database', () {
_declareAll(TestIsolateDatabase());
});
tearDownAll(() async {
for (var file in _createdFiles) {
if (await file.exists()) {
await file.delete();
}
}
});
}
void _declareAll(TestedDatabase db) {
select.main(db);
user_version.main(db);
}

View File

@ -0,0 +1,21 @@
import 'package:test/test.dart';
import '../runners.dart';
void main(TestedDatabase db) {
test('select statements return expected value', () async {
final opened = await db.openMemory();
final prepared = await opened.prepare('SELECT ?');
final result1 = await prepared.select([1]);
expect(result1.columnNames, ['?']);
expect(result1.single.columnAt(0), 1);
final result2 = await prepared.select([2]);
expect(result2.columnNames, ['?']);
expect(result2.single.columnAt(0), 2);
await opened.close();
});
}

View File

@ -0,0 +1,24 @@
import 'package:test/test.dart';
import '../runners.dart';
void main(TestedDatabase db) {
test('can set the user version on a database', () async {
final file = temporaryFile();
final opened = await db.openFile(file);
var version = await opened.userVersion();
expect(version, 0);
await opened.setUserVersion(3);
version = await opened.userVersion();
expect(version, 3);
// ensure that the version is stored on file
await opened.close();
final another = await db.openFile(file);
expect(await another.userVersion(), 3);
await another.close();
});
}