diff --git a/moor_ffi/README.md b/moor_ffi/README.md index 1ffbb693..5dfb221b 100644 --- a/moor_ffi/README.md +++ b/moor_ffi/README.md @@ -1,38 +1,102 @@ # moor_ffi -Moor backend that uses `dart:ffi`. Note that, while we have integration tests -on this package, it depends on the `dart:ffi` apis, which are in "preview" status at the moment. -Thus, this library is not suited for production use. +Experimental bindings to sqlite by using `dart:ffi`. This library contains utils to make +integration with [moor](https://pub.dev/packages/moor) easier, but it can also be used +as a standalone package. -If you want to use moor on Android or iOS, see the [getting started guide](https://moor.simonbinder.eu/docs/getting-started/) -which recommends to use the [moor_flutter](https://pub.dev/packages/moor_flutter) package. -At the moment, this library is targeted at advanced moor users who want to try out the `ffi` -backend. +## Warnings +At the moment, `dart:ffi` is in preview and there will be breaking changes that this +library has to adapt to. This library has been tested on Dart `2.5.0`. + +If you're using a development Dart version (this includes Flutter channels that are not +`stable`), this library might not work. + +If you just want to use moor, using the [moor_flutter](https://pub.dev/packages/moor_flutter) +package is the better option at the moment. ## Supported platforms -At the moment, this plugin only supports Android without further work. However, it's also going -to run on all platforms that expose `sqlite3` as a shared native library (macOS and virtually -all Linux distros, I'm not sure about Windows). Native iOS and macOS support is planned. -As Flutter desktop doesn't support plugins on Windows and Linux yet, we can't bundle the -sqlite library on those platforms. +You can make this library work on any platform that let's you obtain a `DynamicLibrary` +from which moor_ffi loads the functions (see below). + +Out of the box, this libraries supports all platforms where `sqlite3` is installed: +- iOS: Yes +- macOS: Yes +- Linux: Available on most distros +- Windows: When the user has installed sqlite (they probably have) +- Android: Yes when used with Flutter + +This library works with and without Flutter. +If you're using Flutter, this library will bundle `sqlite3` in your Android app. This +requires the Android NDK to be installed (You can get the NDK in the [SDK Manager](https://developer.android.com/studio/intro/update.html#sdk-manager) +of Android Studio). Note that the first `flutter run` is going to take a very long time as +we need to compile sqlite. + +### On other platforms +Using this library on platforms that are not supported out of the box is fairly +straightforward. For instance, if you release your own `sqlite3.so` with your application, +you could use +```dart +import 'dart:ffi'; +import 'dart:io'; +import 'package:moor_ffi/database.dart'; +import 'package:moor_ffi/open_helper.dart'; + +void main() { + open.overrideFor(OperatingSystem.linux, _openOnLinux); + + final db = Database.memory(); + db.close(); +} + +DynamicLibrary _openOnLinux() { + final script = File(Platform.script.toFilePath()); + final libraryNextToScript = File('${script.path}/sqlite3.so'); + return DynamicLibrary.open(libraryNextToScript.path); +} +``` +Just be sure to first override the behavior and then opening the database. Further, +if you want to use the isolate api, you can only use a static method or top-level +function to open the library. + +### Supported datatypes +This library supports `null`, `int`, other `num`s (converted to double), +`String` and `Uint8List` to bind args. Returned columns from select statements +will have the same types. + +## Using without moor +```dart +import 'package:moor_ffi/database.dart'; + +void main() { + final database = Database.memory(); + // run some database operations. See the example for details + database.close(); +} +``` + +You can also use an asynchronous API on a background isolate by using `IsolateDb.openFile` +or `IsolateDb.openMemory`, respectively. be aware that the asynchronous API is much slower, +but it moves work out of the UI isolate. + +Be sure to __always__ call `Database.close` to avoid memory leaks! ## Migrating from moor_flutter +__Note__: For production apps, please use `moor_flutter` until this package +reaches a stable version. + Add both `moor` and `moor_ffi` to your pubspec, the `moor_flutter` dependency can be dropped. ```yaml dependencies: - moor: ^2.0.0 + moor: ^1.7.0 moor_ffi: ^0.0.1 dev_dependencies: - moor: ^2.0.0 + moor_generator: ^1.7.0 ``` -In your main database file, replace the `package:moor_flutter/moor_flutter.dart` import with -`package:moor/moor.dart` and `package:moor_ffi/moor_ffi.dart`. +In the file where you created a `FlutterQueryExecutor`, replace the `moor_flutter` import +with both `package:moor/moor.dart` and `package:moor_ffi/moor_ffi.dart`. + In all other project files that use moor apis (e.g. a `Value` class for companions), just import `package:moor/moor.dart`. -Finally, replace usages of `FlutterQueryExecutor` with `VmDatabase`. - -## Notes -After importing this library, the first Flutter build is going to take a very long time. The reason is that we're -compiling sqlite to bundle it with your app. Subsequent builds should take an acceptable time to execute. +Finally, replace usages of `FlutterQueryExecutor` with `VmDatabase`. \ No newline at end of file diff --git a/moor_ffi/example/main.dart b/moor_ffi/example/main.dart new file mode 100644 index 00000000..afbd1708 --- /dev/null +++ b/moor_ffi/example/main.dart @@ -0,0 +1,27 @@ +import 'package:moor_ffi/database.dart'; + +const _createTable = r''' +CREATE TABLE frameworks ( + id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, + name VARCHAR NOT NULL +); +'''; + +void main() { + final db = Database.memory(); + db.execute(_createTable); + + final insertStmt = db.prepare('INSERT INTO frameworks(name) VALUES (?)'); + insertStmt.execute(['Flutter']); + insertStmt.execute(['AngularDart']); + insertStmt.close(); + + final selectStmt = db.prepare('SELECT * FROM frameworks ORDER BY name'); + final result = selectStmt.select(); + for (var row in result) { + print('${row['id']}: ${row['name']}'); + } + + selectStmt.close(); + db.close(); +} diff --git a/moor_ffi/example/main_async.dart b/moor_ffi/example/main_async.dart new file mode 100644 index 00000000..9f21e7aa --- /dev/null +++ b/moor_ffi/example/main_async.dart @@ -0,0 +1,28 @@ +import 'package:moor_ffi/database.dart'; + +const _createTable = r''' +CREATE TABLE frameworks ( + id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, + name VARCHAR NOT NULL +); +'''; + +void main() async { + final db = await IsolateDb.openMemory(); + await db.execute(_createTable); + + final insertStmt = + await db.prepare('INSERT INTO frameworks(name) VALUES (?)'); + await insertStmt.execute(['Flutter']); + await insertStmt.execute(['AngularDart']); + await insertStmt.close(); + + final selectStmt = await db.prepare('SELECT * FROM frameworks ORDER BY name'); + final result = await selectStmt.select(); + for (var row in result) { + print('${row['id']}: ${row['name']}'); + } + + await selectStmt.close(); + await db.close(); +} diff --git a/moor_ffi/lib/database.dart b/moor_ffi/lib/database.dart index 9b6157eb..319b150b 100644 --- a/moor_ffi/lib/database.dart +++ b/moor_ffi/lib/database.dart @@ -1,6 +1,11 @@ -/// Exports the raw [] +/// Exports the low-level [Database] and [IsolateDb] classes to run operations +/// on a sqflite database. library database; +import 'package:moor_ffi/src/bindings/types.dart'; +import 'src/impl/isolate/isolate_db.dart'; + export 'src/api/database.dart'; export 'src/api/result.dart'; export 'src/impl/database.dart' show SqliteException, Database; +export 'src/impl/isolate/isolate_db.dart'; diff --git a/moor_ffi/lib/moor_ffi.dart b/moor_ffi/lib/moor_ffi.dart index 7954e761..af2469b7 100644 --- a/moor_ffi/lib/moor_ffi.dart +++ b/moor_ffi/lib/moor_ffi.dart @@ -1,4 +1,3 @@ -import 'dart:ffi'; import 'dart:io'; import 'package:moor/backends.dart'; @@ -6,4 +5,3 @@ import 'package:moor/moor.dart'; import 'package:moor_ffi/database.dart'; part 'src/vm_database.dart'; -part 'src/load_library.dart'; diff --git a/moor_ffi/lib/open_helper.dart b/moor_ffi/lib/open_helper.dart new file mode 100644 index 00000000..ed421162 --- /dev/null +++ b/moor_ffi/lib/open_helper.dart @@ -0,0 +1,7 @@ +/// Utils to open a [DynamicLibrary] on platforms that aren't supported by +/// `moor_ffi` by default. +library open_helper; + +import 'dart:ffi'; + +export 'src/load_library.dart'; diff --git a/moor_ffi/lib/src/bindings/bindings.dart b/moor_ffi/lib/src/bindings/bindings.dart index 4b397358..4bc66081 100644 --- a/moor_ffi/lib/src/bindings/bindings.dart +++ b/moor_ffi/lib/src/bindings/bindings.dart @@ -4,7 +4,7 @@ import 'dart:ffi'; -import 'package:moor_ffi/moor_ffi.dart'; +import 'package:moor_ffi/open_helper.dart'; import '../ffi/blob.dart'; diff --git a/moor_ffi/lib/src/impl/isolate/background.dart b/moor_ffi/lib/src/impl/isolate/background.dart new file mode 100644 index 00000000..7f77b342 --- /dev/null +++ b/moor_ffi/lib/src/impl/isolate/background.dart @@ -0,0 +1,179 @@ +import 'dart:async'; +import 'dart:isolate'; + +import 'package:moor_ffi/database.dart'; +import 'package:moor_ffi/src/impl/database.dart'; + +enum IsolateCommandType { + openDatabase, + closeDatabase, + executeSqlDirectly, + prepareStatement, + getUserVersion, + setUserVersion, + getUpdatedRows, + getLastInsertId, + preparedSelect, + preparedExecute, + preparedClose +} + +class IsolateCommand { + final int requestId; + final IsolateCommandType type; + final dynamic data; + + /// If this command operates on a prepared statement, contains the id of that + /// statement as sent by the background isolate. + int preparedStatementId; + + IsolateCommand(this.requestId, this.type, this.data); +} + +class IsolateResponse { + final int requestId; + final dynamic response; + final dynamic error; + + IsolateResponse(this.requestId, this.response, this.error); +} + +/// Communicates with a background isolate over an RPC-like api. +class DbOperationProxy { + /// Stream of messages received by the background isolate. + final StreamController backgroundMsgs; + final ReceivePort _receivePort; + final Map _pendingRequests = {}; + + final SendPort send; + final Isolate isolate; + + int _currentRequestId = 0; + + DbOperationProxy( + this.backgroundMsgs, this._receivePort, this.send, this.isolate) { + backgroundMsgs.stream.listen(_handleResponse); + } + + Future sendRequest(IsolateCommandType type, dynamic data, + {int preparedStmtId}) { + final id = _currentRequestId++; + final cmd = IsolateCommand(id, type, data) + ..preparedStatementId = preparedStmtId; + final completer = Completer(); + _pendingRequests[id] = completer; + + send.send(cmd); + + return completer.future; + } + + void _handleResponse(dynamic response) { + if (response is IsolateResponse) { + final completer = _pendingRequests.remove(response.requestId); + if (response.error != null) { + completer.completeError(response.error); + } else { + completer.complete(response.response); + } + } + } + + void close() { + _receivePort.close(); + backgroundMsgs.close(); + isolate.kill(); + } + + static Future spawn() async { + final foregroundReceive = ReceivePort(); + final backgroundSend = foregroundReceive.sendPort; + final isolate = await Isolate.spawn(_entryPoint, backgroundSend, + debugName: 'moor_ffi background isolate'); + + final controller = StreamController.broadcast(); + foregroundReceive.listen(controller.add); + + final foregroundSend = await controller.stream + .firstWhere((msg) => msg is SendPort) as SendPort; + + return DbOperationProxy( + controller, foregroundReceive, foregroundSend, isolate); + } + + static void _entryPoint(SendPort backgroundSend) { + final backgroundReceive = ReceivePort(); + final foregroundSend = backgroundReceive.sendPort; + + // inform the main isolate about the created send port + backgroundSend.send(foregroundSend); + + BackgroundIsolateRunner(backgroundReceive, backgroundSend).start(); + } +} + +class BackgroundIsolateRunner { + final ReceivePort receive; + final SendPort send; + + Database db; + List stmts = []; + + BackgroundIsolateRunner(this.receive, this.send); + + void start() { + receive.listen((data) { + if (data is IsolateCommand) { + try { + final response = _handleCommand(data); + send.send(IsolateResponse(data.requestId, response, null)); + } catch (e) { + send.send(IsolateResponse(data.requestId, null, e)); + } + } + }); + } + + dynamic _handleCommand(IsolateCommand cmd) { + switch (cmd.type) { + case IsolateCommandType.openDatabase: + assert(db == null); + db = Database.open(cmd.data as String); + break; + case IsolateCommandType.closeDatabase: + db?.close(); + stmts.clear(); + db = null; + break; + case IsolateCommandType.executeSqlDirectly: + db.execute(cmd.data as String); + break; + case IsolateCommandType.prepareStatement: + final stmt = db.prepare(cmd.data as String); + stmts.add(stmt); + return stmts.length - 1; + case IsolateCommandType.getUserVersion: + return db.userVersion(); + case IsolateCommandType.setUserVersion: + final version = cmd.data as int; + db.setUserVersion(version); + break; + case IsolateCommandType.getUpdatedRows: + return db.getUpdatedRows(); + case IsolateCommandType.getLastInsertId: + return db.getLastInsertId(); + case IsolateCommandType.preparedSelect: + final stmt = stmts[cmd.preparedStatementId]; + return stmt.select(cmd.data as List); + case IsolateCommandType.preparedExecute: + final stmt = stmts[cmd.preparedStatementId]; + stmt.execute(cmd.data as List); + break; + case IsolateCommandType.preparedClose: + final index = cmd.preparedStatementId; + stmts[index].close(); + stmts.removeAt(index); + break; + } + } +} diff --git a/moor_ffi/lib/src/impl/isolate/isolate_db.dart b/moor_ffi/lib/src/impl/isolate/isolate_db.dart new file mode 100644 index 00000000..543fa56b --- /dev/null +++ b/moor_ffi/lib/src/impl/isolate/isolate_db.dart @@ -0,0 +1,105 @@ +import 'dart:async'; +import 'dart:io'; + +import 'package:moor_ffi/database.dart'; +import 'package:moor_ffi/src/impl/database.dart'; +import 'package:moor_ffi/src/impl/isolate/background.dart'; + +class IsolateDb implements BaseDatabase { + /// Spawns a background isolate and opens the [file] on that isolate. The file + /// will be created if it doesn't exist. + static Future openFile(File file) => open(file.absolute.path); + + /// Opens a in-memory database on a background isolates. + /// + /// If you're not using extensive queries, a synchronous [Database] will + /// provide better performance for in-memory databases! + static Future openMemory() => open(':memory:'); + + /// Spawns a background isolate and opens a sqlite3 database from its + /// filename. + static Future open(String path) async { + final proxy = await DbOperationProxy.spawn(); + + final isolate = IsolateDb._(proxy); + await isolate._open(path); + + return isolate; + } + + final DbOperationProxy _proxy; + IsolateDb._(this._proxy); + + Future _sendAndAssumeInt(IsolateCommandType type, [dynamic data]) async { + return await _proxy.sendRequest(type, data) as int; + } + + Future _open(String path) { + return _proxy.sendRequest(IsolateCommandType.openDatabase, path); + } + + @override + Future close() async { + await _proxy.sendRequest(IsolateCommandType.closeDatabase, null); + _proxy.close(); + } + + @override + Future execute(String sql) async { + await _proxy.sendRequest(IsolateCommandType.executeSqlDirectly, sql); + } + + @override + Future getLastInsertId() async { + return _sendAndAssumeInt(IsolateCommandType.getLastInsertId); + } + + @override + Future getUpdatedRows() async { + return _sendAndAssumeInt(IsolateCommandType.getUpdatedRows); + } + + @override + FutureOr prepare(String sql) async { + final id = + await _sendAndAssumeInt(IsolateCommandType.prepareStatement, sql); + return IsolatePreparedStatement(this, id); + } + + @override + Future setUserVersion(int version) async { + await _proxy.sendRequest(IsolateCommandType.setUserVersion, version); + } + + @override + Future userVersion() async { + return _sendAndAssumeInt(IsolateCommandType.getUserVersion); + } +} + +class IsolatePreparedStatement implements BasePreparedStatement { + final IsolateDb _db; + final int _id; + + IsolatePreparedStatement(this._db, this._id); + + @override + Future close() async { + await _db._proxy.sendRequest(IsolateCommandType.preparedClose, null, + preparedStmtId: _id); + } + + @override + Future execute([List params]) async { + await _db._proxy.sendRequest(IsolateCommandType.preparedExecute, params, + preparedStmtId: _id); + } + + @override + Future select([List params]) async { + final response = await _db._proxy.sendRequest( + IsolateCommandType.preparedSelect, params, + preparedStmtId: _id); + return response as Result; + } +} diff --git a/moor_ffi/lib/src/load_library.dart b/moor_ffi/lib/src/load_library.dart index db3108e0..433526fc 100644 --- a/moor_ffi/lib/src/load_library.dart +++ b/moor_ffi/lib/src/load_library.dart @@ -1,4 +1,5 @@ -part of 'package:moor_ffi/moor_ffi.dart'; +import 'dart:ffi'; +import 'dart:io'; /// Signature responsible for loading the dynamic sqlite3 library that moor will /// use. diff --git a/moor_ffi/pubspec.yaml b/moor_ffi/pubspec.yaml index 26ba7882..8536dcf0 100644 --- a/moor_ffi/pubspec.yaml +++ b/moor_ffi/pubspec.yaml @@ -8,19 +8,16 @@ environment: sdk: ">=2.5.0-dev <2.6.0" dependencies: - moor: ^2.0.0 + moor: ">=1.7.0 <2.1.0" # flutter: # sdk: flutter dev_dependencies: test: ^1.6.0 + path: ^1.6.0 flutter: plugin: # the flutter.plugin key needs to exists so that this project gets recognized as a plugin when imported. We need to # get recognized as a plugin so that our build scripts are executed. - foo: bar - -dependency_overrides: - moor: - path: ../moor \ No newline at end of file + foo: bar \ No newline at end of file diff --git a/moor_ffi/test/runners.dart b/moor_ffi/test/runners.dart new file mode 100644 index 00000000..279df371 --- /dev/null +++ b/moor_ffi/test/runners.dart @@ -0,0 +1,64 @@ +import 'dart:async'; +import 'dart:io'; + +import 'package:path/path.dart' as p; +import 'package:test/test.dart'; +import 'package:moor_ffi/database.dart'; + +import 'suite/select.dart' as select; +import 'suite/user_version.dart' as user_version; + +var _tempFileCounter = 0; +List _createdFiles = []; +File temporaryFile() { + final count = _tempFileCounter++; + final path = + p.join(Directory.systemTemp.absolute.path, 'moor_ffi_test_$count.db'); + final file = File(path); + _createdFiles.add(file); + return file; +} + +abstract class TestedDatabase { + FutureOr openFile(File file); + FutureOr openMemory(); +} + +class TestRegularDatabase implements TestedDatabase { + @override + BaseDatabase openFile(File file) => Database.openFile(file); + + @override + BaseDatabase openMemory() => Database.memory(); +} + +class TestIsolateDatabase implements TestedDatabase { + @override + Future openFile(File file) => IsolateDb.openFile(file); + + @override + FutureOr openMemory() => IsolateDb.openMemory(); +} + +void main() { + group('regular database', () { + _declareAll(TestRegularDatabase()); + }); + + group('isolate database', () { + _declareAll(TestIsolateDatabase()); + }); + + tearDownAll(() async { + for (var file in _createdFiles) { + if (await file.exists()) { + await file.delete(); + } + } + }); +} + +void _declareAll(TestedDatabase db) { + select.main(db); + user_version.main(db); +} diff --git a/moor_ffi/test/suite/select.dart b/moor_ffi/test/suite/select.dart new file mode 100644 index 00000000..a4719c09 --- /dev/null +++ b/moor_ffi/test/suite/select.dart @@ -0,0 +1,21 @@ +import 'package:test/test.dart'; + +import '../runners.dart'; + +void main(TestedDatabase db) { + test('select statements return expected value', () async { + final opened = await db.openMemory(); + + final prepared = await opened.prepare('SELECT ?'); + + final result1 = await prepared.select([1]); + expect(result1.columnNames, ['?']); + expect(result1.single.columnAt(0), 1); + + final result2 = await prepared.select([2]); + expect(result2.columnNames, ['?']); + expect(result2.single.columnAt(0), 2); + + await opened.close(); + }); +} diff --git a/moor_ffi/test/suite/user_version.dart b/moor_ffi/test/suite/user_version.dart new file mode 100644 index 00000000..cf07277c --- /dev/null +++ b/moor_ffi/test/suite/user_version.dart @@ -0,0 +1,24 @@ +import 'package:test/test.dart'; + +import '../runners.dart'; + +void main(TestedDatabase db) { + test('can set the user version on a database', () async { + final file = temporaryFile(); + final opened = await db.openFile(file); + + var version = await opened.userVersion(); + expect(version, 0); + + await opened.setUserVersion(3); + version = await opened.userVersion(); + expect(version, 3); + + // ensure that the version is stored on file + await opened.close(); + + final another = await db.openFile(file); + expect(await another.userVersion(), 3); + await another.close(); + }); +}