Better types for nested expressions, more 2.0 docs

docs/README.md

@@ -4,7 +4,7 @@ We use [Docsy](https://github.com/google/docsy), a Hugo theme for this website.
 [here](https://www.docsy.dev/docs/getting-started/).
 
 To work on the documentation, first cd into this directory, then run `git submodule update --init --recursive` and
- `npm install`.
+ `npm install`. To update the dependencies, run `git pull --recurse-submodules`-
 
 ## Running the website locally
 After the setup, it's just a simple

docs/assets/scss/_variables_project.scss

@@ -3,4 +3,6 @@ $enable-gradients: false;
 $enable-rounded: false;
 $enable-shadows: false;
 
-$secondary: #4CAF50;
+$secondary: #4CAF50;
+
+$code-color: #dc3545;

docs/content/en/_index.html

@@ -32,17 +32,19 @@ that allows you run fluent queries on your data.
 [Get started now]({{< ref "/docs/Getting started/_index.md" >}})
 {{% /blocks/feature %}}
 
-{{% blocks/feature icon="fas fa-database" title="Prefer SQL? Moor got you covered!" url="https://moor.simonbinder.eu/queries/custom" %}}
-Moor contains a powerful sql parser and analyzer, allowing it to create typesafe APIs for all your sql queries. All queries are
+{{% blocks/feature icon="fas fa-database" title="Prefer SQL? Moor got you covered!" %}}
+Moor contains a powerful sql parser and analyzer, allowing it to create typesafe APIs for all your sql queries. All sql queries are
 validated and analyzed during build-time, so moor can provide hints about potential errors quickly and generate efficient mapping
 code.
+
+[Learn more]({{< ref "/docs/Using SQL/_index.md" >}})
 {{% /blocks/feature %}}
 
 
 {{% blocks/feature icon="fas fa-star" title="And much more!" %}}
-Moor can also provide auto-updating streams emitting new results when the underlying data changes.
-Moor makes dealing with transactions easy (no special parameter to pass around everywhere), lets
-your write modular database code with DAOs and much more.
+Moor provides auto-updating `Streams` for all your queries, makes dealing with transactions and migrations easy
+and lets your write modular database code with DAOs. We even have a [sql IDE](too) builtin to the project
+When using moor, working with databases in Dart is fun!
 
 {{% /blocks/feature %}}
 {{< /blocks/section >}}

docs/content/en/docs/Other engines/vm.md

@@ -1,12 +1,78 @@
 ---
-title: Dart VM
-description: An upcoming version will have a version for the Dart VM
+title: Dart VM (using ffi)
+description: Experimental version of moor using `dart:ffi`
 ---
 
-An upcoming version of moor will have first class support for the Dart VM, 
-so you can use moor on Desktop Flutter applications or Dart apps.
+## Warnings and supported platforms
 
-We're going to use the `dart:ffi` feature for that, which itself is an
-experimental state at the moment. We already have a version of moor that
-runs on the Dart VM (see [#76](https://github.com/simolus3/moor/issues/76))
-and we're going to release it when `dart:ffi` becomes stable.
+Please note that `dart:ffi` is in "preview" at the moment and that there will be breaking
+changes. Using the `moor_ffi` package on non-stable Dart or Flutter versions can break.
+Also, please don't use the package for production apps yet.
+
+At the moment, `moor_ffi` supports iOS, macOS and Android out of the box. Most Linux
+Distros have sqlite available as a shared library, those are supported as well. 
+
+If you're shipping apps for Windows and Linux, it is recommended that you bundle a
+`sqlite3.so` and `sqlite3.dll` file with your app. You can then make `moor_ffi`
+support your setup by running this code before opening the database:
+
+```dart
+import 'dart:ffi';
+import 'dart:io';
+import 'package:moor_ffi/database.dart';
+import 'package:moor_ffi/open_helper.dart';
+
+void main() {
+  open.overrideFor(OperatingSystem.linux, _openOnLinux);
+
+  final db = Database.memory();
+  db.close();
+}
+
+DynamicLibrary _openOnLinux() {
+  final script = File(Platform.script.toFilePath());
+  final libraryNextToScript = File('${script.path}/sqlite3.so');
+  return DynamicLibrary.open(libraryNextToScript.path);
+}
+// _openOnWindows could be implemented similarly by opening `sqlite3.dll`
+
+```
+
+## Migrating from moor_flutter to moor_ffi
+
+__Again__: If you're only looking for Android and iOS support, `moor_flutter` is the
+right library to use. 
+
+1. Adapt your `pubspec.yaml`: You can remove the `moor_flutter` dependency and instead
+   add both the `moor` and `moor_ffi` dependencies:
+   ```yaml
+   dependencies:
+     moor: ^2.0.0
+     moor_ffi: ^0.0.1
+   dev_dependencies:
+     moor_generator: ^2.0.0
+   ```
+   Note: If you were using `FlutterQueryExecutor.inDatabasesFolder`, you should also depend
+   on `path_provider`. For desktop support of that library, see [this readme](https://github.com/google/flutter-desktop-embedding/tree/master/plugins/flutter_plugins).
+2. Adapt your imports:
+  - In the file where you created a `FlutterQueryExecutor`, replace the `moor_flutter` import
+    with `package:moor_ffi/moor_ffi.dart`.
+  - In all other files where you might have import `moor_flutter`, just import `package:moor/moor.dart`.
+3. Replace the executor. This code:
+   ```dart
+   FlutterQueryExecutor.inDatabaseFolder(path: 'db.sqlite')
+   ```
+   can now be written as
+   ```dart
+   import 'package:path_provider/path_provider.dart';
+   import 'package:path/path.dart' as p;
+
+   LazyDatabase(() async {
+       final dbFolder = await getApplicationDocumentsDirectory();
+       final file = File(j.join(dbFolder.path, 'db.sqlite'));
+       return VmDatabase(file);
+   })
+   ```
+   __Important warning__: `FlutterQueryExecutor.inDatabaseFolder` may use a different folder on Android, which
+   can cause data loss. This documentation will provide a better migration guide once `moor_ffi` is stable.
+   Please create an issue if you need guidance on this soon.

docs/content/en/docs/Using SQL/moor_files.md

@@ -10,7 +10,8 @@ aliases:
 Moor files are a new feature that lets you write all your database code in SQL - moor will generate typesafe APIs for them.
 
 ## Getting started
-To use this feature, lets create two files: `database.dart` and `tables.moor`. The Dart file is pretty straightforward:
+To use this feature, lets create two files: `database.dart` and `tables.moor`. The Dart file only contains the minimum code
+to setup the database:
 ```dart
 import 'package:moor/moor.dart';
 
@@ -47,13 +48,14 @@ deleteById: DELETE FROM todos WHERE id = :id;
 watchAllTodos: SELECT * FROM todos;
 ```
 
-After running the build runner, moor will write the `database.g.dart`
+After running the build runner with `flutter pub run build_runner build`,
+moor will write the `database.g.dart`
 file which contains the `_$MoorDb` superclass. Let's take a look at
 what we got:
 
 - Generated data classes (`Todo` and `Category`), and companion versions
   for inserts (see [Dart Interop](#dart-interop) for info). By default,
-  we strip a trailing "s" from the table name for the class. That's why 
+  moor strips a trailing "s" from the table name for the class. That's why 
   we used `AS Category` on the second table - it would have been called
   `Categorie` otherwise.
 - Methods to run the queries:
@@ -111,11 +113,13 @@ All tables reachable from the other file will then also be visible in
 the current file and to the database that `includes` it. Importing
 Dart files into a moor file will also work - then, all the tables
 declared via Dart tables can be used inside queries.
+We support both relative imports and the `package:` imports you
+know from Dart.
 
 ## Dart interop
 Moor files work perfectly together with moor's existing Dart API:
 
-- you can write Dart queries for moor files:
+- you can write Dart queries for tables declared in a moor file:
 ```dart
 Future<void> insert(TodosCompanion companion) async {
       await into(todos).insert(companion);
@@ -126,8 +130,10 @@ Future<void> insert(TodosCompanion companion) async {
 - generated methods for queries can be used in transactions, they work 
   together with auto-updating queries, etc.
 
-You can make most of both SQL and Dart by "Dart Templates", which is a
-Dart expression that gets inlined to a query. To use them, declare a 
+### Dart components in SQL
+
+You can make most of both SQL and Dart with "Dart Templates", which is a
+Dart expression that gets inlined to a query at runtime. To use them, declare a 
 $-variable in a query:
 ```sql
 _filterTodos: SELECT * FROM todos WHERE $predicate;
@@ -139,10 +145,25 @@ Stream<List<Todo>> watchInCategory(int category) {
     return _filterTodos(todos.category.equals(category)).watch();
 }
 ```
-This feature works for
+This lets you write a single SQL query and dynamically apply a predicate at runtime!
+This feature also works for
 
 - expressions
 - single ordering terms: `SELECT * FROM todos ORDER BY $term, id ASC`
   will generate a method taking an `OrderingTerm`.
 - whole order-by clauses: `SELECT * FROM todos ORDER BY $order`
-- limit clauses: `SELECT * FROM todos LIMIT $limit`
+- limit clauses: `SELECT * FROM todos LIMIT $limit`
+
+## Supported statements
+At the moment, the following statements can appear in a `.moor` file.
+
+- `import 'other.moor'`: Import all tables and queries declared in the other file
+   into the current file.
+- DDL statements (`CREATE TABLE`): Declares a table. We don't currently support indices and views,
+   [#162](https://github.com/simolus3/moor/issues/162) tracks support for that.
+- Query statements: We support `INSERT`, `SELECT`, `UPDATE` and `DELETE` statements.
+
+All imports must come before DDL statements, and those must come before the named queries.
+
+If you need support for another statement, or if moor rejects a query you think is valid, please
+create an issue!

docs/content/en/docs/Using SQL/sql_ide.md

@@ -0,0 +1,39 @@
+---
+title: "Experimental IDE"
+weight: 5
+description: Get real-time feedback as you type sql
+---
+
+Moor ships with an experimental analyzer plugin that provides real-time feedback on errors,
+hints, folding and outline.
+
+## Using with VS Code
+
+Make sure that your project depends on moor 2.0 or later. Then
+
+1. In the preferences, make sure that the `dart.analyzeAngularTemplates` option is
+   set to true.
+2. Tell Dart Code to analyze moor files as well. Add this to your `settings.json`:
+   ```json
+   "dart.additionalAnalyzerFileExtensions": [
+        "moor"
+    ]
+    ```
+3. Enable the plugin in Dart: Create a file called `analysis_options.yaml` in your project root,
+   next to your pubspec. It should contain this section:
+   ```yaml
+   analyzer:
+     plugins:
+       - moor
+   ```
+4. Finally, close and reopen your IDE so that the analysis server is restarted. The analysis server will
+   then load the moor plugin and start providing analysis results for `.moor` files. Loading the plugin
+   can take some time (around a minute for the first time).
+
+## Other IDEs
+
+Unfortunately, we can't support IntelliJ and Android Studio yet. Please vote on
+[this issue](https://youtrack.jetbrains.com/issue/WEB-41424) to help us here!
+
+If you're looking for support for an other IDE that uses the Dart analysis server,
+please create an issue. We can very probably make that happen.

docs/content/en/v2/index.md

@@ -0,0 +1,101 @@
+---
+title: Moor v2
+layout: home
+---
+
+{{< blocks/cover title="Moor 2.0: Supercharged SQL for Dart" image_anchor="top" height="min" color="indigo" >}}
+<div class="mx-auto">
+    <p class="lead mt-5">
+        Learn everything about Dart-SQL interop, the SQL IDE, experimental ffi support and everything new in moor
+    </p>
+</div>
+{{< /blocks/cover >}}
+
+{{% blocks/lead color="blue" %}}
+## Generator overhaul
+
+The rewritten compiler is faster than ever, supports more SQL features and gives you
+more flexibility when writing database code.
+
+[Check the updated documentation]({{< ref "../docs/Using SQL/moor_files.md" >}})
+{{% /blocks/lead %}}
+
+{{< blocks/section color="light" >}}
+{{% blocks/feature icon="fas fa-puzzle-piece" title="Improved type inference" %}}
+The new type inference engine provides more accurate results on complex expressions like window
+functions. We also generate simpler methods for queries that only return one column.
+{{% /blocks/feature %}}
+{{% blocks/feature icon="fas fa-database" title="Parser improvements" %}}
+We now support more advanced features like compound select statements and window functions,
+including detailed static analysis and lints.
+{{% /blocks/feature %}}
+{{% blocks/feature icon="fas fa-code-branch" title="Dart-SQL interop" %}}
+Declare tables in Dart, write your queries in SQL. Or do it the other way around. Or do it all in Dart.
+Or all in SQL. Moor makes writing database code fun without taking control over your code. 
+For maximum flexibilty, moor lets you inline Dart expressions into SQL and use the best of both
+worlds.
+{{% /blocks/feature %}}
+
+{{< /blocks/section >}}
+
+{{% blocks/lead color="green" %}}
+## Builtin SQL IDE
+
+Moor 2.0 expands the previous sql parser and analyzer, providing real-time feedback on your
+SQL queries as you type. Moor plugs right into the Dart analysis server, so you don't have
+to install any additional extensions.
+
+[Learn more about the IDE]({{< ref "../docs/Using SQL/sql_ide.md" >}})
+{{% /blocks/lead %}}
+
+{{< blocks/section color="dark" >}}
+{{% blocks/feature icon="fa-lightbulb" title="Quickfixes" %}}
+![](quickfix.png)
+
+Moor lets you write query code faster with helpful actions.
+{{% /blocks/feature %}}
+{{% blocks/feature icon="fas fa-brain" title="Smart warnings" %}}
+![](warning.png)
+
+Moor analyzes statements as you write them and reports errors right away. 
+This helps you identify problems fast, without having to open your app.
+{{% /blocks/feature %}}
+{{% blocks/feature icon="fas fa-info-circle" title="Structure view" %}}
+![](outline.png)
+
+Moor provides an outline of your tables and queries for a better overview.
+{{% /blocks/feature %}}
+
+{{< /blocks/section >}}
+
+{{% blocks/lead color="purple" %}}
+## And much, much more
+
+Moor 2.0 contains a set of optimizations and makes common tasks simpler
+{{% /blocks/lead %}}
+
+{{< blocks/section color="light" >}}
+{{% blocks/feature icon="fa-lightbulb" title="New database helpers" %}}
+New utils to load database from assets or to perform additional work before creating a database.
+{{% /blocks/feature %}}
+{{% blocks/feature icon="fas fa-exclamation" title="Removed deprecated features" %}}
+We removed a whole bunch of deprecated apis that made it harder to develop new features.
+
+[Read the changelog for details](https://pub.dev/packages/moor#-changelog-tab-)
+{{% /blocks/feature %}}
+{{% blocks/feature icon="fas fa-bolt" title="Experimental `dart:ffi` bindings" %}}
+The new [moor_ffi](https://pub.dev/packages/moor_ffi) package brings moor to the desktop and is up to 500x faster than the old
+implementation.
+
+_Please not that the package is still in preview_
+{{% /blocks/feature %}}
+
+{{< /blocks/section >}}
+
+{{< blocks/section color="dark" type="section" >}}
+## Try moor now
+
+- To get started with moor, follow our [getting started guide](ref "../docs/Getting started/_index.md") here.
+- To get started with SQL in moor, or to migrate an project to moor, follow our __TODO: Write migration guide__
+
+{{< /blocks/section >}}

extras/plugin_example/analysis_options.yaml

@@ -1,5 +1,8 @@
 include: package:pedantic/analysis_options.yaml
 
+# The source-controlled version of this file should always have the plugin commented out. Our development version with
+# the websocket proxy causes analysis errors when to plugin doesn't run
+
 #analyzer:
 #  plugins:
-#    - moor
+#   - moor

moor_generator/lib/src/analyzer/moor/parser.dart

@@ -34,6 +34,7 @@ class MoorParser {
 
     for (var error in result.errors) {
       step.reportError(ErrorInMoorFile(
+        severity: Severity.error,
         span: error.token.span,
         message: error.message,
       ));

moor_generator/lib/src/analyzer/sql_queries/lints/linter.dart

@@ -103,7 +103,7 @@ class _LintingVisitor extends RecursiveVisitor<void> {
           .any((t) => t.name.toUpperCase() == c.name.name.toUpperCase()));
 
       if (notPresent.isNotEmpty) {
-        final msg = notPresent.join(', ');
+        final msg = notPresent.map((c) => c.name.name).join(', ');
 
         linter.lints.add(AnalysisError(
           type: AnalysisErrorType.other,

moor_generator/lib/src/backends/plugin/services/outline.dart

@@ -48,7 +48,9 @@ class _OutlineVisitor extends RecursiveVisitor<void> {
 
   @override
   void visitColumnDefinition(ColumnDefinition e) {
-    _startElement(ElementKind.FIELD, e.columnName, e)..returnType = e.typeName;
+    // we use parameters instead of returnType because VS Code doesn't show
+    // the return type but we'd really like it to be shown
+    _startElement(ElementKind.FIELD, e.columnName, e)..parameters = e.typeName;
 
     super.visitChildren(e);
     collector.endElement();

moor_generator/pubspec.yaml

@@ -13,7 +13,7 @@ environment:
 
 dependencies:
   analyzer: '>=0.36.4 <0.39.0'
-  analyzer_plugin:
+  analyzer_plugin: ^0.1.0
   collection: ^1.14.0
   recase: ^2.0.1
   built_value: ^6.3.0

sqlparser/lib/src/analysis/types/resolver.dart

@@ -34,6 +34,21 @@ class TypeResolver {
     _results.putIfAbsent(t, () => result);
   }
 
+  bool _canResolve(Typeable t) {
+    if (t is Column) return true;
+    // neither a column nor an expression. Can't resolve that
+    if (t is! Expression) return false;
+
+    final expr = t as Expression;
+    // if this expression contains a variable we didn't resolve yet, we can't
+    // be sure that we can resolve the entire expression.
+    final containsVariable = expr.selfAndDescendants.any((node) =>
+        (node is Variable || node is DartExpressionPlaceholder) &&
+        !_results.containsKey(node));
+
+    return !containsVariable;
+  }
+
   ResolveResult resolveOrInfer(Typeable t) {
     if (t is Column) {
       return resolveColumn(t);
@@ -314,10 +329,17 @@ class TypeResolver {
         parent is BinaryExpression ||
         parent is BetweenExpression ||
         parent is CaseExpression) {
-      final relevant = parent.childNodes
-          .lastWhere((node) => node is Expression && node != argument);
-      final resolved = resolveExpression(relevant as Expression);
+      final relevant = parent.childNodes.lastWhere((node) {
+        return node != argument &&
+            node is Typeable &&
+            _canResolve(node as Typeable);
+      }, orElse: () => null);
 
+      if (relevant == null) {
+        return const ResolveResult.unknown();
+      }
+
+      final resolved = justResolve(relevant as Typeable);
       // if we have "a x IN argument" expression, the argument will be an array
       if (parent is InExpression && argument == parent.inside) {
         return resolved.mapResult((r) => r.toArray(true));

sqlparser/lib/src/ast/ast.dart

@@ -98,6 +98,13 @@ abstract class AstNode {
     }
   }
 
+  /// Returns an iterable that fields yields this node, followed by
+  /// [allDescendants].
+  Iterable<AstNode> get selfAndDescendants sync* {
+    yield this;
+    yield* allDescendants;
+  }
+
   final Map<Type, dynamic> _metadata = {};
 
   /// Returns the metadata of type [T] that might have been set on this node, or