Upgrade to CodeMirror 6, add SQL autocomplete #1893

bgrins · 2022-11-15T20:52:35Z

In an effort to get closer to table / column autocomplete I took a shot at #1796. I haven't done a lot of testing but would be curious if this fixes some of the concerns raised in #1796 (comment) for example.

Done:

Changed to bundling using rollup as per https://codemirror.net/examples/bundle/
Restored a fromTextArea-like function from https://codemirror.net/docs/migration/
Removed old JS and CSS files (no external CSS needed anymore as per https://codemirror.net/examples/styling/)
Updated instructions for building the bundle

Not done:

cmResize had an error, so commented out the resize handle
Add extraKeys option for shift+enter and tab

📚 Documentation preview 📚: https://datasette--1893.org.readthedocs.build/en/1893/

bgrins · 2022-11-15T20:54:10Z

datasette/templates/_codemirror_foot.html

@@ -1,38 +1,43 @@
 <script>
-window.onload = () => {
+  window.onload = () => {


Should be a separate commit, but switching this to DOMContentLoaded should prevent the flash-of-uncodemirrored-object that can happen between the first render and the load event firing

That sounds great to me.

Filed #1894 to track

bgrins · 2022-11-15T20:55:40Z

Should also minify the bundled output

bgrins · 2022-11-15T21:11:42Z

extraKeys is done - Shift+Enter is added in the helper function, and it appears that the Tab behavior now defaults to what the Tab: false setting was doing (allowing it to escape to the form)

bgrins · 2022-11-15T21:12:38Z

https://github.com/Sphinxxxx/cm-resize isn't compatible with 6. There's a suggestion to try using CSS resize in https://discuss.codemirror.net/t/resizing-codemirror-6/3265/2

bgrins · 2022-11-15T21:24:59Z

datasette/static/cm-editor.js

+    ],
+  });
+
+  // Idea taken from https://discuss.codemirror.net/t/resizing-codemirror-6/3265.


It's unclear to me if this ResizeObserver and requestMeasure is needed. It's suggested in the thread but from some light manual testing the behavior seems the same with or without

For manual testing, any changes to this file require the build as outlined in the docs:

node_modules/.bin/rollup datasette/static/cm-editor.js -f iife -n cm -o datasette/static/cm-editor.bundle.js \ -p @rollup/plugin-node-resolve -p @rollup/plugin-terser

bgrins · 2022-11-15T23:51:35Z

I experimented with autocompleting the actual schema in bgrins@8431c98, but it would need some work (current problems with it listed in the commit message there)

simonw · 2022-11-16T01:21:41Z

I just deployed a demo instance like this (using the commit hash from this PR):

datasette publish vercel fixtures.db \
  --branch 544f7025900b78f63c34b9985522271ba5fd9c0f \
  --project datasette-pr-1893 \
  --scope  datasette \
  --about 'PR 1893' \
  --about_url https://github.com/simonw/datasette/pull/1893

Here's the result: https://datasette-pr-1893.vercel.app/fixtures

simonw · 2022-11-16T01:23:47Z

Autocomplete here looks promising (I've wanted that to work for years!), but it does currently show a whole bunch of suggestions which aren't part of the SQLite SQL dialect:

simonw · 2022-11-16T01:26:59Z

Resizing works great for me - and the page automatically sizes the editor to fit an existing query, e.g. on https://datasette-pr-1893.vercel.app/fixtures?sql=select+id%2C+content%2C+content2%0D%0A++from+primary_key_multiple_columns_explicit_label%0D%0A++order+by+id%0D%0A++limit+101

simonw · 2022-11-16T01:28:30Z

datasette/templates/_codemirror.html

-<link rel="stylesheet" href="{{ base_url }}-/static/codemirror-5.57.0.min.css" />
-<script src="{{ base_url }}-/static/codemirror-5.57.0-sql.min.js"></script>
-<script src="{{ base_url }}-/static/cm-resize-1.0.1.min.js"></script>
+<script src="{{ base_url }}-/static/cm-editor.bundle.js"></script>


I like having the CodeMirror version in the filename, partly to ensure cache busting but mainly because that way I can see at a glance which version was most recently vendored into Datasette.

Makes sense

simonw · 2022-11-16T01:31:27Z

datasette/static/cm-editor.js

+        },
+      ]),
+      sql({
+        dialect: SQLite,


I don't understand why we're seeing all of those weird non-SQLite keywords in autocomplete while using the SQLite dialect here.

It looks like those come from these lists here: https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/src/tokens.ts#L147-L148

I'm not sure why the SQLite dialect isn't over-riding them. Maybe a missing feature in codemirror/lang-sql?

... weirdly I can't find the implementation of the SQLite dialect in that repo at all? https://github.com/search?q=repo%3Acodemirror/lang-sql%20sqlite&type=code

Huh, GitHub code search let me down there. I found it in the code here: https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/src/sql.ts#L231-L239

Here's that full block of code:

/// [SQLite](https://sqlite.org/) dialect. export const SQLite = SQLDialect.define({ keywords: SQLKeywords + "abort analyze attach autoincrement conflict database detach exclusive fail glob ignore index indexed instead isnull notnull offset plan pragma query raise regexp reindex rename replace temp vacuum virtual", types: SQLTypes + "bool blob long longblob longtext medium mediumblob mediumint mediumtext tinyblob tinyint tinytext text bigint int2 int8 unsigned signed real", builtin: "auth backup bail changes clone databases dbinfo dump echo eqp explain fullschema headers help import imposter indexes iotrace lint load log mode nullvalue once print prompt quit restore save scanstats separator shell show stats system tables testcase timeout timer trace vfsinfo vfslist vfsname width", operatorChars: "*+-%<>!=&|/~", identifierQuotes: "`\"", specialVar: "@:?$" })

I think the problem here is that it adds SQLKeywords to that list.

We'd probably be best off defining our own simpler dialect here.

Looking at that code I think it'll be pretty easy to define our own dialect, given you have specific keywords/types/builtins in mind

For reference, here's the SQLKeywords and SQLTypes. Some of those will clearly be applicable here but also sounds like many aren't:

https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/src/tokens.ts#L147-L148

Would be nice if we could pull a similar list directly from SQLite grammar

Confirmed with

--- a/datasette/static/cm-editor-6.0.1.js +++ b/datasette/static/cm-editor-6.0.1.js @@ -1,7 +1,18 @@ import { EditorView, basicSetup } from "codemirror"; import { keymap } from "@codemirror/view"; -import { sql, SQLite } from "@codemirror/lang-sql"; +import { sql, SQLDialect } from "@codemirror/lang-sql"; +const SQLite = SQLDialect.define({ + keywords: + "abort analyze attach autoincrement conflict database detach exclusive fail glob ignore index indexed instead isnull notnull offset plan pragma query raise regexp reindex rename replace temp vacuum virtual", + types: + "bool blob long longblob longtext medium mediumblob mediumint mediumtext tinyblob tinyint tinytext text bigint int2 int8 unsigned signed real", + builtin: + "auth backup bail changes clone databases dbinfo dump echo eqp explain fullschema headers help import imposter indexes iotrace lint load log mode nullvalue once print prompt quit restore save scanstats separator shell show stats system tables testcase timeout timer trace vfsinfo vfslist vfsname width", + operatorChars: "*+-%<>!=&|/~", + identifierQuotes: '`"', + specialVar: "@:?$", +});

that it updates the dialect as we'd expect (not highlighting or autocompleting SQLKeywords that aren't prepended anymore, like select) - so looks like it's only a matter of deciding on the exact lists to get that fixed.

We can simplify this a lot by skipping everything that's not used in regular SELECT queries - anything to do with alter table, insert, update and so on.

Datasette may well support SQL queries that do more than just select data in the future (it already does via the https://datasette.io/plugins/datasette-write plugin) but I'm happy for us to revisit the grammar at that point - we can even serve a different grammar to the version of CodeMirror that's attached to a SQL textarea which can perform more than just SELECT queries.

simonw · 2022-11-16T02:54:22Z

If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values.

simonw · 2022-11-16T02:59:00Z

The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though.

The textarea does get a weird border around it when focused on iPhone though.

Focused:

Not focused:

simonw · 2022-11-16T03:00:04Z

Oops, the tests are failing because of a test failure I introduced here:

Autocomplete text entry for filter values that correspond to facets #1890

simonw · 2022-11-16T03:04:57Z

If you rebase from main you should get the fix for that test failure.

bgrins · 2022-11-16T03:11:46Z

Was just reviewing the SQL options and there's an upperCaseKeywords if we'd rather have SELECT vs select. Datasette seems to prefer lowercase so probably best to keep it as-is

simonw · 2022-11-16T03:16:36Z

Yeah I haven't written this down anywhere but Datasette definitely has an undocumented preference for lower-case SQL.

bgrins · 2022-11-16T03:18:06Z

If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values.

This version "sort of" works when on the main database page where the template passes the relevant data bgrins@8431c98 by doing this and passing that into the schema object:

  let TABLES_DATA = [];
  {% if tables is defined %}   
  TABLES_DATA = {{ tables | tojson(indent=2) }};
  {% endif %}

  // Turn into an object, shaped like https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L27.
  const TABLES_SCHEMA = Object.fromEntries(
    new Map(
      TABLES_DATA.map((table) => {
        return [table.name, table.columns];
      })
    ).entries()
  );

But there are a number of papercuts with it - it's not escaping table names with spaces (likely be fixable from the data being passed into the view) but mainly it doesn't seem to autocomplete columns. I think it might only want to do it when you first type the table name from my read of https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L37. It's possible I'm just passing something wrong, but it may end up being something that needs feature work upstream.

simonw · 2022-11-16T03:59:03Z

Deployed a fresh copy:

datasette publish vercel fixtures.db \
  --branch b7b2942b13f9ea09cfa9f8c73e2869b9bd2349ae \
  --project datasette-pr-1893 \
  --about 'PR 1893' \
  --about_url https://github.com/simonw/datasette/pull/1893 \
  --scope  datasette

https://datasette-pr-1893.vercel.app/fixtures

simonw · 2022-11-16T04:00:12Z

Have you ever seen CodeMirror correctly auto-completing columns? I'm not entirely sure I believe that the feature works anywhere else.

codecov · 2022-11-16T04:05:12Z

Codecov Report

Base: 92.55% // Head: 92.55% // No change to project coverage 👍

Coverage data is based on head (f254be4) compared to base (6f610e1).
Patch has no changes to coverable lines.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1893   +/-   ##
=======================================
  Coverage   92.55%   92.55%           
=======================================
  Files          35       35           
  Lines        4432     4432           
=======================================
  Hits         4102     4102           
  Misses        330      330

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

bgrins · 2022-11-16T04:27:51Z

The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though.

The textarea does get a weird border around it when focused on iPhone though.

The default focus styles appear to be

.c1.cm-editor.cm-focused {
  outline: 1px dotted #212121;
}

Which I also see on desktop. Would be nice to changed to whatever the default UA textarea styles are to blend in better but I wouldn't recommend removing it entirely - just to keep the visual indication that the element is focused. Maybe followup material to have a theming pass

bgrins · 2022-11-16T18:58:43Z

Nice. And is it possible to include another field which is an escaped table name (only when necessary) - i.e. [123_starts_with_digits]. Or is that easy enough to derive on the client? I'm thinking we'd map those to Completion objects so that CM would show the non escaped text but complete to escaped.

bgrins · 2022-11-16T18:59:49Z

Or I guess you could return only the escaped table name and then we could derive the unescaped from the client side (removing the outer [] when present)

bgrins · 2022-11-16T21:19:13Z

Alright, added Cmd+Enter to submit (Ctrl+Enter on Windows as well bc of using Meta-Enter on codemirror). We can make that MacOS only by changing the combo to Cmd+Enter specifically but I think it's probably fine to have both.

bgrins · 2022-11-16T21:49:51Z

I think the table completion still has some quirks to work out. Something like

       schema: {
         "[123_starts_with_digits]": ["content"],
       }

Seems to work alright, although it will append it after any other numbers you've started typing - so you end up with select * from 12[123_starts_with_digits] if you typed "12" to get the completion to appear. This might just be an issue with numeric names, I haven't tested it in a lot of detail.

You can do

          searchable: [
            {
              label: "name with . and spaces",
              apply: "[name with . and spaces]",
            },
            "pk",
            "text1",
            "text2",
          ],

Which is pretty neat and will show the non-escaped string but complete to the escaped one. You can't easily do that with the table names themselves (you can pass a tables array like so https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/src/sql.ts#L121 but it will overwrite the columns from the schema ).

It's buggy enough (bad output for these unusual table names) that I'd suggest that work gets moved into a follow up to the upgrade to 6. That would give space to sort out how to deliver that to the view directly, figure out where name escaping should happen, and have overall testing to uncover bugs and fix papercuts before enabling it.

simonw · 2022-11-16T22:16:03Z

Honestly I'm not too bothered if table names with weird characters don't work correctly here - I care about those in the Datasette fixtures.db database because Datasette aims to support ANY valid SQLite database, so I need stuff in the test suite that includes weird edge cases like this. But I would hope very few people actually create tables with spaces in their names, so it's not a huge concern to me if autocompletion doesn't work properly for those.

simonw · 2022-11-16T22:17:24Z

Deployed 0a649e8: https://datasette-pr-1893.vercel.app/fixtures

bgrins · 2022-11-16T22:59:57Z

I can push up a commit that uses the static fixtures schema for testing, but given that the query used to generate it is authed we would still need some work to make that work on live data, right? Ideally it could come down to db and query views directly to avoid waiting on an extra xhr and managing that state change.On Nov 16, 2022, at 2:16 PM, Simon Willison ***@***.***> wrote: Honestly I'm not too bothered if table names with weird characters don't work correctly here - I care about those in the Datasette fixtures.db database because Datasette aims to support ANY valid SQLite database, so I need stuff in the test suite that includes weird edge cases like this. But I would hope very few people actually create tables with spaces in their names, so it's not a huge concern to me if autocompletion doesn't work properly for those. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

simonw · 2022-11-16T23:08:34Z

I can push up a commit that uses the static fixtures schema for testing, but given that the query used to generate it is authed we would still need some work to make that work on live data, right?

Yeah, push that up. I'm happy to wire in the query right after we land this.

bgrins · 2022-11-16T23:18:17Z

Alright with f254be4 we should be getting autocomplete on fixture data. Give that a test and see what you think

simonw · 2022-11-16T23:44:36Z

Deployed that to https://datasette-pr-1893.vercel.app/fixtures - looks good to me!

simonw · 2022-11-16T23:47:05Z

OK, let's do it! Thanks so much for this.

simonw · 2022-11-16T23:47:13Z

I'll open a follow-up issue to fix the schema.

bgrins · 2022-11-16T23:50:58Z

Should we empty out the fixture schema to avoid fixture autocomplete showing up on live databases in the interim, or are you planning to tackle #1897 shortly?

simonw · 2022-11-16T23:54:02Z

I'm going to tackle #1897 in the next few minutes.

Tests failed due to Prettier check, just pushed a fix so it would ignore .bundle.js too.

Refs #1893

* Upgrade to CodeMirror 6 * Update contributing docs * Change how resizing works * Define a custom SQLite autocomplete dialect * Add meta-enter to submit * Add fixture schema for testing

Refs #1893

bgrins commented Nov 15, 2022

View reviewed changes

simonw reviewed Nov 16, 2022

View reviewed changes

bgrins added 7 commits November 15, 2022 19:21

WIP - CM6

417f34c

minify output and update docs

acd812d

add shift enter key

3fadb08

css resize

5073382

remove log

f8ddec7

remove dead code

e3e2feb

version number in bundle name

b7b2942

bgrins force-pushed the codemirror-6 branch from bfb668f to b7b2942 Compare November 16, 2022 03:47

add meta-enter to submit

f7e5f35

pass along other sql options into editorFromTextArea

0a649e8

add fixture schema for testing

f254be4

simonw changed the title ~~WIP - CodeMirror 6~~ Upgrade to CodeMirror 6, add SQL autocomplete Nov 16, 2022

simonw mentioned this pull request Nov 16, 2022

Serve schema JSON to the SQL editor to enable autocomplete #1897

Closed

simonw merged commit ae11fa5 into simonw:main Nov 16, 2022

simonw added a commit that referenced this pull request Nov 16, 2022

Prettier should ignore bundle.js file - refs #1893

00e233d

bgrins mentioned this pull request Nov 17, 2022

Clicking within the CodeMirror area below the SQL (i.e. when there's only a single line) doesn't cause the editor to get focused #1899

Closed

simonw added a commit that referenced this pull request Nov 18, 2022

Include SQL schema for CodeMirror on query pages, closes #1897

3e61a41

Refs #1893

simonw added a commit that referenced this pull request Nov 18, 2022

Prettier should ignore bundle.js file - refs #1893

8494be0

simonw added a commit that referenced this pull request Nov 18, 2022

Include SQL schema for CodeMirror on query pages, closes #1897

98611b3

Refs #1893

This was referenced Dec 23, 2022

Research an upgrade to CodeMirror 6 #1796

Closed

Try again with SQLite codemirror support #1614

Open

simonw mentioned this pull request Jan 17, 2023

Add ?_extra= mechanism for requesting extra properties in JSON #262

Open

simonw mentioned this pull request Nov 3, 2023

Try out CodeMirror SQL hints #949

Closed

Uh oh!

Upgrade to CodeMirror 6, add SQL autocomplete #1893

Upgrade to CodeMirror 6, add SQL autocomplete #1893

Uh oh!

Conversation

bgrins commented Nov 15, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgrins commented Nov 15, 2022

Uh oh!

bgrins commented Nov 15, 2022

Uh oh!

bgrins commented Nov 15, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgrins commented Nov 15, 2022

Uh oh!

simonw commented Nov 16, 2022

Uh oh!

simonw commented Nov 16, 2022

Uh oh!

simonw commented Nov 16, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonw commented Nov 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonw commented Nov 16, 2022

Uh oh!

simonw commented Nov 16, 2022

Uh oh!

simonw commented Nov 16, 2022

Uh oh!

bgrins commented Nov 16, 2022

Uh oh!

simonw commented Nov 16, 2022

Uh oh!

bgrins commented Nov 16, 2022

Uh oh!

simonw commented Nov 16, 2022

Uh oh!

simonw commented Nov 16, 2022

Uh oh!

codecov bot commented Nov 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bgrins commented Nov 16, 2022

Uh oh!

bgrins commented Nov 16, 2022

bgrins commented Nov 15, 2022 •

edited by github-actions bot

Loading

simonw commented Nov 16, 2022 •

edited

Loading

codecov bot commented Nov 16, 2022 •

edited

Loading