Running Cayley in the browser


#1

As mentioned in this message, Cayley can be compiled to JS using GopherJS and it works quite well in the browser:

This is interesting from multiple points of view:

  • We can run a Cayley engine that works entirely in browser, and even offline.
  • The data can be stored in memory, Local Storage, Indexed Storage (local) or PouchDB (local+remote).
  • Data on the client can be kept in (readonly) sync with Cayley server by streaming replication log back to the client.

And all we need to add the initial support is to add a build script, and maybe switch from current UUID lib to allow the code to compile without “net” dependency.

Then we can try to implement one JS-specific backend using KV abstraction (it’s the easiest way right now) and store the data in Local Storage, for example.

@elliott5 Can you please share the build steps needed for building and minification of the code?


#2

Want to try Cayley in JS yourself?

Why not compile the trivial code at https://elliott5.github.io/cayleyjs/ by copying it into main.go on your machine.

Install GopherJS with: go get -u github.com/gopherjs/gopherjs

Create minified JS from your code with: gopherjs build main.go -m

Then create an HTML file: ```

Look at the JS console! ```

Now just open the HTML file you have created and see that the code has run :wink:


#3

Oh, that’s nice, I expected something more involved… Like installing tools from npm, or something :grin:

OK, I’ll try to compile it, and if everything goes well, we’ll need to figure out what API we want to expose from it and make a separate file under ./cmd/ that will expose this API to JS. After that we will include it in our build process. I’m also working on a new Cayley UI, and it might be interesting to include this API implementation to it.


#4

I’m not sure there is a need to expose a JS API, unless you want to offer Cayley as an NPM module for example.

I write everything in Go, I only ever write JS as glue, when I need to call Go code from HTML, for example:

<button onclick="MyGoGlobal.DoIt(42);">Do It</button>

Where the Go code says:

        doIt := func(x int){
             	go func() { // callbacks need to be goroutines in GopherJS
                        fmt.Println("Do It to", x)
                }()
        }
        js.Global.Set("MyGoGlobal", map[string]interface{}{
             "DoIt":    doIt,
        })

For more on JS specifics, see: https://github.com/gopherjs/gopherjs/wiki/JavaScript-Tips-and-Gotchas


#5

Yes, I’m thinking about the module.

Currently I have a ES6 module that wraps our HTTP API and exposes few JS methods like Query, WriteQuads, etc. And UI calls methods on this module.

Now it’s possible to (optionally) replace this module with in-browser version of Cayley, and UI should still work the same way as it does right now.

There is also a possibility to expose our parsers to verify data and/or queries client-side, highlight errors, etc.


#6

@lnshi I think you might be interested in discussing a common JS API for Cayley :slight_smile:


#7

I’ve been looking into this a little further.

https://github.com/flimzy/kivik provides a database driver for CouchDB/PouchDB in Go/GopherJS.

CouchDB & PouchDB also replicate.

The only issue I can see is that PouchDB does not support transactions.

The alternative would be to use the JS IndexedDB API directly, accepting that support is poor in some browsers. Indeed non-existent for v2.0 in IE/Edge. Using PouchDB has the advantage of choosing which browser database option is likely to work best.


Couchbase as Storage Engine
#8

@elliott5 I just checked Kivik, and it seems like it will fit a nosql API naturally. The subset of operations needed to make it work should be reasonably small, and it will be great if you could make a prototype for CouchDB/PouchDB.

In worst case, we can always fallback to KV API and/or IndexedDB.

Also, it will be great to hear some feedback on NoSQL API - if there is a set of operations that is hard to implement in any of these DBs with current design, it’s a good time to change API for better compatibility. I’m currently working on this part to be able to include ElasticSearch backend as well.

Transactions are a good point, but they are also weak in Mongo. In most case we only need an atomic increment on one document field to make node ref count work. Multi-quad transactions might work properly in Mongo, but if any of JS DBs can properly implement it - I will expose hooks for it.


#9

That’s wonderful @dennwc, thank you. I’ll look at your new nosql API and see if I can knock-up a prototype (working from the nosql-indexes branch of your personal cayley repo).

Meanwhile, I see that the author of GopherJS is making good progress with writing the WASM target for the main go compiler, which is great news: https://github.com/golang/go/issues/18892#issuecomment-347057409


#10

@dennwc Just starting to write some code against the NoSQL api today.

The big issue so far is that context is not always passed through the API and Kivik uses it extensively.


#11

Great news, the UUID library used by cayley has accepted my PR to remove the “net” dependency when using GopherJS: https://github.com/pborman/uuid/pull/40#event-1361743674 - so that (tiny) element of the work is complete.


#12

Regarding NoSQL API losing context, feel free add context to all it’s methods - was considering this anyway :slight_smile:


#13

I suggest that all the Database interface methods have context as the first parameter, and that context is removed from all the “child” methods (like Query.One).
.


#14

This might not work well for DocIterator - it might be passed around and should use the context of the last caller, not the first one. I would say it make sense we can add contexts to Insert, FindByKey and EnsureIndexes and leave other builders as-is, since they already pass context in Do, One, Next, etc.


#15

Good call.