WebAssembly and Blockchain Happenings

I am jazzed by the appearance of CraneLift and WASMTime.

CraneLift is a JIT webassembly compiler, and WASMTime, which is a complete sandboxed environment. However, it appears to be for Non-Web Embedding. Which is what I've been waiting for.

CraneLift will probably be the basis of our toolchain. 

There's also a nuget module for WASMTime that can be called from .net (though I might have to change all my external modules back again to the latest .net core rather than framework 4.8). This might be much simpler than trying to run V8 (via Microsoft ClearScript) -- and much more portable. I also don't feel any great loss from the not having a Javascript environment.

There apparently is a gRPC/protobuf implementation now for  Rust (if the first emissaries need to be written in Rust).

I'm making slow (very-slow) steady progress with the Emissary managers. The browser launches them and they start to do their startup handshakes. However, the next steps, which is mostly being able to log into the various Kudo Servers is going to take some effort. I have the stubs of the new Kudo Authentication engine started.

CraneLift might also let me build server side emissary modules.

I guess I'm going to have to learn Rust after all.


Starting to look at blockchain infrastructure like Corda. The true functioning of the Kudo server is basically a blockchain engine, or at least what it does would benefit from a blockchain journal. A true Kudo (like a game level or a certification) would benefit from blockchain auditing. Nothing much more useless than a blockchain which has an audience of one (like a stand alone system). It's only benefit is being shared far and wide.

I don't think Corda is .net (may be Java/Kotlin). But, it dockerizable, and has an API we can call.

Considering IdentityServer4

Got a bit of organizational work done over the weekend at Orycon.  I'm mostly trying to map out the intra-browser communications via GRPC.   The browser can start up and launch it's Emissary processes.

I'm seriously considering making IdentityServer4 a big part of the Kudo server since it seems to overlap with what it was supposed to do.   There's still a lot to the Kudo server separate from authorization/authentication, but this seems to be a big chunk I don't have to write.

Embryonic Browser Starting

I'm getting the basic browser put together.   Getting all of the thread managers for the Emissary controller projects.   I think I've got the gRPC set up on the browser (on the Unity side, this is the hard part).  I'll get the basics together to launch the basic processes and get a two-way ping going so the Emissary Processes can die when the browser stops (or they'll just pile up).

I think I've even got a good handle on initial security.  I am only doing HTTP/2 on localhost.  I'm not planning on other security yet.  The browser will generate a random token which it will pass to its child processes in the environment (not the command line).   That should keep other users from attacking the processes.

Having emissaries and other such things running on servers will have to wait until we have basically a VRWorlds server running on each machine in the cluster -- but I am planning the ability to have a browser cluster.

Initial Sketch of the Loading Area

Interfaces Defined

Not much progress in the last week or two.  My trip to Bubonicon though I managed to do a pretty creditable list of all the initial APIs.  So, I'll be putting this into a document and beginning the effort of defining all of the gRPC definitions for them.

I'm looking forward to having enough pieces to get the prototype browser working, with multiple processes and the ability to load in a working emissary.

In the meantime I've been deep-learning kubernetes by putting a large bit of my own production into docker, and porting some python from v2 to v3.

API Layers

Cranking up the old Visio again...

Ogres are like onions, because they have lots of layers...

If you see the crazy number of empty projects I've been checking in, you'll see now what I intended.   This drawing was also to give myself some more clarity of what I was just randomly creating as I saw a need.   Now the thing this doesn't show is what APIs (or parts of APIs) are implemented where.  I will work on this part next, so I can start to get a full skeleton of what the API trees are going to need to contain.

I also saw the need as I was writing for a provisioning API (how many servers are where--mostly interacting with the Data Plane and the Orchestration (though their data is stored in etcd).   Also most of the server types will initially need a conventional web page for signing up and utilizing.  Someday that will all get done in VR.  Ultimately there will be some kind of Dashboard -- again someday visualized in VR.  For now, I'll find some conventional monitoring tool.

I also like that I've have some good structure happening on the browser.  I think that I know how that's supposed to be built.   Also, I've mostly figured out out how its Control/Data Plane is going to work (ProtoBufs over Named Pipes--though I wish I could just use gRPC).   I also had an amusing idea--no real reason Named Pipes wouldn't work across multiple machines if you want to turbo-charge your browser.  Though also should allow conventional sockets so your browser-assist can come from a Linux box.  Also probably should consider building a Unity Linux VR browser, so need to be agnostic on this part of the infrastructure.  I could make co-processor boxes that you attach to your regular machine.

Anyway.  Little steps happening in the background.

Browser Construction

I'm starting to make a push now to actually get a browser foundation started.  My current issue is .net frameworks.   My common libraries, and gRPC schema libraries are all built in .NET Standard, but to my knowledge now, there are no such versions of ClearScript, which I'm relying on for my V8 embedding. 

So, experimenting, I set the browser compatibility to .NET Standard 2.0, the thing starts up (after doing lots up updates to unity and the packages), with ClearScript installed with NuGet--even though it's a .NET 4.5 assembly--and I'm not yet instantiating the thing or doing anything with the V8 engine.   That's probably next.  Plan is to make a nice generic interface which the libraries can call to get attached to the V8 (which should be done rather formally to assure security and robustness -- and lets me change out the V8 module as needed).

Though, I've been thinking again.  That's the trouble with not hitting "publish".   I was looking at how the Chrome browser talks to itself across it's different processes...  It does it via their own IPC via named-pipes.  I was wondering how easy this would be with gRPC.  Maybe I can just have my own C# executable running there communicating this way.  It would help with scaleout really well.  Anyway, continuing experiments building a browser foundation. 

Server-Side Embedded Scripting

I'm starting to look very hard the corresponding server-side analog to the Emissary, the Coordinator, or Implementer or some other $10 word to be introduced in the future.

I'm pretty much committed to writing my own V8 wrapper on the browser --- I really don't think I have much choice there, but I'm not sure I need to go that way on the server side (though I did discover that the v8.net wrapper for .net seems to run fine in .net core).   Though I think I can use MicroSoft ClearScript on the browser side.  But on the server side I don't have to do this.  I've been looking at alternatives and node.js is pretty much out.  It's just kind of hot mess sandbox-wise.  It assumes it has full user-level system access, like any piece of code on the metal would.

However, I am looking very hard at deno.   It has the ability to be run as a secure sandbox (you must enable network and disk reading/writing explicitly), and I think I can extend it to do what I need to.   It was scale up and out as needed in the cloud.

Basically, it needs to be able to load and implement gRPC (and protocol buffer) bindings to the Emissary.  Also I think there needs to be able to load multiple "Coordinators" in each V8.  Maybe it's light enough where I can limit a v8 to only one manufacturer.  I know I can't get a way with this on the browser side just yet.  Anyway, it has to be able to call the API for that server level to implement the business logic for that type of server (kudo, world, entity, avatar--also world and avatar servers are also entity servers depending on their role in the scene).

Also, it does seem like it works just fine with wasm, which is where I ultimately want to go.   They're making lots of use of typescript, which is also where I want to go (if I must do JavaScript like coding), and with Rust, and so maybe I want to look harder at that.  I may have to write the bindings in Rust, but we'll see.

...

The more I look at this, it may be impractical to directly add what deno calls "bindings".  Perhaps this can be done via gRPC or AMQP with the network layer turned on.   In this case, each docker images would be pre-created and perhaps constrained with iptables, since we should know what code we need to handle -- we can roll this out and handle versioning by just shifting to a new docker and have the old one stand down after it goes idle.   At least we can start with this.   The API layer should be nicely abstracted so we can change how it works.

Looks like I seriously need to up my TypeScript game.

...

And after even more looking, deno isn't quite ready for prime time yet.   I think for initial implementations, I'm being a little paranoid.   Ultimately if it implements the API it can be run, and it can be sandboxed pretty well in docker.  Be it in node.js, or in python, or whatever.  True sandboxing perfection we can work on over time.