Random Thoughts, Some Progress

 Lots of work at work, so not as much time for projects.

Smattering of thoughts:  

1. Godot doesn't really seem ready for prime-time with c# yet.  Not bad, and we will keep and eye on it.   I'll try to make my libraries in a way that theoretically we could make them agnostic to the engine (probably biggest issue there will be the way the engine does mesh representation), and hopefully as little code as possible on the engine side.
2. Keeping a close eye on wasmer/wasmtime.   I think I will still need wasmer, since it is .netstandard 2.0.  The Wasmtime restriction to .netstandard 2.1 is a problem with unity (and probably godot).   Though I can run the emissaries as separate processes, I think I want to do that later.  Let's get it working before we go too crazy with a 'scale out' plan, just not preclude ourselves from going that way.  I have very little time to code, so have to make a bee-line to functionality.   We will probably revisit this when Unity changes their .net model, to .net core or possible .net 5 (the .net version to rule them all).  Both of these projects are ticking a long.   I still find wasmer's WAPM projects a powerful missing link, though version pinning and signed downloads of external code would have to happen.
3. Starting to look at some naked-codeless emissaries, which just have content in them.  It's one option for the emissary, just to hold objects, or to hold references to objects.   I'm refactoring the ldraw code so I can just put lego entities in.  Later these can indirect to an URL rather than needing to be in the emissary--though they will have to match a full checksum when they download--they will then be able to be proxy signed.   Looking at getting an STL loader too.  Then some work on a universal (protobuf based) mesh format.
4. Very intrigued by Kafka.   I think log based messages might be a giant win over Rabbit/AMQP style pub/sub.   The use-case is that the world server can simply make every client a producer and everyone can be a consumer to see their scenes change.   The interesting thing is that perhaps, if you don't have a complete cache of the scene, you can then traverse the log in the reverse direction back to a fixed low-water mark (which represents having every entity having presented it's status in the log at least once), fully recapitulating the status of the scene.  You have to traverse it backwards since a given entity might report many times, and you don't want to have the scene be remade in order.   Physics will have to be suspended during the repaint.  Also, if  you mark each status with the timestamp of the last previous change, you can make a hash to ignore all but the last status report of every entity (and not have to keep inactive entities in your hash -- if their last-changed is back before the low-water mark.   Also Kafka messages can be done in protobuf (a supported standard).    World server will still need a database to track everything.   Might be easier than the redis idea.  Have to experiment.  I need a skeleton world server.  I'm building a Kafka server to try out.
5. These log based systems like Kafka (and really like the mongo oplog) are intriguing.  I'm thinking of a way to do the image and mesh caches that use this, where they use a given number of fixed length cache files (likely memory mapped), and the oldest one times out and causes the oldest cached entries to be reloaded into a real file.   This doesn't fix evacuating trivial files or keeping core-files which are constantly used. Trivial files of course will fall out when the logs get rotated.  It's a very powerful scheme for keeping high-integrity databases without needed to do wild random access during setup, or any need to do garbage collection.

Doing some Rethinking

So, I've been progressing somewhat with Emissaries...   

But, I have been doing a little rethinking.   First, if I go with Unity as the browser platform (as I have), it might make sense to look harder at Wasmer (github), rather than WasmTime (github).   Wasmer is available in .net standard 2.0, which will permit it to run directly in Unity, rather than requiring me to build out the multi-process architecture initially to make it so that I can call WasmTime WebAssembly via a separate Emissary process.   This would be much easier as an initial effort, and would make is possible to have Entities that can directly interact with the browser engine.   I'm also very interested in the WAPM effort, which is the WebAssembly -Package Manager--inspired by NPM.   I think this is basically a linker, possibly even a dynamic-linker.  It's something I thought we'd need sooner rather than later...  Ultimately we want WebAssembly modules to act like shared libraries, so that only one instance of a given module need to be in memory on a given emissary server.

However, an even bigger and more momentous change, I'm starting to look at Godot games engine.   It's not nearly as sophisticated as Unity (Unreal doesn't seem like it's any kind of fit for what I need).   Also, I'm essentially building what Unity calls a "platform".   This isn't something you're really supposed to do in Unity, though since it would be open source and would probably not really be charging money for it (at least directly, I would like to be able to monetize it somewhat---consulting, hosting, or something Patreon-like), I may or may not get into issues.   It also might be something Unity would be interested in sponsoring if this became popular.  You never know.

But, Godot is MIT license and includes all the sources.   It seems like it's got a good following, and a small but very talented core team.    Exposing the Godot API into wasm might even be something the Godot group would be enthusiastic about--it would be the holy-grail of scripting, not being tied to any particular language--though VRWorlds requires more sandboxing than that.  Perhaps the compromise there is to permit the sandbox to be created, and what APIs are exposed to it are done at creation (some level of WASI and Godot API or VRWorlds APIs can be specified).   As WASI embeddable WebAssembly capable languages start to become more plentiful (again, that's pretty much just Rust, C, C++, and maybe Javascript for now), this might become very attractive.

Godot does support procedural geometry, which we need since we don't really use the games engine to directly manage our content--it gets loaded with the emissaries.

GD script seems appealing.  I really enjoy python (though I  despise JavaScript).    It might even be worthwhile to dispense with c# and just do the interface code in GD and embed wasmer or wasmtime in c++ (theoretically it can also be done dynamically in GD script)--c# is apparently a brand new feature.   I can also put it into the engine myself, since all of the code is on GitHub.   So, I will be soul-searching and learning Godot a bit more.   It's possible that they might grow up together.  Obviously a smaller open-source project is riskier than a big commercial effort like Unity, but perhaps it will turn out somewhat like Blender did.

I can write modules directly in C and C++.   This might permit a lot of the frame-buffer operations I want to do to be very fast as opposed to having to write lots of unsafe pointer code in C#, though it's possible to call C/C++ from C# with interop.

Emissary Architecture Starting to Take Shape

So, I've been pretty deep in WASM architecture, and the WASI APIs.   I've pretty much gotten to the point where we can launch an emissary and interact with it.    So, now, it's time to actually build out the 3DOM APIs for interacting with the emissary and the message passing loops.   Initially I'm going to start building a Rust Module for this.  I do want to do this with other languages as it becomes practical.   I'd love to be able to do C# and Python, but I don't think they're quite there yet.   I've used mono-wasm to generate c# wasm, and pyodide for Python, but so far they're not particularly oriented to WASI yet and are more intended to run on a website (and interact with a DOM).

I think it will mostly go like this: one exposed setup method from the API layer, which will receive a protobuf buffer.   This will be using 'proto 3' and will be decoded and processed.  This gives us a backward and forward compatible startup.    The startup will register callbacks on the rust side for the message pump to process.   Then the emissary's 'main' basically returns.    A channel is created via grpc to the browser and eventually to the entity/avatar server.  Protobuf packets are passed in via these callback mechanisms.   The emissary runs and then returns and possibly returns one or more protobuf packets.   For now, I don't intend the emissary to have a thread to run when it is not processing a callback.   Initially I'd rather an emissary not be able to do things like mining bitcoin or other such undesirable activity.   The service managing the emissary will probably have a call timeout and will disable it if it doesn't respond in a timely fashion.

This facility kind of makes it hard to directly do gRPC, but at least we can do gRPC-like behavior passing protobufs around.   I can cheat and have a grpc wrapper call which contains a command type and the protobuf  as a raw vector.   The server calls are tricker, as the interior emissary-to-server gRPC calls are theoretically private (and ad-hoc)--so their formats can't be known by this middleware layer which is operating the emissary for the browser.    There are some up-and-coming grpc proxies out there.   Or I will have to do my own, which is just calling methods on the emissary and returning an answer (no actual network involved).   With my WASI layer, I could actually permit the emissary to open a socket, but I don't trust this yet. 

So, now I actually have to work out exactly what the browser-emissary interaction (which I've been calling the 3DOM api) actually is and how it works.   I've got some prototypes of this that I worked out last summer, but now I have to start building my .proto files to implement it.

Initially there will be no world, and no entity server to work this out.  I'll have the concept of a "system entity", which can simply run inside the loading area (which will eventually will become a mini-world which is served from inside the browser).   Also, eventually we'll have an avatar server, but initially the avatar is mostly just a vr camera FPS rig, with no smarts.    I'll be doing some checkins soon of some of this structure. 

The emissary will have an initial built-in parasitic overhead to have to have a copy of the loading module and the protobuf module.  Eventually I'm sure I'll be able to dynamically link such things so they can be shared between emissaries.   There's an effort for a package manager for webassembly (like npm) called wapm.  Perhaps there can be a package manifest and emissaries can be dynamically loaded.   I am ultimately intending for there to be hundreds, if not tens of thousands of emissaries to exist (in the distant future).   So optimization, sharing, hot-deploy, and versioning will have to be seamless.

Webassembly Madness

Spent a lot of time over the weekend getting webassembly jammed into my brain and tooling up.

I looked hard at mono-wasm (now built into the mainline distribution).   It was generating huge wasm files, so putting lots of code I wasn't explicitly adding into it.  I have to look much harder at this.   It seems like they're making an attempt to get the .net foundation classes implemented (though a lot of the code basically throws a "Not Implemented" error).   Still looks promising.  I'd rather the first gen emissaries be c#.   It's not really focused at WASI and embedding yet.  I think the emphasis is mostly for browser use and the DOM.

Discovered that there's a PyPi module with wasmtime in it, so I can embed that and make a test framework.  Trying to get my hello-world modules working in Rust and establishing the tool-chain for all of that.

They split out the wasmtime .net framework into a new package.  I think perhaps that they utterly redid the API, though it seems much easier to use.   It doesn't have to discover the host-wasi modules via reflection, so I can have a call which sets up which api's are available to each class of emissary (a method call can define each method).

I need to learn webassembly assembler.   It's been a good 40+ years since I did assembler regularly, but this one might end up being the ultimate superset assembler.   It would be interesting to build a real assembler for it (rather than the lisp-like fully regularized syntax it has now).

2020 Has Eaten My Brains...

Wow.  I've had a busy several months.   I think about this project quite a lot, but have had little time.  The SF conventions that I normally spend coding on this stuff have been cancelled.  Alas Norwescon...

Getting the First Person framework working on the loading deck.   I've got the VR integrated, but needing to get all the movement stuff working.   Unity is having problems with the SteamVR plugin with some of it's more recent versions.   I'm trying to run on 2019.3, but still have to remove several of the integration methods to get it to compile.   I'm going to try to get teleport movement working.  Supposedly Valve is going to write a new version of steamVR (or OpenVR--not exactly sure).  This might limit me to the 2019 version of unity for a while.  I think--rereading all this and updates, that I need to start using the XR framework.

Keeping an eye on Wasmtime.  They're doing good work there.   There's a new nuget preview.  It looks like if you look in the sources though that they're changing and simplifying the calling structure.  It looks like there's going to be a much easier way to integrate the private APIs I need to expose to it via private WASI.  You can just call a method to define a function.  So I can start getting together some classes to autogen the APIs.  It also lets me define different APIs and only load the ones that are appropriate (entities, avatars, and worlds likely have variations in what APIs are present).  And eventually will start to implement the standard WASI interfaces that are appropriate (like file IO, but only to the sandboxed and read-only emissary filesystem).    It's still looking like Rust for the first language for doing Emissaries, though I need to see how other systems are working.   Blazor is going nicely in the dotnet/c# world, but it works rather differently -- There's a CLR interpreter running in webassembly -- I might look into this and see if this is sufficiently well documented to write my own CLR classes for it or if I'll need my own webassembly module (it may be assuming a DOM and other non-embedded things--though eventually, it might be worthwhile having a working DOM).

In the just-fun category, I had an interesting idea.   Someone has an actual firmware VT100 emulator running on a virtual javascript 8080 cpu.   This is not what is conventionally known as a "vt100 emulator".   This truly is running the rom code from a vt100.   It was be possible to build one of these into an emissary (I see some 8080 emulators in rust).   I was originally thing that these emulators would run server-side and expose themselves in the VR environment via a pseudo-serial connection.    So, it would be truly interesting to do this with the other SIMH emulators and Mame and libretro.   These are mostly in C, it might be possible to get them into webassembly.   Mame has some other VT emulators built into them as well as the million other things Mame can do.   I hear rumors that there's an SDL layer that's been ported to webassembly (though probably for a DOM Canvas, but I can probably map that onto a texture).

Next big effort too is to get the various multi-process servers to bidirectionally ping each other and shut down when the browser exits.   There a ping that occurs the when the service comes up to indicate that it's ready.   I also had the thought to have some kind of indicator of presence and health on the loading deck, like a luminous sphere for each server.  Color could indicate load or CPU %.   It was a bit of an epiphany that I could start having simple emissaries that don't really require an external entity server, and can show them onto the loading deck.   Getting all of the pieces together for loading external content is still a ways off.  The initial avatar won't really need any emissaries, until we want to start showing a second-person view.   Though we might eventually show hands and such in first-player view--though that might be initially hardcoded and rely on what is available in SteamVr.

I had a sketch of the necessary emissary APIs, so I will also start fleshing those out.   Those need some practical hammering so we can come up with some nice coherent architecture for that.  Mostly it allows the emissary to manipulate the 3d object which it controls (via hopefully the new Entity framework in Unity).   There has to be a bidirectional message pump between the browser process and the emissaries--the emissary requests a change and the data gets passed to the browser and implemented (or it requests a callback to return info back to the emissary).   I need to see how GRPC has progressed in the last few months.  Perhaps the windows GRPC via named pipe has been implemented in .NET and we can have very fast communication for these message loops.

I'm also seeing lots of nice API documentation on github.  Need to work on that for the new code.

Found a Named Pipe implementation for GRPC.

Progress with WasmTime

After holidays and bad colds and work trips, I'm starting to make lots of progress with WasmTime and with WASI.    I've started a new emissary common class, which I am getting ready to experiment with.

I am now at the point where I have to figure out how a emissary works inside (things like how to initialize it and call it, and feed it events).   So far, with WasmTime I think I have to generate a single module per webassembly file in memory (likely to change over time as wasmtime is in steady motion).  This of course was far from possible with V8.  I'm hoping that modules are light enough that we can have thousands, if not eventually hundreds of thousands, of these at the same time (though that might be 5 to 10 years from now -- I think computers of the future will be able to cope).

I also have kind of figured out how to handle WASI security.   There are several kinds of emissaries, and they need different APIs, and the concept of the IHost at this point just lets me define one.  So, I will need to put all the APIs together.  I can however put a simple, "Is thing thing on?" check to disallow calls that aren't for that kind of API.

I've started a big spreadsheet to start figuring out the actual methods I need for all of the different APIs.   And since I've expanded the idea of the emissary to basically encompass all most all of the VR business logic (if I can refer to it as such), with be done via an emissary.

And, since it seems like it's first the to gate, initial libraries for webassembly will be done in Rust.  Though I suspect most all languages will get there eventually.

I'm also thinking of possibly using protobuf to pass the initialization and perhaps some of the more complex objects between emissary and processor rather than trying to use regular objects.  I think I now need an Common Emissary Protobuf (and possibly gRPC) library, and likely a separate github repository so they can be git submodules.

I'm also starting to consider the use of rabbitMQ (AMQP) to do more event oriented work.   I can even install rabbit (and erlang) onto a local windows  machine.   I'd have to figure out the demarcation between gRPC and AMQP.  Though likely the AMQP messages will be protobuf.

I'm starting to question the windows-centric deployment of the browser -- this is likely required, though eventually I might be able to do hybrid deployment in docker on windows-10.  This also might be a good solution for browser clustering.   It certainly would be an easier way to deploy the messaging (amqp), and possibly the various non-ui servers (emissaries, and caches).

WebAssembly and Blockchain Happenings

I am jazzed by the appearance of CraneLift and WASMTime.

CraneLift is a JIT webassembly compiler, and WASMTime, which is a complete sandboxed environment. However, it appears to be for Non-Web Embedding. Which is what I've been waiting for.

CraneLift will probably be the basis of our toolchain. 

There's also a nuget module for WASMTime that can be called from .net (though I might have to change all my external modules back again to the latest .net core rather than framework 4.8). This might be much simpler than trying to run V8 (via Microsoft ClearScript) -- and much more portable. I also don't feel any great loss from the not having a Javascript environment.

There apparently is a gRPC/protobuf implementation now for  Rust (if the first emissaries need to be written in Rust).

I'm making slow (very-slow) steady progress with the Emissary managers. The browser launches them and they start to do their startup handshakes. However, the next steps, which is mostly being able to log into the various Kudo Servers is going to take some effort. I have the stubs of the new Kudo Authentication engine started.

CraneLift might also let me build server side emissary modules.

I guess I'm going to have to learn Rust after all.


Starting to look at blockchain infrastructure like Corda. The true functioning of the Kudo server is basically a blockchain engine, or at least what it does would benefit from a blockchain journal. A true Kudo (like a game level or a certification) would benefit from blockchain auditing. Nothing much more useless than a blockchain which has an audience of one (like a stand alone system). It's only benefit is being shared far and wide.

I don't think Corda is .net (may be Java/Kotlin). But, it dockerizable, and has an API we can call.

Considering IdentityServer4

Got a bit of organizational work done over the weekend at Orycon.  I'm mostly trying to map out the intra-browser communications via GRPC.   The browser can start up and launch it's Emissary processes.

I'm seriously considering making IdentityServer4 a big part of the Kudo server since it seems to overlap with what it was supposed to do.   There's still a lot to the Kudo server separate from authorization/authentication, but this seems to be a big chunk I don't have to write.

Embryonic Browser Starting

I'm getting the basic browser put together.   Getting all of the thread managers for the Emissary controller projects.   I think I've got the gRPC set up on the browser (on the Unity side, this is the hard part).  I'll get the basics together to launch the basic processes and get a two-way ping going so the Emissary Processes can die when the browser stops (or they'll just pile up).

I think I've even got a good handle on initial security.  I am only doing HTTP/2 on localhost.  I'm not planning on other security yet.  The browser will generate a random token which it will pass to its child processes in the environment (not the command line).   That should keep other users from attacking the processes.

Having emissaries and other such things running on servers will have to wait until we have basically a VRWorlds server running on each machine in the cluster -- but I am planning the ability to have a browser cluster.

Initial Sketch of the Loading Area

Interfaces Defined

Not much progress in the last week or two.  My trip to Bubonicon though I managed to do a pretty creditable list of all the initial APIs.  So, I'll be putting this into a document and beginning the effort of defining all of the gRPC definitions for them.

I'm looking forward to having enough pieces to get the prototype browser working, with multiple processes and the ability to load in a working emissary.

In the meantime I've been deep-learning kubernetes by putting a large bit of my own production into docker, and porting some python from v2 to v3.

API Layers

Cranking up the old Visio again...

Ogres are like onions, because they have lots of layers...

If you see the crazy number of empty projects I've been checking in, you'll see now what I intended.   This drawing was also to give myself some more clarity of what I was just randomly creating as I saw a need.   Now the thing this doesn't show is what APIs (or parts of APIs) are implemented where.  I will work on this part next, so I can start to get a full skeleton of what the API trees are going to need to contain.

I also saw the need as I was writing for a provisioning API (how many servers are where--mostly interacting with the Data Plane and the Orchestration (though their data is stored in etcd).   Also most of the server types will initially need a conventional web page for signing up and utilizing.  Someday that will all get done in VR.  Ultimately there will be some kind of Dashboard -- again someday visualized in VR.  For now, I'll find some conventional monitoring tool.

I also like that I've have some good structure happening on the browser.  I think that I know how that's supposed to be built.   Also, I've mostly figured out out how its Control/Data Plane is going to work (ProtoBufs over Named Pipes--though I wish I could just use gRPC).   I also had an amusing idea--no real reason Named Pipes wouldn't work across multiple machines if you want to turbo-charge your browser.  Though also should allow conventional sockets so your browser-assist can come from a Linux box.  Also probably should consider building a Unity Linux VR browser, so need to be agnostic on this part of the infrastructure.  I could make co-processor boxes that you attach to your regular machine.

Anyway.  Little steps happening in the background.

Browser Construction

I'm starting to make a push now to actually get a browser foundation started.  My current issue is .net frameworks.   My common libraries, and gRPC schema libraries are all built in .NET Standard, but to my knowledge now, there are no such versions of ClearScript, which I'm relying on for my V8 embedding. 

So, experimenting, I set the browser compatibility to .NET Standard 2.0, the thing starts up (after doing lots up updates to unity and the packages), with ClearScript installed with NuGet--even though it's a .NET 4.5 assembly--and I'm not yet instantiating the thing or doing anything with the V8 engine.   That's probably next.  Plan is to make a nice generic interface which the libraries can call to get attached to the V8 (which should be done rather formally to assure security and robustness -- and lets me change out the V8 module as needed).

Though, I've been thinking again.  That's the trouble with not hitting "publish".   I was looking at how the Chrome browser talks to itself across it's different processes...  It does it via their own IPC via named-pipes.  I was wondering how easy this would be with gRPC.  Maybe I can just have my own C# executable running there communicating this way.  It would help with scaleout really well.  Anyway, continuing experiments building a browser foundation. 

Server-Side Embedded Scripting

I'm starting to look very hard the corresponding server-side analog to the Emissary, the Coordinator, or Implementer or some other $10 word to be introduced in the future.

I'm pretty much committed to writing my own V8 wrapper on the browser --- I really don't think I have much choice there, but I'm not sure I need to go that way on the server side (though I did discover that the v8.net wrapper for .net seems to run fine in .net core).   Though I think I can use MicroSoft ClearScript on the browser side.  But on the server side I don't have to do this.  I've been looking at alternatives and node.js is pretty much out.  It's just kind of hot mess sandbox-wise.  It assumes it has full user-level system access, like any piece of code on the metal would.

However, I am looking very hard at deno.   It has the ability to be run as a secure sandbox (you must enable network and disk reading/writing explicitly), and I think I can extend it to do what I need to.   It was scale up and out as needed in the cloud.

Basically, it needs to be able to load and implement gRPC (and protocol buffer) bindings to the Emissary.  Also I think there needs to be able to load multiple "Coordinators" in each V8.  Maybe it's light enough where I can limit a v8 to only one manufacturer.  I know I can't get a way with this on the browser side just yet.  Anyway, it has to be able to call the API for that server level to implement the business logic for that type of server (kudo, world, entity, avatar--also world and avatar servers are also entity servers depending on their role in the scene).

Also, it does seem like it works just fine with wasm, which is where I ultimately want to go.   They're making lots of use of typescript, which is also where I want to go (if I must do JavaScript like coding), and with Rust, and so maybe I want to look harder at that.  I may have to write the bindings in Rust, but we'll see.

...

The more I look at this, it may be impractical to directly add what deno calls "bindings".  Perhaps this can be done via gRPC or AMQP with the network layer turned on.   In this case, each docker images would be pre-created and perhaps constrained with iptables, since we should know what code we need to handle -- we can roll this out and handle versioning by just shifting to a new docker and have the old one stand down after it goes idle.   At least we can start with this.   The API layer should be nicely abstracted so we can change how it works.

Looks like I seriously need to up my TypeScript game.

...

And after even more looking, deno isn't quite ready for prime time yet.   I think for initial implementations, I'm being a little paranoid.   Ultimately if it implements the API it can be run, and it can be sandboxed pretty well in docker.  Be it in node.js, or in python, or whatever.  True sandboxing perfection we can work on over time.

How the Servers Work

So, I've been thinking somewhat on how the servers actually are going to work.  I had though to them initializing and handling their security and bringing up the web services that happen at the top level (the fixed gRPC channels).  Though, except for the example content, I had always thought that implementors will simply add code to the system. 

I'm not thinking now that that's going to be best way to do this.   I've started thinking of the Emissary idea that I had for the client, and am thinking of possibly extending that to the server side.  I was never thinking that the server needed to be sandboxed like the client would, but maybe it would make sense to have a similar technology on the server (which as a starting name, I'm calling the Coordinator).   Basically the Coordinator would handle the server logic for one or more client side Emissary.   The servers would handle a lot of types of infrastructure natively that the Coordinator could take advantage of and make calls to though the APIs.

This all depends on the V8 engines and the Microsoft ClearScript engine working in .net Core (though this is where everything is going, it's likely to work).  The server would be a lot more capable of handling multiple servers and V8 threads to handle some scale-out on the server.

Even with an engine like this, other parts of the server should be written in a plugin-friendly way.  Where a set of C# assemblies that conform to the proper interfaces can be consumed by the server and loaded up at runtime.  The thought runs, that, for example, the world-persistence layer might be written to use Mongo, but a different persistence layer might be added which persists in, say, MSSQL.  So, have to think ahead about what things might need overloaded functionality which could be pluggable.

I've learned a little bit more about C# on webassembly.  What I understand Blazor does is that there's basically a webassembly version of an IL interpreter.   So C# assemblies are directly consumed by this interpreter.  C# is not compiled down into webassembly yet.  Still that might work.  It may not be the most natural language for this though.  Go and Rust are still in the running.  I'm obviously already ready for Non-Web Embedding, but I'm not sure what's been done for this so far.

Basic PKI (VRWorlds-CA) is feature-complete

Basic high level PKI is now functionally completed.

VRWorlds-CA basically seems to do what it needs to do.  It creates and manages root certificates and intermediates/signer certificates and exports encrypted files which can be used by the kudo servers (security servers) for VRWorlds and encrypted backup archives to keep offsite.

Lots of testing left.

The intention was that critical info should be encrypted, and that all communications should be using SSL and I didn't want to leave that to last, so i did it first.

Now, onward to building the server side components to assimilate the data produced by the CA.  Also need to work on rearchitecting the existing VRWorlds-3DOM library repository into basically a set of server common libraries, and a browser library and the beginnings of the basic scaffolds for the 4 backend server types (Kudo, World, Entity, Avatar).   I already have pieces of these scattered here and there and they all need structure. 

All of the server parts are .net core and the browser is whatever is needed to run in Unity.

Working on first hello-world functionality, which is to log into an Avatar and have it materialize onto a plane with various cubes and spheres in scene, whose movement are being manipulated by client-side emissary code.

Don't Use RSA

Interesting article on asymmetric cryptography...

Don't Use RSA

So, looks like I will be using Elliptic Curves with Curve 25519.   I'm not exactly sure how well TLS supports this.  I know that SSH does.  We micromanage the connections between the browser and t he server, and between servers, so likely they'll accommodate it.

That's why this whole subject is fraught.   I do hope I can get some real experts to look at all this while we're in the experimental stages.

CA Progress and Nuget

I've been making steady progress with the VRWorlds-CA project.  The idea is to build a CA/Cheap-HSM out of something like a Raspberry Pi, which would be Air-Gapped and kept in a safe.   The system would generate an encrypted archive which would contain all of the root certificate info (though not the root's private keys) and the intermediate signing certificates which would get imported by the kudo servers via a thumb drive, and encrypted offsite backups which would be kept in a secured location. 

I've got the whole flow down and determined what classes and methods are needed and what they have to do.  I just need to fill in the blanks now.

I'm looking at uploading the basic libraries onto nuget, so I can simply import them into the browser (in unity).   I've got the current VRWorlds-3DOM building in .net-core mode (which is how I want the servers to run so they can run in Linux docker).  But, Unity will need probably .net 4.6 -- I'll have to play with that.  I think it's easy enough to build multiple targets and nuget must have a way of dealing with this.

Data Plane

Seems like lots of big system are going with the new infrastructure components, the control plane and the data plane.   Haproxy now has a nice idea for a data plane API.

Anyway, I thought this was interesting.  We might look at this.   Heading out to spikecon this week, hoping to get some good work done on this project.   Working on the CA beginning infrastructure so I can actually get it so you can connect to the servers and log in your avatar.

Cache/Config and Message Infrastructure

So, as I add on infrastructure I think I'm going with the following pieces, they should be deployed in a high-availability cluster configuration.

• Configuration information - ETCD
• Possible secrets storage - via Crypt, or possibly Vault
• Caching - Redis
• Messaging, Events and Subscriptions - AMQP via RabbitMQ

EtcD and Redis really do have different purposes and so it actually makes sense to have both.   I've used Redis, so I know it, as opposed to MemCacheD.  Etcd actually used gRPC internally so it kind of makes sense too.

I do need a place to store passphrases and keys.   This will likely go in EtcD, though encrypted themselves.  I'm not that fond of Vault, but I haven't used it in a long time -- I used an early version.  Once I get the PKI infrastructure working, I need to put it somewhere.

Kudo PKI infrastructure

So, now that I've gotten the basic infrastructure together for building our microservices with gRpc, I thought I'd sit down and start thinking about how the PKI and GUID identifiers work.   So, I cranked up the old Visio.

VRWorlds PKI and master Identifiers

Each manufacturer (entities doing VR work), has a single cluster of Kudo servers, any number of World clusters, and possibly one or more Entity and Avatar Forges with their attendant cluster of servers.   Worlds almost always have a dedicated Entity server to support it, but you can create Entities outside of Worlds too.   I envisioned that Avatar manufacturers would primarily be in the business to do that, but I can also see having Avatars, or at least Aspects (alternate bodies you can switch into), which were customized for a particular world.

I was thinking, at least initially, to build an inexpensive Raspberry Pi based HSM (which I'm sure makes any real security professional cringe and swear).  At the very first, this code will be run by hobbyists and developers.   Later this might become more complex and involve money and at that point, real security infrastructure and professional fiduciary care will need to be employed.  I want to try to build a good infrastructure behind the scenes to provide for future needs.   As I am just a humble developer working out of his metaphorical garage, and any code I write is provided "As-Is", with no expectation of real cryptological rigor. 

Anyway this PI would be kept air-gapped and be kept in a safe with a backup when not in use, and would generate and keep the root certificate and the signers, including future signers--likely using a thumb drive to transfer the data to the server (another cringe is heard).  I was thinking that keys would have a relatively short span -- a year or less, and perhaps keeping some power of 2 (4-16) actual intermediate signer certificates.  This reduces the metaphorical Blast Radius if a key needs to be repudiated.   Also they might be projected out a year or two into the future.  This way, a client will be able to prepare and handle the rollover transition from one certificate to the next as they expire.  That way they know the new public key is the same owner as the current one it trusts.

I do have some odd ideas when it comes to PKI and am adapting patterns from TLS and the web to other realms with some impedance mismatch between them.  But, I am not so stupid as to think I have the intelligence and skill with crypto to invent and build my own tools--that is the ultimate fool's road paved with good intentions.  I do want to use and understand what are supposed to be the best tools available for open source.  I also want to try to give a lot of attention to poking holes in my own infrastructure to make sure it's as strong as I can make it.

One of my conundrums is what happens when someone tries to poach/replay your GUIDs in their own fake server.   I think ultimately these will have to be combined with the public key and not assumed to be universally unique--though they're intended to be.   A kudo server can securely verify that it owns a public key.   Perhaps all 3rd parties must identify a server by a combination of the public key (or a uuid4 version of it) and the actual manufacturer/world/entity/avatar GUID--with the understanding that the public key rotates as it expires.   Initially I may even want the certs to expire every few days to force this rollover code to be well-tested.

The other big thing I want to start thinking about is how I will keep the secrets for this system.  At least things like keys should always be encrypted at rest and transit.  And of course, passphrases would be the same.   Kubernetes has a secrets infrastructure and docker sort of does.

Anyway, I'll start putting structure around this.  I'll likely prototype my hack-Pi-HSM (bash scripts and python are likely) and just run them with no expectation of actual security for now.  I am committed to all real communications with the servers forcing TLS on day one.

Next steps are boiling all this down to use-cases and working on what services need to be built were so I can start sketching out the servers.   All servers need to authenticate to their kudo server and come online.   Browsers need to log into the Avatar's kudo server.   Getting the avatar logged in would be a great foundation -- we can start dealing with meshes.