Random Thoughts, Some Progress

 Lots of work at work, so not as much time for projects.

Smattering of thoughts:  

1. Godot doesn't really seem ready for prime-time with c# yet.  Not bad, and we will keep and eye on it.   I'll try to make my libraries in a way that theoretically we could make them agnostic to the engine (probably biggest issue there will be the way the engine does mesh representation), and hopefully as little code as possible on the engine side.
2. Keeping a close eye on wasmer/wasmtime.   I think I will still need wasmer, since it is .netstandard 2.0.  The Wasmtime restriction to .netstandard 2.1 is a problem with unity (and probably godot).   Though I can run the emissaries as separate processes, I think I want to do that later.  Let's get it working before we go too crazy with a 'scale out' plan, just not preclude ourselves from going that way.  I have very little time to code, so have to make a bee-line to functionality.   We will probably revisit this when Unity changes their .net model, to .net core or possible .net 5 (the .net version to rule them all).  Both of these projects are ticking a long.   I still find wasmer's WAPM projects a powerful missing link, though version pinning and signed downloads of external code would have to happen.
3. Starting to look at some naked-codeless emissaries, which just have content in them.  It's one option for the emissary, just to hold objects, or to hold references to objects.   I'm refactoring the ldraw code so I can just put lego entities in.  Later these can indirect to an URL rather than needing to be in the emissary--though they will have to match a full checksum when they download--they will then be able to be proxy signed.   Looking at getting an STL loader too.  Then some work on a universal (protobuf based) mesh format.
4. Very intrigued by Kafka.   I think log based messages might be a giant win over Rabbit/AMQP style pub/sub.   The use-case is that the world server can simply make every client a producer and everyone can be a consumer to see their scenes change.   The interesting thing is that perhaps, if you don't have a complete cache of the scene, you can then traverse the log in the reverse direction back to a fixed low-water mark (which represents having every entity having presented it's status in the log at least once), fully recapitulating the status of the scene.  You have to traverse it backwards since a given entity might report many times, and you don't want to have the scene be remade in order.   Physics will have to be suspended during the repaint.  Also, if  you mark each status with the timestamp of the last previous change, you can make a hash to ignore all but the last status report of every entity (and not have to keep inactive entities in your hash -- if their last-changed is back before the low-water mark.   Also Kafka messages can be done in protobuf (a supported standard).    World server will still need a database to track everything.   Might be easier than the redis idea.  Have to experiment.  I need a skeleton world server.  I'm building a Kafka server to try out.
5. These log based systems like Kafka (and really like the mongo oplog) are intriguing.  I'm thinking of a way to do the image and mesh caches that use this, where they use a given number of fixed length cache files (likely memory mapped), and the oldest one times out and causes the oldest cached entries to be reloaded into a real file.   This doesn't fix evacuating trivial files or keeping core-files which are constantly used. Trivial files of course will fall out when the logs get rotated.  It's a very powerful scheme for keeping high-integrity databases without needed to do wild random access during setup, or any need to do garbage collection.

Doing some Rethinking

So, I've been progressing somewhat with Emissaries...   

But, I have been doing a little rethinking.   First, if I go with Unity as the browser platform (as I have), it might make sense to look harder at Wasmer (github), rather than WasmTime (github).   Wasmer is available in .net standard 2.0, which will permit it to run directly in Unity, rather than requiring me to build out the multi-process architecture initially to make it so that I can call WasmTime WebAssembly via a separate Emissary process.   This would be much easier as an initial effort, and would make is possible to have Entities that can directly interact with the browser engine.   I'm also very interested in the WAPM effort, which is the WebAssembly -Package Manager--inspired by NPM.   I think this is basically a linker, possibly even a dynamic-linker.  It's something I thought we'd need sooner rather than later...  Ultimately we want WebAssembly modules to act like shared libraries, so that only one instance of a given module need to be in memory on a given emissary server.

However, an even bigger and more momentous change, I'm starting to look at Godot games engine.   It's not nearly as sophisticated as Unity (Unreal doesn't seem like it's any kind of fit for what I need).   Also, I'm essentially building what Unity calls a "platform".   This isn't something you're really supposed to do in Unity, though since it would be open source and would probably not really be charging money for it (at least directly, I would like to be able to monetize it somewhat---consulting, hosting, or something Patreon-like), I may or may not get into issues.   It also might be something Unity would be interested in sponsoring if this became popular.  You never know.

But, Godot is MIT license and includes all the sources.   It seems like it's got a good following, and a small but very talented core team.    Exposing the Godot API into wasm might even be something the Godot group would be enthusiastic about--it would be the holy-grail of scripting, not being tied to any particular language--though VRWorlds requires more sandboxing than that.  Perhaps the compromise there is to permit the sandbox to be created, and what APIs are exposed to it are done at creation (some level of WASI and Godot API or VRWorlds APIs can be specified).   As WASI embeddable WebAssembly capable languages start to become more plentiful (again, that's pretty much just Rust, C, C++, and maybe Javascript for now), this might become very attractive.

Godot does support procedural geometry, which we need since we don't really use the games engine to directly manage our content--it gets loaded with the emissaries.

GD script seems appealing.  I really enjoy python (though I  despise JavaScript).    It might even be worthwhile to dispense with c# and just do the interface code in GD and embed wasmer or wasmtime in c++ (theoretically it can also be done dynamically in GD script)--c# is apparently a brand new feature.   I can also put it into the engine myself, since all of the code is on GitHub.   So, I will be soul-searching and learning Godot a bit more.   It's possible that they might grow up together.  Obviously a smaller open-source project is riskier than a big commercial effort like Unity, but perhaps it will turn out somewhat like Blender did.

I can write modules directly in C and C++.   This might permit a lot of the frame-buffer operations I want to do to be very fast as opposed to having to write lots of unsafe pointer code in C#, though it's possible to call C/C++ from C# with interop.