So, I've been pretty deep in WASM architecture, and the WASI APIs. I've pretty much gotten to the point where we can launch an emissary and interact with it. So, now, it's time to actually build out the 3DOM APIs for interacting with the emissary and the message passing loops. Initially I'm going to start building a Rust Module for this. I do want to do this with other languages as it becomes practical. I'd love to be able to do C# and Python, but I don't think they're quite there yet. I've used mono-wasm to generate c# wasm, and pyodide for Python, but so far they're not particularly oriented to WASI yet and are more intended to run on a website (and interact with a DOM).
I think it will mostly go like this: one exposed setup method from the API layer, which will receive a protobuf buffer. This will be using 'proto 3' and will be decoded and processed. This gives us a backward and forward compatible startup. The startup will register callbacks on the rust side for the message pump to process. Then the emissary's 'main' basically returns. A channel is created via grpc to the browser and eventually to the entity/avatar server. Protobuf packets are passed in via these callback mechanisms. The emissary runs and then returns and possibly returns one or more protobuf packets. For now, I don't intend the emissary to have a thread to run when it is not processing a callback. Initially I'd rather an emissary not be able to do things like mining bitcoin or other such undesirable activity. The service managing the emissary will probably have a call timeout and will disable it if it doesn't respond in a timely fashion.
This facility kind of makes it hard to directly do gRPC, but at least we can do gRPC-like behavior passing protobufs around. I can cheat and have a grpc wrapper call which contains a command type and the protobuf as a raw vector. The server calls are tricker, as the interior emissary-to-server gRPC calls are theoretically private (and ad-hoc)--so their formats can't be known by this middleware layer which is operating the emissary for the browser. There are some up-and-coming grpc proxies out there. Or I will have to do my own, which is just calling methods on the emissary and returning an answer (no actual network involved). With my WASI layer, I could actually permit the emissary to open a socket, but I don't trust this yet.
So, now I actually have to work out exactly what the browser-emissary interaction (which I've been calling the 3DOM api) actually is and how it works. I've got some prototypes of this that I worked out last summer, but now I have to start building my .proto files to implement it.
Initially there will be no world, and no entity server to work this out. I'll have the concept of a "system entity", which can simply run inside the loading area (which will eventually will become a mini-world which is served from inside the browser). Also, eventually we'll have an avatar server, but initially the avatar is mostly just a vr camera FPS rig, with no smarts. I'll be doing some checkins soon of some of this structure.
The emissary will have an initial built-in parasitic overhead to have to have a copy of the loading module and the protobuf module. Eventually I'm sure I'll be able to dynamically link such things so they can be shared between emissaries. There's an effort for a package manager for webassembly (like npm) called wapm. Perhaps there can be a package manifest and emissaries can be dynamically loaded. I am ultimately intending for there to be hundreds, if not tens of thousands of emissaries to exist (in the distant future). So optimization, sharing, hot-deploy, and versioning will have to be seamless.
Showing posts with label 3DOM. Show all posts
Showing posts with label 3DOM. Show all posts
Sunday, May 3, 2020
Sunday, April 5, 2020
2020 Has Eaten My Brains...
Wow. I've had a busy several months. I think about this project quite a lot, but have had little time. The SF conventions that I normally spend coding on this stuff have been cancelled. Alas Norwescon...
Getting the First Person framework working on the loading deck. I've got the VR integrated, but needing to get all the movement stuff working. Unity is having problems with the SteamVR plugin with some of it's more recent versions. I'm trying to run on 2019.3, but still have to remove several of the integration methods to get it to compile. I'm going to try to get teleport movement working. Supposedly Valve is going to write a new version of steamVR (or OpenVR--not exactly sure). This might limit me to the 2019 version of unity for a while. I think--rereading all this and updates, that I need to start using the XR framework.
Keeping an eye on Wasmtime. They're doing good work there. There's a new nuget preview. It looks like if you look in the sources though that they're changing and simplifying the calling structure. It looks like there's going to be a much easier way to integrate the private APIs I need to expose to it via private WASI. You can just call a method to define a function. So I can start getting together some classes to autogen the APIs. It also lets me define different APIs and only load the ones that are appropriate (entities, avatars, and worlds likely have variations in what APIs are present). And eventually will start to implement the standard WASI interfaces that are appropriate (like file IO, but only to the sandboxed and read-only emissary filesystem). It's still looking like Rust for the first language for doing Emissaries, though I need to see how other systems are working. Blazor is going nicely in the dotnet/c# world, but it works rather differently -- There's a CLR interpreter running in webassembly -- I might look into this and see if this is sufficiently well documented to write my own CLR classes for it or if I'll need my own webassembly module (it may be assuming a DOM and other non-embedded things--though eventually, it might be worthwhile having a working DOM).
In the just-fun category, I had an interesting idea. Someone has an actual firmware VT100 emulator running on a virtual javascript 8080 cpu. This is not what is conventionally known as a "vt100 emulator". This truly is running the rom code from a vt100. It was be possible to build one of these into an emissary (I see some 8080 emulators in rust). I was originally thing that these emulators would run server-side and expose themselves in the VR environment via a pseudo-serial connection. So, it would be truly interesting to do this with the other SIMH emulators and Mame and libretro. These are mostly in C, it might be possible to get them into webassembly. Mame has some other VT emulators built into them as well as the million other things Mame can do. I hear rumors that there's an SDL layer that's been ported to webassembly (though probably for a DOM Canvas, but I can probably map that onto a texture).
Next big effort too is to get the various multi-process servers to bidirectionally ping each other and shut down when the browser exits. There a ping that occurs the when the service comes up to indicate that it's ready. I also had the thought to have some kind of indicator of presence and health on the loading deck, like a luminous sphere for each server. Color could indicate load or CPU %. It was a bit of an epiphany that I could start having simple emissaries that don't really require an external entity server, and can show them onto the loading deck. Getting all of the pieces together for loading external content is still a ways off. The initial avatar won't really need any emissaries, until we want to start showing a second-person view. Though we might eventually show hands and such in first-player view--though that might be initially hardcoded and rely on what is available in SteamVr.
I had a sketch of the necessary emissary APIs, so I will also start fleshing those out. Those need some practical hammering so we can come up with some nice coherent architecture for that. Mostly it allows the emissary to manipulate the 3d object which it controls (via hopefully the new Entity framework in Unity). There has to be a bidirectional message pump between the browser process and the emissaries--the emissary requests a change and the data gets passed to the browser and implemented (or it requests a callback to return info back to the emissary). I need to see how GRPC has progressed in the last few months. Perhaps the windows GRPC via named pipe has been implemented in .NET and we can have very fast communication for these message loops.
I'm also seeing lots of nice API documentation on github. Need to work on that for the new code.
Found a Named Pipe implementation for GRPC.
Getting the First Person framework working on the loading deck. I've got the VR integrated, but needing to get all the movement stuff working. Unity is having problems with the SteamVR plugin with some of it's more recent versions. I'm trying to run on 2019.3, but still have to remove several of the integration methods to get it to compile. I'm going to try to get teleport movement working. Supposedly Valve is going to write a new version of steamVR (or OpenVR--not exactly sure). This might limit me to the 2019 version of unity for a while. I think--rereading all this and updates, that I need to start using the XR framework.
Keeping an eye on Wasmtime. They're doing good work there. There's a new nuget preview. It looks like if you look in the sources though that they're changing and simplifying the calling structure. It looks like there's going to be a much easier way to integrate the private APIs I need to expose to it via private WASI. You can just call a method to define a function. So I can start getting together some classes to autogen the APIs. It also lets me define different APIs and only load the ones that are appropriate (entities, avatars, and worlds likely have variations in what APIs are present). And eventually will start to implement the standard WASI interfaces that are appropriate (like file IO, but only to the sandboxed and read-only emissary filesystem). It's still looking like Rust for the first language for doing Emissaries, though I need to see how other systems are working. Blazor is going nicely in the dotnet/c# world, but it works rather differently -- There's a CLR interpreter running in webassembly -- I might look into this and see if this is sufficiently well documented to write my own CLR classes for it or if I'll need my own webassembly module (it may be assuming a DOM and other non-embedded things--though eventually, it might be worthwhile having a working DOM).
In the just-fun category, I had an interesting idea. Someone has an actual firmware VT100 emulator running on a virtual javascript 8080 cpu. This is not what is conventionally known as a "vt100 emulator". This truly is running the rom code from a vt100. It was be possible to build one of these into an emissary (I see some 8080 emulators in rust). I was originally thing that these emulators would run server-side and expose themselves in the VR environment via a pseudo-serial connection. So, it would be truly interesting to do this with the other SIMH emulators and Mame and libretro. These are mostly in C, it might be possible to get them into webassembly. Mame has some other VT emulators built into them as well as the million other things Mame can do. I hear rumors that there's an SDL layer that's been ported to webassembly (though probably for a DOM Canvas, but I can probably map that onto a texture).
Next big effort too is to get the various multi-process servers to bidirectionally ping each other and shut down when the browser exits. There a ping that occurs the when the service comes up to indicate that it's ready. I also had the thought to have some kind of indicator of presence and health on the loading deck, like a luminous sphere for each server. Color could indicate load or CPU %. It was a bit of an epiphany that I could start having simple emissaries that don't really require an external entity server, and can show them onto the loading deck. Getting all of the pieces together for loading external content is still a ways off. The initial avatar won't really need any emissaries, until we want to start showing a second-person view. Though we might eventually show hands and such in first-player view--though that might be initially hardcoded and rely on what is available in SteamVr.
I had a sketch of the necessary emissary APIs, so I will also start fleshing those out. Those need some practical hammering so we can come up with some nice coherent architecture for that. Mostly it allows the emissary to manipulate the 3d object which it controls (via hopefully the new Entity framework in Unity). There has to be a bidirectional message pump between the browser process and the emissaries--the emissary requests a change and the data gets passed to the browser and implemented (or it requests a callback to return info back to the emissary). I need to see how GRPC has progressed in the last few months. Perhaps the windows GRPC via named pipe has been implemented in .NET and we can have very fast communication for these message loops.
I'm also seeing lots of nice API documentation on github. Need to work on that for the new code.
Found a Named Pipe implementation for GRPC.
Thursday, January 23, 2020
Progress with WasmTime
After holidays and bad colds and work trips, I'm starting to make lots of progress with WasmTime and with WASI. I've started a new emissary common class, which I am getting ready to experiment with.
I am now at the point where I have to figure out how a emissary works inside (things like how to initialize it and call it, and feed it events). So far, with WasmTime I think I have to generate a single module per webassembly file in memory (likely to change over time as wasmtime is in steady motion). This of course was far from possible with V8. I'm hoping that modules are light enough that we can have thousands, if not eventually hundreds of thousands, of these at the same time (though that might be 5 to 10 years from now -- I think computers of the future will be able to cope).
I also have kind of figured out how to handle WASI security. There are several kinds of emissaries, and they need different APIs, and the concept of the IHost at this point just lets me define one. So, I will need to put all the APIs together. I can however put a simple, "Is thing thing on?" check to disallow calls that aren't for that kind of API.
I've started a big spreadsheet to start figuring out the actual methods I need for all of the different APIs. And since I've expanded the idea of the emissary to basically encompass all most all of the VR business logic (if I can refer to it as such), with be done via an emissary.
And, since it seems like it's first the to gate, initial libraries for webassembly will be done in Rust. Though I suspect most all languages will get there eventually.
I'm also thinking of possibly using protobuf to pass the initialization and perhaps some of the more complex objects between emissary and processor rather than trying to use regular objects. I think I now need an Common Emissary Protobuf (and possibly gRPC) library, and likely a separate github repository so they can be git submodules.
I'm also starting to consider the use of rabbitMQ (AMQP) to do more event oriented work. I can even install rabbit (and erlang) onto a local windows machine. I'd have to figure out the demarcation between gRPC and AMQP. Though likely the AMQP messages will be protobuf.
I'm starting to question the windows-centric deployment of the browser -- this is likely required, though eventually I might be able to do hybrid deployment in docker on windows-10. This also might be a good solution for browser clustering. It certainly would be an easier way to deploy the messaging (amqp), and possibly the various non-ui servers (emissaries, and caches).
I am now at the point where I have to figure out how a emissary works inside (things like how to initialize it and call it, and feed it events). So far, with WasmTime I think I have to generate a single module per webassembly file in memory (likely to change over time as wasmtime is in steady motion). This of course was far from possible with V8. I'm hoping that modules are light enough that we can have thousands, if not eventually hundreds of thousands, of these at the same time (though that might be 5 to 10 years from now -- I think computers of the future will be able to cope).
I also have kind of figured out how to handle WASI security. There are several kinds of emissaries, and they need different APIs, and the concept of the IHost at this point just lets me define one. So, I will need to put all the APIs together. I can however put a simple, "Is thing thing on?" check to disallow calls that aren't for that kind of API.
I've started a big spreadsheet to start figuring out the actual methods I need for all of the different APIs. And since I've expanded the idea of the emissary to basically encompass all most all of the VR business logic (if I can refer to it as such), with be done via an emissary.
And, since it seems like it's first the to gate, initial libraries for webassembly will be done in Rust. Though I suspect most all languages will get there eventually.
I'm also thinking of possibly using protobuf to pass the initialization and perhaps some of the more complex objects between emissary and processor rather than trying to use regular objects. I think I now need an Common Emissary Protobuf (and possibly gRPC) library, and likely a separate github repository so they can be git submodules.
I'm also starting to consider the use of rabbitMQ (AMQP) to do more event oriented work. I can even install rabbit (and erlang) onto a local windows machine. I'd have to figure out the demarcation between gRPC and AMQP. Though likely the AMQP messages will be protobuf.
I'm starting to question the windows-centric deployment of the browser -- this is likely required, though eventually I might be able to do hybrid deployment in docker on windows-10. This also might be a good solution for browser clustering. It certainly would be an easier way to deploy the messaging (amqp), and possibly the various non-ui servers (emissaries, and caches).
Friday, April 26, 2019
On Meshes
Since most of the life of an emissary is to push reactive changes from the server into the manifestation of the Entity, which is mostly simply a group of meshes, I wanted to make sure there was a robust API for doing mesh manipulation -- somewhat beyond the functionality of Unity itself -- I'm looking hard at G3Sharp. Which seems like an awesome library, and it's Unity compatible.
I need mesh alteration, composition and eventually mesh additive and subtraction operations. So, some of this API will get wrapped and exported into the 3DOM APIs. Also, you can have mesh files in your emissary payload (in several formats, including STL, OBJ), which can get loaded.
This is also part of the tooling projects I'm contemplating, which I'm calling the Wood-Shop tool. I'm not much of a 3d artist, but I can craft pretty fancy 3d stuff in the real world, so I'm trying for a metaphor that's closer to working with wood and metal tools.
I need mesh alteration, composition and eventually mesh additive and subtraction operations. So, some of this API will get wrapped and exported into the 3DOM APIs. Also, you can have mesh files in your emissary payload (in several formats, including STL, OBJ), which can get loaded.
This is also part of the tooling projects I'm contemplating, which I'm calling the Wood-Shop tool. I'm not much of a 3d artist, but I can craft pretty fancy 3d stuff in the real world, so I'm trying for a metaphor that's closer to working with wood and metal tools.
Subscribe to:
Posts (Atom)