Kudo PKI infrastructure

So, now that I've gotten the basic infrastructure together for building our microservices with gRpc, I thought I'd sit down and start thinking about how the PKI and GUID identifiers work.   So, I cranked up the old Visio.

VRWorlds PKI and master Identifiers

Each manufacturer (entities doing VR work), has a single cluster of Kudo servers, any number of World clusters, and possibly one or more Entity and Avatar Forges with their attendant cluster of servers.   Worlds almost always have a dedicated Entity server to support it, but you can create Entities outside of Worlds too.   I envisioned that Avatar manufacturers would primarily be in the business to do that, but I can also see having Avatars, or at least Aspects (alternate bodies you can switch into), which were customized for a particular world.

I was thinking, at least initially, to build an inexpensive Raspberry Pi based HSM (which I'm sure makes any real security professional cringe and swear).  At the very first, this code will be run by hobbyists and developers.   Later this might become more complex and involve money and at that point, real security infrastructure and professional fiduciary care will need to be employed.  I want to try to build a good infrastructure behind the scenes to provide for future needs.   As I am just a humble developer working out of his metaphorical garage, and any code I write is provided "As-Is", with no expectation of real cryptological rigor. 

Anyway this PI would be kept air-gapped and be kept in a safe with a backup when not in use, and would generate and keep the root certificate and the signers, including future signers--likely using a thumb drive to transfer the data to the server (another cringe is heard).  I was thinking that keys would have a relatively short span -- a year or less, and perhaps keeping some power of 2 (4-16) actual intermediate signer certificates.  This reduces the metaphorical Blast Radius if a key needs to be repudiated.   Also they might be projected out a year or two into the future.  This way, a client will be able to prepare and handle the rollover transition from one certificate to the next as they expire.  That way they know the new public key is the same owner as the current one it trusts.

I do have some odd ideas when it comes to PKI and am adapting patterns from TLS and the web to other realms with some impedance mismatch between them.  But, I am not so stupid as to think I have the intelligence and skill with crypto to invent and build my own tools--that is the ultimate fool's road paved with good intentions.  I do want to use and understand what are supposed to be the best tools available for open source.  I also want to try to give a lot of attention to poking holes in my own infrastructure to make sure it's as strong as I can make it.

One of my conundrums is what happens when someone tries to poach/replay your GUIDs in their own fake server.   I think ultimately these will have to be combined with the public key and not assumed to be universally unique--though they're intended to be.   A kudo server can securely verify that it owns a public key.   Perhaps all 3rd parties must identify a server by a combination of the public key (or a uuid4 version of it) and the actual manufacturer/world/entity/avatar GUID--with the understanding that the public key rotates as it expires.   Initially I may even want the certs to expire every few days to force this rollover code to be well-tested.

The other big thing I want to start thinking about is how I will keep the secrets for this system.  At least things like keys should always be encrypted at rest and transit.  And of course, passphrases would be the same.   Kubernetes has a secrets infrastructure and docker sort of does.

Anyway, I'll start putting structure around this.  I'll likely prototype my hack-Pi-HSM (bash scripts and python are likely) and just run them with no expectation of actual security for now.  I am committed to all real communications with the servers forcing TLS on day one.

Next steps are boiling all this down to use-cases and working on what services need to be built were so I can start sketching out the servers.   All servers need to authenticate to their kudo server and come online.   Browsers need to log into the Avatar's kudo server.   Getting the avatar logged in would be a great foundation -- we can start dealing with meshes.

Milestone Accomplished - Ping Embryonic Kudo Server

Last checkin manages a ping.   I wrote a tiny client which calls the ping servers on the server via grpc with protocol buffers--proto3.   So, that's nice.  Hello Kudo Server!

I need to now build a real client and get several of the basic calls done as well as at least the Kudo, World, and Avatar.  Though might as well do the stubs for Entity too.   I also need to build myself a CA (certificate authority).   Initially I'll just build them via normal command line (or one of the simple command line CA packages).  I want the services to be SSL right off, and handle a proper authentication.   These can get sucked into a mongo database so the Kudo server can do the right thing.

Eventually I will build a Raspberry Pi to manage root certificates and generating the signing certificates.  Was thinking it would emit needed data onto a thumb drive.   This Pi should be kept in a safe.

Flatbuffers too new for C# gRPC?

I think I may be stymied by the current version of Flatbuffers.   I find it slightly odd that it doesn't support arrays in structs (though I've seen proposals to fix this).   Also you can't set it to mark particular interfaces as private or internal so you can force encapsulation.  However, I get by all this.

My biggest problem is that the gRPC template generator for Flatbuffers doesn't work with C#.   It works with Java and I could probably generate those and then port them over, but I haven't got that kind of interest in messing with it.

Flatbuffers seems like a huge win in performance over everything else (except maybe Capn'Proto -- I'll have to see how it integrates with gRPC).  However, I might switch over to protocol buffers for a while, at least to prototype with.  I might also think about attaching the serialization layer to the rest of the system via a batch of interfaces which might make the serialization layer pluggable.

Next Steps After Emissary Builder - Microservice Stubs

So, have been proceeding with next steps after the Emissary Builder is prototyped.   I've decided that the server/browser communication will proceed with C# .NET Core MicroServices running GRPC utilizing the Flatbuffers serialization layer.  I've been working on figure out how to do this and how to build using these parts.   I'm assuming that the microservices will be ASP.NET Core, but I'm not entirely sure yet. 

So now, I'm starting to break down the functionality so I can start to get a handle on the MicroService boundaries, and how to create the authentication and eventually build all of the Flatbuffers contracts and RPC calls. 

I will need to start working on the emissaries themselves.  I'm thinking of doing that in TypeScript or possibly in Rust or Go in Webassembly (really, WASI).   I really am not sure about the state of C# webassembly, blazor notwithstanding.

I have to restructure my 3DOM library, which is where all my current active code is.  It needs to be strategically split into pieces, but I'm not sure where the edges are yet.  I'm trying to avoid monoliths.

Here's my brainstorm of the various services I know offhand I might need.  I'm sure there are many more, however, these are the main published services.   Each World, Avatar, and Entity can create its own layer of custom gRPC calls to handle the provisioning and operation of each emissary and they must remain dynamic.

I think--Stage 1 Hello World--will consist of all 4 servers working with their Ping servers.   Though that's not really how the use-case goes. 

Ultimately the completed startup is more like this:

• Browser asks Avatar server where Avatar's Kudo Server is
• Browser logs into Avatar and Aspect
• Avatar's Emissary is Marshaled
• Browser connects to world that Avatar wants to materialize in
• Browser asks world's Kudo server to grant permission (based on Visa or open permission)
• World's Emissary is Marshaled
• World then emits the inventory for the scene
• Entities from everywhere begin to Marshal their Emissaries
• World causes the Avatar to materialize in the designated position in the Scene

But, we have to get there incrementally.  Server skeletons first, then ping.   Enough of a microservice that we can get this running in a Kubernetes Cluster, or at least in Docker on one machine. 

Boy, that was ramble...

Tentative Services

Kudo Server

• Ping
• Locate Worlds
• Locate Avatar Service
• Locate Entity Service
• Vouch Self
• Vouch Service
• Vouch CA
• Validate Login Ticket
• Sign [Developer]
• Login from Server [Server]
• Login from Avatar [Avatar]
• Banking Transactions

Kudos:

• Vouch for Kudo
• Generate Kudo
• Transfer Kudo
• Revoke Kudo
• Manipulate Values in Kudo

Standard Kudos:

• Deed
• Lease
• Visa
• Access Grant
• Ban
• World Level
• World Coin

Avatar Server

• Ping
• Locate Kudo Route
• Vouch Self
• Marshal Avatar (As Primary Avatar)
• Marshal Entity (Avatar as Entity)
• Transform Aspect
• Manage Active Inventory
• Transfer Entities Into/From

World Server

• Ping
• Locate Kudo Route
• Vouch Self
• Log Avatar In
• Marshal World
• Unveil Scene Inventory/State
• Marshal Entity (World as Entity)
• Transfer Entities Into/From

Entity Server

• Ping
• Locate Kudo Route
• Vouch Self
• Marshal Entity
• Transfer Entity
• Transfer Entities Into/From
• Instantiate Entity Into

Emissary Builder Prototype Done

I've got the basic emissary builder done in c# in the 3DOM library.

So now, there are a ton of things to do. 

• Start the browser
• Need skeletons and initial functionality for all server types
• Kudo
• Avatar
• Entity
• World

I need to figure out what rules are for putting Unity projects into github.    I know you're not supposed to post Unity code so I've basically got to do it with my own code or with pieces that I download from github, not from unity.  I probably need a really good .gitignore file.

So, the next thing will be to get enough pieces together so that I login to a world, and start loading up the world and then I can marshal all of the emissaries as they come down.   I have envisioned the hello world state for the browser that materialized an invisible avatar in the middle of a scene with a couple of cubes on a plane with all of the particating servers supplying data.

Onward and upward...

Client/Server Protocols

I've been thinking for a while about how server/client communication will need to occur in several different directions.

This front-facing use-cases are the APIs to the various servers (Kudo, World, Entity, Avatar), and have thought that these need to be rather high-performance.   I've been looking to newer alternatives to json and xml and all of the old paradigms for web-api serialization, so I've been looking at things like Protocol Buffers, Cap'n Proto, and FlatBuffers.   The first and third are Google supported.  I'm also interested in the meta level of having an RPC layer like gRpc, and this supposedly can use ProtoBuffs and FlatBuffers.  Cap'n Proto also has an RPC functionality (though not that I can see for c#).

Many of the API calling code will be built into the browser and so interfaces would be totally static (though versioned probably).

The other big client-server interaction is between servers and the Emissaries...   Much of the design of this architecture is inspired by the DDP model in the Meteor Framework.   However, I think the kind of information that gets synchronized between the server and the Emissary is a little different.   Simply doing a two way reactive subscription to a chunk of a Mongo collection isn't the best model for the Emissary.   Reactivity: yes, Synchronization: yes, but the data is more specialized. 

And, for the Emissary, most of these protocols have a schema language and generates classes that you use in the tool-chain to build the emissary.  These can't be compiled into the browser, and so have to run in the Emissary code with probably some high level communication -- perhaps passing the websocket handle down to the code.   So, likely there will be some specific hooks in the APIs for this code.

I'm still looking for a sync model I like, though due to the nature of the way the Emissary is setup, you really can roll your own if you like as long as the API is giving you enough rope for you to pull on.  And this is obviously one of those points where security is vital.  I think the idea was that the browser would set up the secured channel, and then the emissary can do what it needs.

I'm getting to the point where I need Hello-World functionality from all of the basic servers, so I need to get the infrastructure ready to go.   I need some very basic functionality from them as well.  I need World to get me an inventory, then I need to marshal all those entities from an entity server.  I can skip some of the security stuff in the Kudo server, but I'd like the Cert Chain for the Emissaries to be checked (right now, it will simply accept the cert I give it -- it has to be signed, but it makes no checks to verify who signer is and that the cert chain goes back to the Manufacturer Guid (and that the signer is authorized to sign for it)).

I'm also thinking I may need a "Server Common" project.   Simple "Common" might include stuff that the browser doesn't really need.

On Emissaries and Meshes (Idea for Mesh Cache)

So, the original idea for the _payload directory in the Emissary was to allow the initial state of meshes and whatever other assets might be needed for the Entity to operation.  Problem with that is, after the mesh moves from its initial state, that part of the Emissary file now becomes unusable dross, weighing down the file and requiring it to be transferred each time it is cached. 

I'm thinking instead, that what is needed is a separate cache database (probably in sqlite) which contains the various meshes in whichever states are needed, and to have a separate way to download them to the browser.   Basically, you'd have an Entity Id, the Mesh Id, and probably a state id (which might be simply a incrementing integer, or some other form of version).  We need a fast and efficient serialization/deserialization format for meshes.   Maybe there's a standard out there.

_payload is likely still useful for other fixed assets, like audio files or jpeg images.