Thoughts on Emissary Structure

Internal form of an Emissary Load File:

This is inspired by Android APK and Java Jar files.   But, I think instead of using a zip format, I will use a TAR format.   The reason for this is speed.   Also, after a possible compression layer is removed, the tar file can be internally seek'ed in order to emulate a high speed filesystem - while this is problematic for compressed files.   Since it is the intention of the Emissary file to become a read-only virtual filesystem for V8, this works rather well.   

Files at the top level (speculative):

• CERT.CA - Certificate of the code signer
• Manifest - A JSON file containing cryptographic checksums (sha256) of all the files in the module.   This should be signed by the above certificate.
• Manifest.sign - the digitally signed version of the sha256 of Manifest
• _code - directory containing the emissary code
• _code/main.js - the startup script
• _code/main.wasm (future) - a webassembly startup
• _payload - the root of the filesystem for the Emissary.  

Loading the Emissary is known as Marshaling it.

The system will open this file and inventory it (traverse the 512 byte TAR file headers and build an index) and store the index in a sqlite database.   If will check the certificate and load the manifest and check that it's properly signed.  Then it will checksum all the files and compare these with the manifest to validate the load.  Any errors will cause the Marshal phase to fail.   

If the original mime-type comes back as something like GZIP or a compatible compression type, it will be decompressed into a TAR file that is placed in a cache area (compressing these is recommended and may become mandatory, though we may ultimately support different algorithms).  That way, it will be available - already marshaled without needing to reload it.  However, new software revisions will have different codes and these will be automatically loaded.  Theoretically we can do hot code changes this way.   

Note on the read-only nature of the Emissary:  Besides the security aspect of allowing this to be modified, state is only allowed to be cached on the Emissary side -- we will likely give access to some small database or redis for this.  True persisted state exists on the server side, so we don't want to encourage the Emissary to think it owns state.   If we need to cache big Emissary-side objects (like big meshes) we'll make a way to do that inside 3DOM.

There are theoretically some seekable gzip libraries out there.  We might use this someday to reduce disk space.