DWN: SDKs and as a Service

[alr]

I love Web5 for the simplicity it gives developers. Decentralization is tough. Many devs want to build decentralized applications - but they can’t be creating their own decentralized networks.

So folks building atop the Web5 programming model get decentralization for free. That’s powerful. It’s why application servers and container orchestration engines are so prevalent - they do the heavy lifting of resource management and plumbing, freeing app developers to focus on business logic. All they’ve got to do is use the Web5 APIs.

In MVC architectures we do this all the time - there’s the application layer, and above that, an HTTP layer (the controller) which does request/response parsing and calls the application.

Looks crudely like this:

Sweet. The user application is free to define its own API, and exposes whatever it wants as HTTP endpoints. Those HTTP endpoints do whatever parsing/validation/plumbing nonsense they want and call the application’s APIs.

I envision a similar setup for decentralized applications - a DWN/Web5 layer.

The decentralization engine for Web5 is an implementation of DIF’s Decentralized Web Node spec. At TBD we have one in development now, implemented in TypeScript/JavaScript.

As an SDK, it’s meant to be imported into a user’s project and called upon directly. As a first implementation for us, this makes sense because we are building a web-based Wallet as a browser extension. That wallet needs to “speak DWN” - sending DWN-formatted messages and sending them over the wire.

So now our application can look a little like this:

Here we’ve swapped the HTTP layer for a DWN one. And because there’s no Java Servlet engine, or Node Express, or any of the other ecosystem support for DWN as there is for HTTP, we import an implementation of the DWN SDK. The application’s DWN layer interacts with the application and uses the DWN SDK to send and receive messages.

For many users, this may be preferable. Make your application, import the DWN SDK, and call upon the DWN SDK in-process.

There are some observably true limitations to this approach I think are worth addressing:

• The application will need a language-compliant implementation of the DWN SDK. Either through bindings (often: ew) or in a compatible runtime. If we have only a TypeScript/JavaScript implementation, folks in Go, Java, Python, etc are left without a solution until there’s an impl of the SDK for them.
• There’s a development and maintenance cost to each language impl of the emerging DWN standard.
• We would like for as many people as possible to use DWN as early as possible.
• The TypeScript/JavaScript DWN implementation we have now will not be able to be consumed by our Go-based SSI SDK and Service.

My opinions from those are:

• It doesn’t make sense for us to deliver DWN SDKs in several languages in the near-term. It likely does as things mature and our community grows.

So I propose a language-agnostic API for DWN, based on HTTP. We have some precedent for this; we’re already doing it in the SSI Service. Folks wanting to call upon the operations that the SSI SDK provides can do so over the SSI Service’s HTTP endpoints. We can wrap the DWN SDK in a DWN Service and expose its API over HTTP.

That’d look like this:

Here we’ve removed the DWN SDK from the user application. Instead, we’ve exposed the DWN SDK to the application, not to the world, through an HTTP API instead of an language-specific API. It’s adding an API to the existing DWN SDK over a network boundary instead of an in-process one.

Then folks could run the DWN as a service. It has the following benefits:

• Easily run and secured via a container
• Developers could call upon it using familiar means with wide ecosystem support, HTTP
• Immediately opens DWN for use in any language, even curl for testing/prototyping/scripting
• Larger enterprises who are hesitant to bring in new dependencies are more likely to run services with well-defined containerized security policies, which are easier to enforce.
• Upgrade path requires pulling a new container image of the service, rather than rebuilding and redeploying the user application

Drawbacks:

• I don’t see any. We still ship the DWN SDK and folks are free to use it directly if the in-process approach better serves their use case. For many this will be true.

Note: this is a different discussion from the issue of decentralized message format and transport. That exists between the DWN SDK and the WAN. This is only about how user applications, the SSI Service included, may call upon the DWN.

Excited to hear your thoughts.

S,
ALR

[moe]

@alr this is a bit of a nit but i want to be sure i understand what you’re conveying.

Re:

Here we’ve swapped the HTTP layer for a DWN one. And because there’s no Java Servlet engine, or Node Express, or any of the other ecosystem support for DWN as there is for HTTP, we import an implementation of the DWN SDK. The application’s DWN layer interacts with the application and uses the DWN SDK to send and receive messages.

by “HTTP layer” do you mean “RESTful layer”? because it’s very likely that DWN will listen over HTTP, just not in a RESTful manner.

[alr]

@moe

Yep - HTTP layer, because the flow may not map to RESTful principles, like you call out. Whatever exposes the DWN API over HTTP in a language-agnostic way.

Looking for y’all to poke holes and advise why this approach isn’t technically feasible, or if there’s a better solution!

S,
ALR

[moe]

@alr, before providing my POV, i want to make sure i understand one of the main points you’re driving home in favor of DWNs as a service. Will you let me know if the summary below is accurate?

Importing libraries / SDKs into projects introduces risks such as an increased likelihood of vulnerabilities and attack vectors exposed through transitive dependencies (e.g. log4j exploit). Certain institutions / enterprises have rigorous auditing processes for each dependency they introduce into a critical path so as to avoid falling victim to said risks. When/if certain dependencies surpass a specific risk threshold but still provide some necessary functionality, insitutions will opt to isolate a given dependency via network boundaries. This is done by deplyoing that dependency as an independent service and exposing its functionality over some transport mechanism (e.g. HTTP, AMQP etc.). This service is then deployed into an area in the network topology that is cordoned off from any sensitive services or datastores so as to mitigate the magnitude of any potential vulnerability exploits. If a given dependency does have a vuln that gets exploited, the bad actor’s reach is minimized because there’s no way to reach out to any “gold mines” that may exist within an internal network.

Providing DWN as a service could very well increase the likelihood of adoption by institutions (e.g. banks) that may be hesitant to introduce new dependencies for the reasons described above

[moe]

i see loose analogy between DWNaaS and NGINX Reverse Proxy. Like NGINX, DWNaaS would act as a WAN entrypoint for inbound requests/messages and reverse proxy them to pre-registered/configured services over HTTP. There are 3 notable differences that come to mind:

1. NGINX relies on paths or request headers to decide where to proxy a request whereas DWNaaS would rely on certain properties within DWeb Messages. Not really an issue IMO.

2. DWNaaS will need access to private keys so that it can encrypt, decrypt, and sign messages. These keys are typically stored in a KMS of sorts (at least for institutions). Presumably, access to this KMS will requires acess to the most secured part of an internal network. This could end up being a tough pill for insitutions to swallow given that DWNaaS is a public entrypoint and now presumably has direct access to a KMS

3. DWNs imply storage. All inbound messages are stored assuming that they pass a series of integrity checks. As Gabe alluded to in his prior message, data storage has all sorts of implications ranging from auditing requirements to staffing up a team that knows how to keep the data locked down and the service up. It may be worthwhile to introduce the ability to turn storage off (via config, a feature flag etc.) so that DWNaaS can act just as an integrity checking message router.

Additionally, one apect that I think merits a deep-dive is data flow. More specifically, When DWNaaS proxies a message, does it send a request to the service, wait for a response, which it then DWMify’s and sends as a response to the client? or is async behavior expected?

Instinctively, I feel like this needs to be sync because imagine a scenario where an ASK or CredentialApplication is malformed (e.g. missing required property, incorrect type for property value etc.). I think we’d want to inform the client that it’s malformed as a response to the request (aka synchronously). seems strange to handle it async where the client sends a malformed CredentialApplication which gets stored in addition to a message that contains the error details which the client is now polling for. This effectively means that the “reply” to the CredentialApplication is an error message.

[mic]

I think it is good to separate “DWN as a service” as in a remotely addressible service (via http/json/rest/whatever) from “DWN running as a daemon/service on loopback” as 2 different (but related) concerns.

The latter I think is a good starting point for rapid adoption from many client languages, but doesn’t require (yet) leaping towards a truly remote service (which is interesting, but brings up more issues around trust and storage).

So it is my understanding that DWN’s current reference implementation in JS/TS is just to the “web native” nature of javascript runtimes (browser extensions and nodejs and more), which makes a lot of sense.

Providing the ability for the DWN SDK to listen on a port (limited to loopback perhaps) then opens this up to really any developer and use case.

A “curl-able” api for teaching/experimentation is an amazingly low barrier, and writing language “binding” to this style of API is relatively trivial compared to multiple per language implementations to start with.

This is not without precedent either, some ones that spring to mind to me are:

• LLVM jit: can be used as a library in process or as a service (daemon)
• I hear photoshop (the app!) has this similar ability to run as library or service depending on needs
• Open Policy Agent.

The last one is an interesting precedent: they offer both a library (currently with golang) and a WASM option (which can be bound in to many runtime/in process situations) but also run as a daemon with a well defined HTTP interface.

Integrating OPA

It says REST api is most popular right now, but there are tradeoffs:

so do you think an interesting starting point would be a purely localhost/loopback service (which has a side effect of keeping the API super simple for consumers) is interesting in a similar fashion?

A truly remote “as a service” is interesting for other reasons (perhaps quick experimentation and so on) but has other security and access concerns vs a purely local service that joins up with endpoints (and can be packaged in all sorts of ways, distributed as source or binaries that people can quickly start up and then interact with from their programming language/script of choice).

Thoughts?

[longform]

It looks like it would help devs dip their toe in decentralization without having to massively rewrite existing applications?

[andor]

Just want to express my interest in going more in depth on this topic. If there’s any conversations going on outside of the forums (i.e on discord), I’d love to get involved. I hadn’t seen this, and was just talking to @moe yesterday about a closely related topic. TLDR; If anyone else is interested in diving into this more, please let me know.