Skip to main content

A REST-RPC Hybrid for Distributed Computing

It's everything the Web was never meant to do...and so much more!

While I was doing some consulting with Warren DeLano of PyMOL, we envisioned a REST-RPC hybrid for use in distributed computing.

Imagine an RPC system built on top of REST. (I know that REST enthusiasts tend to really dislike the RPC style, but bear with me.) That means two things. It means you're calling remote functions, but it also means you should take advantage of HTTP as much as possible. That means you should use URLs, proxies, links, the various HTTP verbs, etc.

Imagine every RPC function takes one JSON value as input and provides one JSON value as output. Such a system could be used with relative easy in a wide range of programming languages, even C. I'm thinking of a P2P system with many servers who talk to each using RESTful Web APIs, but do so in an XML-RPC sort of way. However, instead of using XML, they use JSON.

Here's what I think URLs should look like: http://server:port/path/to/object/;method?arg1=foo&args2=bar. This would convert to a JSON list of args. There could be alternate syntax for the arguments, such as ?__argument__=some_encoded_json if the JSON structure is deep.

The method may choose to support GET or POST in the usual RESTful way. GET requests should not have side effects.

To support asynchronous operation, you can make an RPC and then pass it a callback URL which the remote server can call later via RPC. There must be a way of registering a temporary callback URL in order to support asynchronous operation. You might even use a list of callbacks in order to create a pipeline. This is like a stack of continuations.

Various things like HTTP auth may be used. Various HTTP status codes should be used in ways that make sense. Internally, bindings for languages like Python should translate HTTP error status codes to exceptions. If a server returns an error status code, a traceback object in JSON makes sense.

It makes sense to do something useful for actual UNIX pipes. For instance, if an app is started in a pipeline, it can output URLs to STDOUT so that the two processes connected via a UNIX pipe can connect using the protocol.

This platform would allow you to make use of the strengths and weaknesses of any platform such as C, Python, IronPython, MPI in C, etc. There might be a server written in C that can use SIMD or the GPU. There might be another server that can output stuff to the screen within an OpenGL context.

It needs some way to stream data. Perhaps the protocol is used to do all the setup, and then it gives you a URL where you can stream stuff in any format. After all, it is a web server. You could probably even stream JSON objects, one per line.

I'd like to thank Warren DeLano for giving me an opportunity to blog about our conversations.

Comments

Darrin Eden said…
Oddly enough I believe I may have settled on a freakishly similar approach in a recent project. The difference being I want to characterize this as 'message passing' and I return a list of objects instead of a single object.
jfischer said…
Interesting idea. I think this is more in the original spirit of SOAP than the existing "Web Services" specs.

You seem to have two different ways of specifying methods - as an HTTP method (GET, POST, etc.) and in the URL. How would you choose one approach over the other? One idea might be to use the URL to provide the method name and the HTTP method to indicate (to caches) the impact of the method on the state of the server. Is that what you were thinking?

P.S. I enjoy your blog!
Donovan said…
I've been developing a similar idea. I've been experimenting with the idea of mixing Actors with REST.

Basically, PUT/DELETE/GET on a url will create, kill, or get the current state of an Actor. These verbs are restricted to those that have the ability to perform these actions.

POST will send a message to the Actor. It gets a little fuzzy here since I'm still experimenting, but a POST to the Actor's url will cast a message to the Actor. Instead of giving out the raw url to the Actor I would give out capabilities to the Actor, which is a url to a proxy service that only lets through POST.

I'd like to come up with some way to layer call semantics on top of this so that the POST doesn't have to be completely async. I like the ;method syntax but I think I would prefer always encoding arguments in the POST body.

The Actor could also expose more GET urls in it's namespace; in this case arguments would be encoded in url segments instead of things that look like query parameters. But this is entirely a style issue.
> Interesting idea. I think this is more in the original spirit of SOAP than the existing "Web Services" specs.

That's an interesting comment.

> You seem to have two different ways of specifying methods - as an HTTP method (GET, POST, etc.) and in the URL. How would you choose one approach over the other? One idea might be to use the URL to provide the method name and the HTTP method to indicate (to caches) the impact of the method on the state of the server. Is that what you were thinking?

Exactly. I don't buy the resource oriented architecture approach of turning everything into a noun and only using 5 (or so verbs). Rather, I think you should say GET or POST, which mean two different things, and then use a real verb for the method name.

> P.S. I enjoy your blog!

Thanks! :)
> Oddly enough I believe I may have settled on a freakishly similar approach in a recent project.

Ah, interesting.

> The difference being I want to characterize this as 'message passing'

Well, you're from the Ruby world, and I'm from the Python world. We both think of method calls as message passing ;)

> and I return a list of objects instead of a single object.

I specified a JSON object, which could trivially be a list if you need it to.

Thanks for your comment :)
> I've been developing a similar idea. I've been experimenting with the idea of mixing Actors with REST.

That's exactly what I was thinking as well.

> I like the ;method syntax but I think I would prefer always encoding arguments in the POST body.

For things that don't involve state changes, I really like using GET so you can specify everything in the URL. URLs can be linked to.

> But this is entirely a style issue.

Yeah, I actually don't care very much about the specific URL format, although I really think that it should be standardized to save on confusion.

Donovan, thanks for reading. I hear from you so rarely, that I didn't think you even read my blog.
I found this in my Scala book:

Akka is a project to implement a platform for building fault-tolerant, distributed applications based on REST, Actors, etc. (http://akkasource.org/).
Pypes is a library for dataflow programming. It uses stackless and REST.
Xeno said…
done. (https://github.com/orubel/restrpc)