Skip to main content

A REST-RPC Hybrid for Distributed Computing

It's everything the Web was never meant to do...and so much more!

While I was doing some consulting with Warren DeLano of PyMOL, we envisioned a REST-RPC hybrid for use in distributed computing.

Imagine an RPC system built on top of REST. (I know that REST enthusiasts tend to really dislike the RPC style, but bear with me.) That means two things. It means you're calling remote functions, but it also means you should take advantage of HTTP as much as possible. That means you should use URLs, proxies, links, the various HTTP verbs, etc.

Imagine every RPC function takes one JSON value as input and provides one JSON value as output. Such a system could be used with relative easy in a wide range of programming languages, even C. I'm thinking of a P2P system with many servers who talk to each using RESTful Web APIs, but do so in an XML-RPC sort of way. However, instead of using XML, they use JSON.

Here's what I think URLs should look like: http://server:port/path/to/object/;method?arg1=foo&args2=bar. This would convert to a JSON list of args. There could be alternate syntax for the arguments, such as ?__argument__=some_encoded_json if the JSON structure is deep.

The method may choose to support GET or POST in the usual RESTful way. GET requests should not have side effects.

To support asynchronous operation, you can make an RPC and then pass it a callback URL which the remote server can call later via RPC. There must be a way of registering a temporary callback URL in order to support asynchronous operation. You might even use a list of callbacks in order to create a pipeline. This is like a stack of continuations.

Various things like HTTP auth may be used. Various HTTP status codes should be used in ways that make sense. Internally, bindings for languages like Python should translate HTTP error status codes to exceptions. If a server returns an error status code, a traceback object in JSON makes sense.

It makes sense to do something useful for actual UNIX pipes. For instance, if an app is started in a pipeline, it can output URLs to STDOUT so that the two processes connected via a UNIX pipe can connect using the protocol.

This platform would allow you to make use of the strengths and weaknesses of any platform such as C, Python, IronPython, MPI in C, etc. There might be a server written in C that can use SIMD or the GPU. There might be another server that can output stuff to the screen within an OpenGL context.

It needs some way to stream data. Perhaps the protocol is used to do all the setup, and then it gives you a URL where you can stream stuff in any format. After all, it is a web server. You could probably even stream JSON objects, one per line.

I'd like to thank Warren DeLano for giving me an opportunity to blog about our conversations.

Comments

Darrin Eden said…
Oddly enough I believe I may have settled on a freakishly similar approach in a recent project. The difference being I want to characterize this as 'message passing' and I return a list of objects instead of a single object.
jfischer said…
Interesting idea. I think this is more in the original spirit of SOAP than the existing "Web Services" specs.

You seem to have two different ways of specifying methods - as an HTTP method (GET, POST, etc.) and in the URL. How would you choose one approach over the other? One idea might be to use the URL to provide the method name and the HTTP method to indicate (to caches) the impact of the method on the state of the server. Is that what you were thinking?

P.S. I enjoy your blog!
Donovan Preston said…
I've been developing a similar idea. I've been experimenting with the idea of mixing Actors with REST.

Basically, PUT/DELETE/GET on a url will create, kill, or get the current state of an Actor. These verbs are restricted to those that have the ability to perform these actions.

POST will send a message to the Actor. It gets a little fuzzy here since I'm still experimenting, but a POST to the Actor's url will cast a message to the Actor. Instead of giving out the raw url to the Actor I would give out capabilities to the Actor, which is a url to a proxy service that only lets through POST.

I'd like to come up with some way to layer call semantics on top of this so that the POST doesn't have to be completely async. I like the ;method syntax but I think I would prefer always encoding arguments in the POST body.

The Actor could also expose more GET urls in it's namespace; in this case arguments would be encoded in url segments instead of things that look like query parameters. But this is entirely a style issue.
jjinux said…
> Interesting idea. I think this is more in the original spirit of SOAP than the existing "Web Services" specs.

That's an interesting comment.

> You seem to have two different ways of specifying methods - as an HTTP method (GET, POST, etc.) and in the URL. How would you choose one approach over the other? One idea might be to use the URL to provide the method name and the HTTP method to indicate (to caches) the impact of the method on the state of the server. Is that what you were thinking?

Exactly. I don't buy the resource oriented architecture approach of turning everything into a noun and only using 5 (or so verbs). Rather, I think you should say GET or POST, which mean two different things, and then use a real verb for the method name.

> P.S. I enjoy your blog!

Thanks! :)
jjinux said…
> Oddly enough I believe I may have settled on a freakishly similar approach in a recent project.

Ah, interesting.

> The difference being I want to characterize this as 'message passing'

Well, you're from the Ruby world, and I'm from the Python world. We both think of method calls as message passing ;)

> and I return a list of objects instead of a single object.

I specified a JSON object, which could trivially be a list if you need it to.

Thanks for your comment :)
jjinux said…
> I've been developing a similar idea. I've been experimenting with the idea of mixing Actors with REST.

That's exactly what I was thinking as well.

> I like the ;method syntax but I think I would prefer always encoding arguments in the POST body.

For things that don't involve state changes, I really like using GET so you can specify everything in the URL. URLs can be linked to.

> But this is entirely a style issue.

Yeah, I actually don't care very much about the specific URL format, although I really think that it should be standardized to save on confusion.

Donovan, thanks for reading. I hear from you so rarely, that I didn't think you even read my blog.
jjinux said…
I found this in my Scala book:

Akka is a project to implement a platform for building fault-tolerant, distributed applications based on REST, Actors, etc. (http://akkasource.org/).
jjinux said…
Pypes is a library for dataflow programming. It uses stackless and REST.
orubel said…
done. (https://github.com/orubel/restrpc)

Popular posts from this blog

Drawing Sierpinski's Triangle in Minecraft Using Python

In his keynote at PyCon, Eben Upton, the Executive Director of the Rasberry Pi Foundation, mentioned that not only has Minecraft been ported to the Rasberry Pi, but you can even control it with Python. Since four of my kids are avid Minecraft fans, I figured this might be a good time to teach them to program using Python. So I started yesterday with the goal of programming something cool for Minecraft and then showing it off at the San Francisco Python Meetup in the evening.

The first problem that I faced was that I didn't have a Rasberry Pi. You can't hack Minecraft by just installing the Minecraft client. Speaking of which, I didn't have the Minecraft client installed either ;) My kids always play it on their Nexus 7s. I found an open source Minecraft server called Bukkit that "provides the means to extend the popular Minecraft multiplayer server." Then I found a plugin called RaspberryJuice that implements a subset of the Minecraft Pi modding API for Bukkit s…

Apple: iPad and Emacs

Someone asked my boss's buddy Art Medlar if he was going to buy an iPad. He said, "I figure as soon as it runs Emacs, that will be the sign to buy." I think he was just trying to be funny, but his statement is actually fairly profound.

It's well known that submitting iPhone and iPad applications for sale on Apple's store is a huge pain--even if they're free and open source. Apple is acting as a gatekeeper for what is and isn't allowed on your device. I heard that Apple would never allow a scripting language to be installed on your iPad because it would allow end users to run code that they hadn't verified. (I don't have a reference for this, but if you do, please post it below.) Emacs is mostly written in Emacs Lisp. Per Apple's policy, I don't think it'll ever be possible to run Emacs on the iPad.

Emacs was written by Richard Stallman, and it practically defines the Free Software movement (in a manner of speaking at least). Stal…

ERNOS: Erlang Networked Operating System

I've been reading Dreaming in Code lately, and I really like it. If you're not a dreamer, you may safely skip the rest of this post ;)

In Chapter 10, "Engineers and Artists", Alan Kay, John Backus, and Jaron Lanier really got me thinking. I've also been thinking a lot about Minix 3, Erlang, and the original Lisp machine. The ideas are beginning to synthesize into something cohesive--more than just the sum of their parts.

Now, I'm sure that many of these ideas have already been envisioned within Tunes.org, LLVM, Microsoft's Singularity project, or in some other place that I haven't managed to discover or fully read, but I'm going to blog them anyway.

Rather than wax philosophical, let me just dump out some ideas:Start with Minix 3. It's a new microkernel, and it's meant for real use, unlike the original Minix. "This new OS is extremely small, with the part that runs in kernel mode under 4000 lines of executable code." I bet it&…