Skip to main content

A REST-RPC Hybrid for Distributed Computing

It's everything the Web was never meant to do...and so much more!

While I was doing some consulting with Warren DeLano of PyMOL, we envisioned a REST-RPC hybrid for use in distributed computing.

Imagine an RPC system built on top of REST. (I know that REST enthusiasts tend to really dislike the RPC style, but bear with me.) That means two things. It means you're calling remote functions, but it also means you should take advantage of HTTP as much as possible. That means you should use URLs, proxies, links, the various HTTP verbs, etc.

Imagine every RPC function takes one JSON value as input and provides one JSON value as output. Such a system could be used with relative easy in a wide range of programming languages, even C. I'm thinking of a P2P system with many servers who talk to each using RESTful Web APIs, but do so in an XML-RPC sort of way. However, instead of using XML, they use JSON.

Here's what I think URLs should look like: http://server:port/path/to/object/;method?arg1=foo&args2=bar. This would convert to a JSON list of args. There could be alternate syntax for the arguments, such as ?__argument__=some_encoded_json if the JSON structure is deep.

The method may choose to support GET or POST in the usual RESTful way. GET requests should not have side effects.

To support asynchronous operation, you can make an RPC and then pass it a callback URL which the remote server can call later via RPC. There must be a way of registering a temporary callback URL in order to support asynchronous operation. You might even use a list of callbacks in order to create a pipeline. This is like a stack of continuations.

Various things like HTTP auth may be used. Various HTTP status codes should be used in ways that make sense. Internally, bindings for languages like Python should translate HTTP error status codes to exceptions. If a server returns an error status code, a traceback object in JSON makes sense.

It makes sense to do something useful for actual UNIX pipes. For instance, if an app is started in a pipeline, it can output URLs to STDOUT so that the two processes connected via a UNIX pipe can connect using the protocol.

This platform would allow you to make use of the strengths and weaknesses of any platform such as C, Python, IronPython, MPI in C, etc. There might be a server written in C that can use SIMD or the GPU. There might be another server that can output stuff to the screen within an OpenGL context.

It needs some way to stream data. Perhaps the protocol is used to do all the setup, and then it gives you a URL where you can stream stuff in any format. After all, it is a web server. You could probably even stream JSON objects, one per line.

I'd like to thank Warren DeLano for giving me an opportunity to blog about our conversations.

Comments

Darrin Eden said…
Oddly enough I believe I may have settled on a freakishly similar approach in a recent project. The difference being I want to characterize this as 'message passing' and I return a list of objects instead of a single object.
jfischer said…
Interesting idea. I think this is more in the original spirit of SOAP than the existing "Web Services" specs.

You seem to have two different ways of specifying methods - as an HTTP method (GET, POST, etc.) and in the URL. How would you choose one approach over the other? One idea might be to use the URL to provide the method name and the HTTP method to indicate (to caches) the impact of the method on the state of the server. Is that what you were thinking?

P.S. I enjoy your blog!
Donovan Preston said…
I've been developing a similar idea. I've been experimenting with the idea of mixing Actors with REST.

Basically, PUT/DELETE/GET on a url will create, kill, or get the current state of an Actor. These verbs are restricted to those that have the ability to perform these actions.

POST will send a message to the Actor. It gets a little fuzzy here since I'm still experimenting, but a POST to the Actor's url will cast a message to the Actor. Instead of giving out the raw url to the Actor I would give out capabilities to the Actor, which is a url to a proxy service that only lets through POST.

I'd like to come up with some way to layer call semantics on top of this so that the POST doesn't have to be completely async. I like the ;method syntax but I think I would prefer always encoding arguments in the POST body.

The Actor could also expose more GET urls in it's namespace; in this case arguments would be encoded in url segments instead of things that look like query parameters. But this is entirely a style issue.
jjinux said…
> Interesting idea. I think this is more in the original spirit of SOAP than the existing "Web Services" specs.

That's an interesting comment.

> You seem to have two different ways of specifying methods - as an HTTP method (GET, POST, etc.) and in the URL. How would you choose one approach over the other? One idea might be to use the URL to provide the method name and the HTTP method to indicate (to caches) the impact of the method on the state of the server. Is that what you were thinking?

Exactly. I don't buy the resource oriented architecture approach of turning everything into a noun and only using 5 (or so verbs). Rather, I think you should say GET or POST, which mean two different things, and then use a real verb for the method name.

> P.S. I enjoy your blog!

Thanks! :)
jjinux said…
> Oddly enough I believe I may have settled on a freakishly similar approach in a recent project.

Ah, interesting.

> The difference being I want to characterize this as 'message passing'

Well, you're from the Ruby world, and I'm from the Python world. We both think of method calls as message passing ;)

> and I return a list of objects instead of a single object.

I specified a JSON object, which could trivially be a list if you need it to.

Thanks for your comment :)
jjinux said…
> I've been developing a similar idea. I've been experimenting with the idea of mixing Actors with REST.

That's exactly what I was thinking as well.

> I like the ;method syntax but I think I would prefer always encoding arguments in the POST body.

For things that don't involve state changes, I really like using GET so you can specify everything in the URL. URLs can be linked to.

> But this is entirely a style issue.

Yeah, I actually don't care very much about the specific URL format, although I really think that it should be standardized to save on confusion.

Donovan, thanks for reading. I hear from you so rarely, that I didn't think you even read my blog.
jjinux said…
I found this in my Scala book:

Akka is a project to implement a platform for building fault-tolerant, distributed applications based on REST, Actors, etc. (http://akkasource.org/).
jjinux said…
Pypes is a library for dataflow programming. It uses stackless and REST.
orubel said…
done. (https://github.com/orubel/restrpc)

Popular posts from this blog

Ubuntu 20.04 on a 2015 15" MacBook Pro

I decided to give Ubuntu 20.04 a try on my 2015 15" MacBook Pro. I didn't actually install it; I just live booted from a USB thumb drive which was enough to try out everything I wanted. In summary, it's not perfect, and issues with my camera would prevent me from switching, but given the right hardware, I think it's a really viable option. The first thing I wanted to try was what would happen if I plugged in a non-HiDPI screen given that my laptop has a HiDPI screen. Without sub-pixel scaling, whatever scale rate I picked for one screen would apply to the other. However, once I turned on sub-pixel scaling, I was able to pick different scale rates for the internal and external displays. That looked ok. I tried plugging in and unplugging multiple times, and it didn't crash. I doubt it'd work with my Thunderbolt display at work, but it worked fine for my HDMI displays at home. I even plugged it into my TV, and it stuck to the 100% scaling I picked for the othe

ERNOS: Erlang Networked Operating System

I've been reading Dreaming in Code lately, and I really like it. If you're not a dreamer, you may safely skip the rest of this post ;) In Chapter 10, "Engineers and Artists", Alan Kay, John Backus, and Jaron Lanier really got me thinking. I've also been thinking a lot about Minix 3 , Erlang , and the original Lisp machine . The ideas are beginning to synthesize into something cohesive--more than just the sum of their parts. Now, I'm sure that many of these ideas have already been envisioned within Tunes.org , LLVM , Microsoft's Singularity project, or in some other place that I haven't managed to discover or fully read, but I'm going to blog them anyway. Rather than wax philosophical, let me just dump out some ideas: Start with Minix 3. It's a new microkernel, and it's meant for real use, unlike the original Minix. "This new OS is extremely small, with the part that runs in kernel mode under 4000 lines of executable code.&quo

Haskell or Erlang?

I've coded in both Erlang and Haskell. Erlang is practical, efficient, and useful. It's got a wonderful niche in the distributed world, and it has some real success stories such as CouchDB and jabber.org. Haskell is elegant and beautiful. It's been successful in various programming language competitions. I have some experience in both, but I'm thinking it's time to really commit to learning one of them on a professional level. They both have good books out now, and it's probably time I read one of those books cover to cover. My question is which? Back in 2000, Perl had established a real niche for systems administration, CGI, and text processing. The syntax wasn't exactly beautiful (unless you're into that sort of thing), but it was popular and mature. Python hadn't really become popular, nor did it really have a strong niche (at least as far as I could see). I went with Python because of its elegance, but since then, I've coded both p