Skip to main content

A REST-RPC Hybrid for Distributed Computing

It's everything the Web was never meant to do...and so much more!

While I was doing some consulting with Warren DeLano of PyMOL, we envisioned a REST-RPC hybrid for use in distributed computing.

Imagine an RPC system built on top of REST. (I know that REST enthusiasts tend to really dislike the RPC style, but bear with me.) That means two things. It means you're calling remote functions, but it also means you should take advantage of HTTP as much as possible. That means you should use URLs, proxies, links, the various HTTP verbs, etc.

Imagine every RPC function takes one JSON value as input and provides one JSON value as output. Such a system could be used with relative easy in a wide range of programming languages, even C. I'm thinking of a P2P system with many servers who talk to each using RESTful Web APIs, but do so in an XML-RPC sort of way. However, instead of using XML, they use JSON.

Here's what I think URLs should look like: http://server:port/path/to/object/;method?arg1=foo&args2=bar. This would convert to a JSON list of args. There could be alternate syntax for the arguments, such as ?__argument__=some_encoded_json if the JSON structure is deep.

The method may choose to support GET or POST in the usual RESTful way. GET requests should not have side effects.

To support asynchronous operation, you can make an RPC and then pass it a callback URL which the remote server can call later via RPC. There must be a way of registering a temporary callback URL in order to support asynchronous operation. You might even use a list of callbacks in order to create a pipeline. This is like a stack of continuations.

Various things like HTTP auth may be used. Various HTTP status codes should be used in ways that make sense. Internally, bindings for languages like Python should translate HTTP error status codes to exceptions. If a server returns an error status code, a traceback object in JSON makes sense.

It makes sense to do something useful for actual UNIX pipes. For instance, if an app is started in a pipeline, it can output URLs to STDOUT so that the two processes connected via a UNIX pipe can connect using the protocol.

This platform would allow you to make use of the strengths and weaknesses of any platform such as C, Python, IronPython, MPI in C, etc. There might be a server written in C that can use SIMD or the GPU. There might be another server that can output stuff to the screen within an OpenGL context.

It needs some way to stream data. Perhaps the protocol is used to do all the setup, and then it gives you a URL where you can stream stuff in any format. After all, it is a web server. You could probably even stream JSON objects, one per line.

I'd like to thank Warren DeLano for giving me an opportunity to blog about our conversations.

Comments

Darrin Eden said…
Oddly enough I believe I may have settled on a freakishly similar approach in a recent project. The difference being I want to characterize this as 'message passing' and I return a list of objects instead of a single object.
jfischer said…
Interesting idea. I think this is more in the original spirit of SOAP than the existing "Web Services" specs.

You seem to have two different ways of specifying methods - as an HTTP method (GET, POST, etc.) and in the URL. How would you choose one approach over the other? One idea might be to use the URL to provide the method name and the HTTP method to indicate (to caches) the impact of the method on the state of the server. Is that what you were thinking?

P.S. I enjoy your blog!
Donovan Preston said…
I've been developing a similar idea. I've been experimenting with the idea of mixing Actors with REST.

Basically, PUT/DELETE/GET on a url will create, kill, or get the current state of an Actor. These verbs are restricted to those that have the ability to perform these actions.

POST will send a message to the Actor. It gets a little fuzzy here since I'm still experimenting, but a POST to the Actor's url will cast a message to the Actor. Instead of giving out the raw url to the Actor I would give out capabilities to the Actor, which is a url to a proxy service that only lets through POST.

I'd like to come up with some way to layer call semantics on top of this so that the POST doesn't have to be completely async. I like the ;method syntax but I think I would prefer always encoding arguments in the POST body.

The Actor could also expose more GET urls in it's namespace; in this case arguments would be encoded in url segments instead of things that look like query parameters. But this is entirely a style issue.
jjinux said…
> Interesting idea. I think this is more in the original spirit of SOAP than the existing "Web Services" specs.

That's an interesting comment.

> You seem to have two different ways of specifying methods - as an HTTP method (GET, POST, etc.) and in the URL. How would you choose one approach over the other? One idea might be to use the URL to provide the method name and the HTTP method to indicate (to caches) the impact of the method on the state of the server. Is that what you were thinking?

Exactly. I don't buy the resource oriented architecture approach of turning everything into a noun and only using 5 (or so verbs). Rather, I think you should say GET or POST, which mean two different things, and then use a real verb for the method name.

> P.S. I enjoy your blog!

Thanks! :)
jjinux said…
> Oddly enough I believe I may have settled on a freakishly similar approach in a recent project.

Ah, interesting.

> The difference being I want to characterize this as 'message passing'

Well, you're from the Ruby world, and I'm from the Python world. We both think of method calls as message passing ;)

> and I return a list of objects instead of a single object.

I specified a JSON object, which could trivially be a list if you need it to.

Thanks for your comment :)
jjinux said…
> I've been developing a similar idea. I've been experimenting with the idea of mixing Actors with REST.

That's exactly what I was thinking as well.

> I like the ;method syntax but I think I would prefer always encoding arguments in the POST body.

For things that don't involve state changes, I really like using GET so you can specify everything in the URL. URLs can be linked to.

> But this is entirely a style issue.

Yeah, I actually don't care very much about the specific URL format, although I really think that it should be standardized to save on confusion.

Donovan, thanks for reading. I hear from you so rarely, that I didn't think you even read my blog.
jjinux said…
I found this in my Scala book:

Akka is a project to implement a platform for building fault-tolerant, distributed applications based on REST, Actors, etc. (http://akkasource.org/).
jjinux said…
Pypes is a library for dataflow programming. It uses stackless and REST.
orubel said…
done. (https://github.com/orubel/restrpc)

Popular posts from this blog

Drawing Sierpinski's Triangle in Minecraft Using Python

In his keynote at PyCon, Eben Upton, the Executive Director of the Rasberry Pi Foundation, mentioned that not only has Minecraft been ported to the Rasberry Pi, but you can even control it with Python . Since four of my kids are avid Minecraft fans, I figured this might be a good time to teach them to program using Python. So I started yesterday with the goal of programming something cool for Minecraft and then showing it off at the San Francisco Python Meetup in the evening. The first problem that I faced was that I didn't have a Rasberry Pi. You can't hack Minecraft by just installing the Minecraft client. Speaking of which, I didn't have the Minecraft client installed either ;) My kids always play it on their Nexus 7s. I found an open source Minecraft server called Bukkit that "provides the means to extend the popular Minecraft multiplayer server." Then I found a plugin called RaspberryJuice that implements a subset of the Minecraft Pi modding API for B

Ubuntu 20.04 on a 2015 15" MacBook Pro

I decided to give Ubuntu 20.04 a try on my 2015 15" MacBook Pro. I didn't actually install it; I just live booted from a USB thumb drive which was enough to try out everything I wanted. In summary, it's not perfect, and issues with my camera would prevent me from switching, but given the right hardware, I think it's a really viable option. The first thing I wanted to try was what would happen if I plugged in a non-HiDPI screen given that my laptop has a HiDPI screen. Without sub-pixel scaling, whatever scale rate I picked for one screen would apply to the other. However, once I turned on sub-pixel scaling, I was able to pick different scale rates for the internal and external displays. That looked ok. I tried plugging in and unplugging multiple times, and it didn't crash. I doubt it'd work with my Thunderbolt display at work, but it worked fine for my HDMI displays at home. I even plugged it into my TV, and it stuck to the 100% scaling I picked for the othe

Creating Windows 10 Boot Media for a Lenovo Thinkpad T410 Using Only a Mac and a Linux Machine

TL;DR: Giovanni and I struggled trying to get Windows 10 installed on the Lenovo Thinkpad T410. We struggled a lot trying to create the installation media because we only had a Mac and a Linux machine to work with. Everytime we tried to boot the USB thumb drive, it just showed us a blinking cursor. At the end, we finally realized that Windows 10 wasn't supported on this laptop :-/ I've heard that it took Thomas Edison 100 tries to figure out the right material to use as a lightbulb filament. Well, I'm no Thomas Edison, but I thought it might be noteworthy to document our attempts at getting it to boot off a USB thumb drive: Download the ISO. Attempt 1: Use Etcher. Etcher says it doesn't work for Windows. Attempt 2: Use Boot Camp Assistant. It doesn't have that feature anymore. Attempt 3: Use Disk Utility on a Mac. Erase a USB thumb drive: Format: ExFAT Scheme: GUID Partition Map Mount the ISO. Copy everything from