Python: sort | uniq -c via the subprocess module
Here is "sort | uniq -c" pieced together using the subprocess module:
Now, here's the question. Why does the program freeze if I put the two Popen lines together? I don't understand why I can't setup the pipeline, then feed it data, then close the stdin, and then read the result.
from subprocess import Popen, PIPENote, I'm not bothering to check the exit status. You can see my previous post about how to do that.
p1 = Popen(["sort"], stdin=PIPE, stdout=PIPE)
p1.stdin.write('FOO\nBAR\nBAR\n')
p1.stdin.close()
p2 = Popen(["uniq", "-c"], stdin=p1.stdout, stdout=PIPE)
for line in p2.stdout:
print line.rstrip()
Now, here's the question. Why does the program freeze if I put the two Popen lines together? I don't understand why I can't setup the pipeline, then feed it data, then close the stdin, and then read the result.
Comments
p1 = Popen(["sort"], stdin=PIPE, stdout=PIPE)
p1.stdin.write('FOO\nBAR\nBAR\n')
p1.stdin.close()
Will only work for large writes because sort happens to be a "sponge", i.e. sucks up all its input before emitting any output.
If it were a filter it could block waiting for a reader, which would block you (the writer) once the pipe's buffer filled.
I usually spawn a thread for the input end of a pipeline for this reason.
That must be it. Thanks!
ltrace/strace helps in such cases usually, since it becomes clear I/O is blocking
kmh
That's a great tip. Thanks!
pipeline class.
from pipes import Template
t = Template()
t.append('p4 print -q \"$IN\"', 'f-')
# iconv from src encoding to UTF-8
t.append('iconv -f %s -t UTF-8' % (r.p4encoding), '--')
# Turn on debugging
t.debug(1)
# start it
t.copy('//Work/foo/bar', '/tmp/foo')
* it takes strings, not lists, so it's doomed to quoting horror
* unlike pretty much the entire rest of python, it ignores errors (it doesn't even look *possible* to get the exit statuses out.)
(Sadly both of those are based on the interface, not the implementation, so it's not just a matter of fixing bugs.)