Shell Has a Forth-like Quality

Why Sponsor Oils? |<br>blog | oilshell.org

Shell Has a Forth-like Quality

2017-01-13

[ Russian<br>Translation by Babur Muradov ]

This post takes a short break from ASDL to expand on one of my Hacker News<br>comments.

I realize I haven't done much of what I mentioned in the second<br>sentence of this blog: explain why the Unix shell is<br>interesting. This is partly because almost nobody has questioned this project<br>— it appears programmers do want a better shell.

But there are a aspects of the language design that are rare, and they're worth<br>explaining. This is the first of at least three topics.

Configuring Unix Daemons

Systemd has the stated goal of replacing shell scripts in the boot<br>process of a Linux system.

In response to yet another thread about systemd's design<br>tradeoffs, agumonkey claimed that shell doesn't have enough abstraction<br>power. He suggested instead a Lisp-like configuration system:

(-><br>(path "/usr/bin/some-service" "arg0" ...)<br>(wrap :pre (lambda () ...)<br>:post (lambda () ..))<br>(retry 5)<br>...<br>(timeout 5))

I pointed out that shell already supports this kind of higher-order<br>programming. For example, here's a function that takes a command and tries it<br>five times:

retry() {<br>local n=$1<br>shift<br>for i in $(seq $n); do<br>"$@"<br>done

It can be used like this:

$ retry 5 hello-sleep 0.1<br>hello<br>hello<br>hello<br>hello<br>hello

Here we are passing an integer 5 and a code snippet hello-sleep 0.1 to<br>the retry function. Because retry treats code as data, you can call it a<br>higher-order function .

Taking it further, we can compose our retry function with the timeout<br>binary in coreutils by prepending two more words:

$ timeout 0.3 $0 retry 5 hello-sleep 0.1<br>hello<br>hello<br>hello # killed after 0.3 seconds

(Runnable code is available in forth-like directory of the<br>oilshell/blog-code repository).

Because functions can be composed by simple juxtaposition , I said that<br>shell has a Forth-like quality .

In the Forth language, functions can be composed like this because they<br>work on an implicit stack of arguments and return values. If that doesn't make<br>sense, this blog post may help.

Shell doesn't have an implicit stack, but the uniform representation of words in<br>the argv array, and "splicing" with "$@", results in code that feels<br>similar.

In contrast, this mechanism isn't idiomatic in Python or JavaScript. I tried<br>porting demo.sh to Python with demo.py, and it sort of works if you write<br>all functions like f(*args). But this goes against the grain of the<br>ecosystem. In these languages, functions and arguments are treated differently<br>from a syntactic and semantic point of view.

daemontools and Bernstein Chaining

In the book The Art of Unix Programming, which is a great exposition<br>of the Unix philosophy, Eric Raymond calls the technique<br>Bernstein chaining.

Daniel J. Bernstein uses this shell technique in software like<br>qmail and daemontools to follow the principle of least<br>privilege.

In contrast to systemd, daemontools is a Unix init toolkit which<br>relies on the idiom of small C programs composed with shell scripts .<br>Celebrating daemontools makes a good case for it and shows examples.<br>Here's an excerpt that uses Bernstein chaining of setuidgid and softlimit,<br>as well as the builtin exec:

# change to the user 'sample', and then limit the stack segment<br># to 2048 bytes, the number of open file descriptors to 3, and<br># the number of processes to 1:<br>exec setuidgid sample \<br>softlimit -n 2048 -o 3 -p 1 \<br>some-small-daemon -n

Daemontools is minimally documented and doesn't see much use today, but<br>runit has the same architecture, as well as a collection of tiny shell<br>scripts that illustrate its use.

They are admittedly a bit cryptic, but the architecture is what I care about.<br>systemd does separate some of this functionality in a separate<br>systemd-nspawn binary, but it doesn't appear to be used much without the<br>rest of systemd.

Conclusion

daemontools and systemd are interesting because they represent extremes<br>with respect to the modularity of their design.

Since I'm writing a shell, it shouldn't be a surprise that I'm biased toward<br>the style of daemontools. But systemd has valid criticisms of shell<br>scripts. The language has many problems, array syntax<br>being one example.

On the other hand, I wouldn't be surprised if systemd configuration<br>accidentally turns Turing-complete, as shell and make<br>did.

I don't know what the best answer is, but I think that an improved shell will<br>help the situation. At the very least, Lisp isn't necessary. With oil, I<br>aim to preserve the timeless architectural characteristics of shell, while<br>abandoning ugly, inconsistent syntax, and smoothing over its sharp corners.

Appendix: Commands that Compose

Here is a list of tools that can be composed in this Forth-like manner:

sudo: Run a command as another user.

chroot: Run a command with a different root directory.

env: Run a command in a different environment.

/usr/bin/time: Run a command and system summarize resource usage.

su:...