Can Lisp Macro Change Lisp Syntax?

By Xah Lee. Date: . Last updated: .
byte mag cover 1979-08 lisp
Byte magazine cover, 1979-08. (image from a lisp hacker Kazimir Majorinc At http://kazimirmajorinc.blogspot.com/2010/07/cover-of-byte-1979-lisp-issue.html. Used with permission)

What Does “Syntax” Mean in Lisp?

The word “syntax” has special meaning in lisps. (no, we are not talking about the nested parenthesis.)

in other languages, such as Java Ruby JavaScript, the syntax of the language is what you see in source code.

in lisps, sometimes they call that surface syntax.

in lisps, there's a thing called “reader”, which is part of the interpreter/compiler. The “reader” reads in the source code, then turn that into a sequence of lisp primitives such as lists, symbols, numbers, etc. This, lispers think of as “the lisp syntax”.

See how Racket doc explains:

The syntax of Racket is not defined directly in terms of character streams. Instead, the syntax is determined by two layers:

〔from http://docs.racket-lang.org/guide/Pairs__Lists__and_Racket_Syntax.html#%28tech._expander%29

Here's how Clojure doc explains:

Clojure is a homoiconic language, which is a fancy term describing the fact that Clojure programs are represented by Clojure data structures. This is a very important difference between Clojure (and Common Lisp) and most other programming languages - Clojure is defined in terms of the evaluation of data structures and not in terms of the syntax of character streams/files. It is quite common, and easy, for Clojure programs to manipulate, transform and produce other Clojure programs.

That said, most Clojure programs begin life as text files, and it is the task of the reader to parse the text and produce the data structure the compiler will see. This is not merely a phase of the compiler. The reader, and the Clojure data representations, have utility on their own in many of the same contexts one might use XML or JSON etc.

One might say the reader has syntax defined in terms of characters, and the Clojure language has syntax defined in terms of symbols, lists, vectors, maps etc. The reader is represented by the function read, which reads the next form (not character) from a stream, and returns the object represented by that form.

〔from http://clojure.org/reader

Here's how a popular Common Lisp book 〈Practical Common Lisp〉 explains:

In Common Lisp things are sliced up a bit differently, with consequences for both the implementer and for how the language is defined. Instead of a single black box that goes from text to program behavior in one step, Common Lisp defines two black boxes, one that translates text into Lisp objects and another that implements the semantics of the language in terms of those objects. The first box is called the reader, and the second is called the evaluator.

〔from http://www.gigamonkeys.com/book/syntax-and-semantics.html

Lisp Macros Change Syntax?

lisp when macro
emacs lisp macro when

you sometime hear lispers say that lisp macro/reader allows you to “change lisp syntax”.

Here's a quote from 〈Practical Common Lisp〉:

While Common Lisp supports both these methods of extending the language, macros give Common Lisp yet another way. As I discussed briefly in Chapter 4, each macro defines its own syntax, determining how the s-expressions it's passed are turned into Lisp forms. With macros as part of the core language it's possible to build new syntax--control constructs such as WHEN, DOLIST, and LOOP as well as definitional forms such as DEFUN and DEFPARAMETER — as part of the “standard library” rather than having to hardwire them into the core.

〔from http://www.gigamonkeys.com/book/macros-standard-control-constructs.html

Note how it says “it's possible to build new syntax”.

Here's from Clojure doc:

Clojure has a programmatic macro system which allows the compiler to be extended by user code. Macros can be used to define syntactic constructs which would require primitives or built-in support in other languages. Many core constructs of Clojure are not, in fact, primitives, but are normal macros.

Some macros produce simple combinations of primitive forms. For example, when combines if and do:

(macroexpand '(when (pos? a) (println "positive") (/ b a)))
;; (if (pos? a) (do (println "positive") (/ b a)))

Other macros re-arrange forms in useful ways, like the -> macro, which recursively inserts each expression as the first argument of the next expression:

(-> {} (assoc :a 1) (assoc :b 2))
;; {:b 2, :a 1}

 (macroexpand '(-> {} (assoc :a 1) (assoc :b 2)))
;; (assoc (clojure.core/-> {} (assoc :a 1)) :b 2)

〔from http://clojure.org/reference/macros

Note how it says “Macros can be used to define syntactic constructs which would require primitives or built-in support in other languages”.

actually, lisp macros cannot change syntax in any standard sense of the word “syntax”. For example, Lisp macros cannot change the nested parenthesis you have to type when coding lisp.

here, see Racket doc explains.

The macro facilities defined in the preceding chapter let a programmer define syntactic extensions to a language, but a macro is limited in two ways:

〔from http://docs.racket-lang.org/guide/languages.html

So, in the end, can lisp macro or reader really change syntax?

Not really. If you want python syntax, or haskell syntax, or Java syntax, lisp macro/reader's helpless. You gonna need to write a parser, just like any language.