Can Lisp Macro Change Lisp Syntax?

What Does “Syntax” Mean in Lisp?

The word “syntax” has special meaning in lisps. (no, we are not talking about the nested parenthesis.)

in other languages, such as Java Ruby JavaScript, the syntax of the language is what you see in source code.

in lisps, sometimes they call that surface syntax.

in lisps, there's a thing called “reader”, which is part of the interpreter/compiler. The “reader” reads in the source code, then turn that into a sequence of lisp primitives such as lists, symbols, numbers, etc. This, lispers think of as the lisp syntax.

See how Racket doc explains:

The syntax of Racket is not defined directly in terms of character streams. Instead, the syntax is determined by two layers:

• a reader layer, which turns a sequence of characters into lists, symbols, and other constants; and
• an expander layer, which processes the lists, symbols, and other constants to parse them as an expression.

Here is how Clojure doc explains:

Clojure is a homoiconic language, which is a fancy term describing the fact that Clojure programs are represented by Clojure data structures. This is a very important difference between Clojure (and Common Lisp) and most other programming languages - Clojure is defined in terms of the evaluation of data structures and not in terms of the syntax of character streams/files. It is quite common, and easy, for Clojure programs to manipulate, transform and produce other Clojure programs.

That said, most Clojure programs begin life as text files, and it is the task of the reader to parse the text and produce the data structure the compiler will see. This is not merely a phase of the compiler. The reader, and the Clojure data representations, have utility on their own in many of the same contexts one might use XML or JSON etc.

One might say the reader has syntax defined in terms of characters, and the Clojure language has syntax defined in terms of symbols, lists, vectors, maps etc. The reader is represented by the function read, which reads the next form (not character) from a stream, and returns the object represented by that form.

Here is how a popular Common Lisp book Practical Common Lisp explains:

In Common Lisp things are sliced up a bit differently, with consequences for both the implementer and for how the language is defined. Instead of a single black box that goes from text to program behavior in one step, Common Lisp defines two black boxes, one that translates text into Lisp objects and another that implements the semantics of the language in terms of those objects. The first box is called the reader, and the second is called the evaluator.

[from http://www.gigamonkeys.com/book/syntax-and-semantics.html ]

Lisp Macros Can Change Syntax?

you sometime hear lispers say that lisp macro/reader allows you to “change lisp syntax”.

Here is a quote from Practical Common Lisp:

While Common Lisp supports both these methods of extending the language, macros give Common Lisp yet another way. As I discussed briefly in Chapter 4, each macro defines its own syntax, determining how the s-expressions it's passed are turned into Lisp forms. With macros as part of the core language it's possible to build new syntax--control constructs such as WHEN, DOLIST, and LOOP as well as definitional forms such as DEFUN and DEFPARAMETER — as part of the “standard library” rather than having to hardwire them into the core.

[from http://www.gigamonkeys.com/book/macros-standard-control-constructs.html ]

Note how it says “it's possible to build new syntax”.

Here is from Clojure doc:

Clojure has a programmatic macro system which allows the compiler to be extended by user code. Macros can be used to define syntactic constructs which would require primitives or built-in support in other languages. Many core constructs of Clojure are not, in fact, primitives, but are normal macros.

Some macros produce simple combinations of primitive forms. For example, when combines if and do:

(macroexpand '(when (pos? a) (println "positive") (/ b a)))
;; (if (pos? a) (do (println "positive") (/ b a)))

Other macros re-arrange forms in useful ways, like the -> macro, which recursively inserts each expression as the first argument of the next expression:

(-> {} (assoc :a 1) (assoc :b 2))
;; {:b 2, :a 1}

 (macroexpand '(-> {} (assoc :a 1) (assoc :b 2)))
;; (assoc (clojure.core/-> {} (assoc :a 1)) :b 2)

[from http://clojure.org/reference/macros ]

Note how it says “Macros can be used to define syntactic constructs which would require primitives or built-in support in other languages”.

Lisp Macro Cannot Change Syntax

Actually, lisp macros cannot change syntax in any standard sense of the word “syntax”. For example, Lisp macros cannot change the nested parenthesis you have to type when coding lisp.

here, see Racket doc explains.

The macro facilities defined in the preceding chapter let a programmer define syntactic extensions to a language, but a macro is limited in two ways:

• a macro cannot restrict the syntax available in its context or change the meaning of surrounding forms; and
• a macro can extend the syntax of a language only within the parameters of the language's lexical conventions, such as using parentheses to group the macro name with its subforms and using the core syntax of identifiers, keywords, and literals.

[from http://docs.racket-lang.org/guide/languages.html ]

So, in the end, can lisp macro or reader really change syntax?

Not really. If you want python syntax, or haskell syntax, or Java syntax, lisp macro/reader's helpless. You gonna need to write a parser, just like any language.