racket2 lisp ditching parenthesis
racket scheme lisp ditching sexp
https://github.com/racket/racket/wiki/Racket2
[from https://groups.google.com/forum/m/#!msg/racket-users/HiC7z3A5O-k/XPR2wbSJCQAJ]
- Date: 2019-07-20
- by Matthew Flatt
This message is intended as a prose form of what I said at RacketCon, but it includes some extra explanation triggered by the discussion so far. Where that happens, I apologize that it isn't in the form of a more direct response in the original thread.
The Racket2 Idea
Racket's design and implementation is on solid ground, and from this point, it can continue to evolve and improve in many ways. No matter how Racket evolves, the community is committed to preserving what we have achieved: today's `#lang racket` programs will run in the future, and today's `#lang racket` modules can be used in future Racket programs. At the same time, the current language design may be close to a local maximum, and it's not in the nature of the Racket community to be satisfied with a local maximum.
By starting all programs with `#lang`, we have created a path to leap from one peak of design to a different, higher peak without sacrificing the old one and while staying Rackety. Roughly, that's what we mean by “Racket2”. The name “Racket2” stands for some language that builds on the current Racket technology and implementation, but that may not directly accommodate the same bindings and expression forms that appear in the body of a `#lang racket` module.
Although we could literally write `#lang racket2` for modules in the new language, “Racket2” is normally understood as a placeholder for a more distinct name. As a matter of branding, perhaps the language would be called “Racket X” for some good X — shortened to “X” in some contexts — and Racket X program would be written with `#lang X` line at the top. Maybe the name depends on how different Racket2 turns out to be from Racket, so it makes sense to pick a name later.
Venturing out from Racket's current peak opens a large space of possibilities, so the first step is to agree on a set of goals to be met by the next peak. The goals can serve as a starting point for developing roadmap for deciding on technical details in a follow-up process.
Possible Language Changes
The Racket community has long discussed possibilities for Racket2. Here are a few potential changes from the wish list:
* Rename functions to make the conventions more uniform, and make better use of keyword arguments to reduce names and increase consistency.
* Change structures types to have more or fewer features.
* Use generic datatypes (such as streams) pervasively, instead of writing code that expects a particular concrete representation (such as lists).
* Make strings immutable by default (e.g., result of `format` or `string-append`).
* Adjust the semantics of common syntax forms, such as raising an error on fall-through for `cond` or `match`, and change the meaning of some primitives, such as making `integer?` mean “exact integer”.
* Make pattern matching more pervasive and available by default.
* Change module, evaluation, and loading callbacks (e.g., protocol for `current-module-name-resolver`) to improve errors and extensibility.
More changes have been suggested, and no doubt many other changes would make sense. As a first cut, to synthesize some principles from these ideas — as a way of selecting which wishes should be tackled and how — the current wish list might be organized and summarized by two overall goals:
* Make the language more consistent
Consistency means making terminology and binding names more consistent, and it means unifying related syntactic constructs. For example, the `define` and `define-syntax` forms should perhaps be less syntactically distinct. Similarly, pattern matching and syntax matching seem closely related and could benefit from using the same pattern language. Also, pattern matching should be pervasively available among binding forms. Etc.
* Make the language constructs more generic
In many ways, the current Racket language and libraries encourage the use of concrete data structures instead of abstract datatypes. The `map` function works on lists, for example, not arbitrary streams, and even `for` demands concrete declarations like `in-list` to provide the best performance. Refining the capabilities of structures and providing a smoother path among dictionaries, structures, and objects may also fit into this category.
There seems to be a widespread agreement/assumption among the community that we could one day switch all the general documentation, such as the Racket Guide, to use `#lang X`. That would preserve the ability to run `#lang racket` modules and import them into `#lang X` modules, but it would effectively relegate `#lang racket` to a kind of compatibility mode. The Racket community has already done this a couple of times, moving from `(module m mzscheme ...)` to `#lang scheme` to `#lang racket`, and some of the Racket distribution is still implemented in `#lang scheme`. After another switch, the constructs for `racket` would remain documented and maintained, in the same way as the constructs for `mzscheme` and `scheme` remain documented and maintained, but `racket` and its documentation would fade to the background.
Based on the wish list, the scale of changes would likely be much larger than in previous transitions. There will many more changes to names and changes to the meaning of identifiers that are spell the same way. The scale of those changes doesn't seem to worry the Racket community.
Surface Syntax
You won't find a wish for significant changes the Racket surface syntax anywhere on the Racket2 wish list. Nevertheless, I've suggested tacking it on.
Why now?
If we decide that a change to the surface syntax is a good move, it would make sense to try that at the same time as overhauling names and syntactic forms. Pattern matching, for example, seems like a place where syntactic flexibility can help. Besides letting different kinds of changes reinforce each other, we'd revise all of the documentation just once instead of twice.
There are downsides to tackling both semantic and syntactic changes at once. If a change to surface syntax is on the table, though, then we should at least consider bundling the efforts.
Why at all?
Partly, I believe that a syntax with infix operators, fewer parentheses, etc., would be better. That's a matter of opinion, but at least it's an opinion backed by enough experience with parentheses that it can't be written off as uninformed. More significantly, parentheses are certainly an obstacle for some potential users of Racket. Given the fact of that obstacle, it's my opinion that we should try to remove or reduce the obstacle.
Syntax is not just an obstacle for potential users of Racket as a programming language, but also for potential users of Racket as a programming-language programming language. The idea of language-oriented programming (LOP) doesn't apply only to languages with parentheses, and we need to demonstrate that. Although we already have tools for parsing non-parenthesis syntax into parentheses and letting macros take over from there, that's not the kind of smooth path from simple extension to full language building that we have in the S-expression world. While these research problems may not be forefront on the minds of all Racket users, keep in mind that Racket exists because of research and remains supported almost entirely by research.
Adjusting the syntax of Racket to accommodate more users could be characterized as an effort to make Racket more popular. I don't think it's helpful to characterize the idea that way. Granted, any effort to reduce barriers is effectively an effort to make Racket more popular. In that sense, our efforts on Racket's documentation were an attempt to make Racket more popular. The work we put into building the Racket distribution and making it easy to install is an attempt to make Racket more popular. Creating the package system was an attempt to make Racket more popular. All strictly true, I suppose, but the characterization feels wrong. There's a difference between reducing technical barriers for users of various stripes versus marketing to make a product more popular. When I point out that parentheses can be a deal-breaker to people outside the Racket community, I do not have in mind that we just need a better marketing message; I have in mind that the obstacle is real, it affects many people, and we should address it.
Who?
From a community perspective, the rub is that the “many people” for Racket syntax is a barrier are mostly people outside the community. It's fair to ask who would benefit. It's difficult to ask a community to tackle a problem whose solution targets a different community.
There's also a question of who really needs to participate in the development of a new surface syntax. That part of the project could take place in its own `#lang` and stay out of the way of `#lang racket` users for years. As a practical matter, that will certainly happen, even if we think that that new syntax will eventually take over. We've already done some side experiments in separate `#lang`s, though, and now is a time to check whether there is community buy-in and participation for a more ambitious effort.
How?
Parentheses are definitely useful for Racket-style language-oriented programming, and parentheses certainly helped Racket's macro system get to where it is now. There would be no point to a change in syntax if it meant interfering with Racket's main capability. Although a macro system without parentheses is a difficult problem, I think we know how to solve it.
I have in mind “Honu: Syntactic Extension for Algebraic Notation through Enforestation”, GPCE 2012. It shows how we can bridge the relatively linear structure of non-() programs to the tree structure of S-expressions. Specifically, the bridge is called “enforestation”. It sits roughly between the reader and the expander, but enforestation needs to be interleaved with expansion to get the level of expressiveness that Racketeers would expect.
You may worry that I'm putting a lot of faith in one expriment and paper --- and a paper written by a PhD student and me, at that. You may be right. I have only my intuition that enforestation is the key to the problem of non-() syntax and extensibility. The details in the paper don't seem to me exactly right for Racket2, but it's a big first step toward addressing the problem, and it already seems sufficient. Others may have figured out even more. I wouldn't blame anyone for concluding that the difficulty of the task is too large for the size of the obstacle that it removes, but that's not how I size things up.
The gap between 2012 and now reflects that Racket had more problems (in my view) than just syntax, and for various reasons, those other problems took precedence for my own work. This also explains my answer to the “why now” question. The other problems seem to be resolved or well on the way, and so syntax seems like the next thing to tackle.
How to Proceed
Ideally, we would first decide on whether we want to try changing surface syntax ever. Then, only if we have enough consensus that it's worth a try, we could move on to setting overall goals and picking a roadmap. Finally, if the planing of the roadmap succeeds, we could follow it while getting into the details.
Unfortunately, it doesn't work like that. We need concrete examples to discuss the possibility of changing syntax, potential roadmaps to see whether there's anywhere we want to go, and so on. Some of the discussion will necessarily put a cart before a horse. Delving into some details may incorrectly give the impression that prerequisites are a foregone conclusion. We've seen all of that already, starting with my attempt to open up the discussion in the middle of RacketCon and probably continuing with this message. Perhaps we can just accept that this is all okay in the spirit of working together.
I originally thought that we should have a more specific process in place for discussion at this point, but I think I've been proved wrong. Let's just discuss, for now. I'll be posting more myself.
lol, the racket scheme lisp people are considering a new lang racket2 sans the paren. i don't understand why the lispers still don't get it after 30 years. what's wrong with McCarthy's m-expr as implemented in Mathematica by Wolfram?
also, i thought racket forks switched to python syntax lisp pyret few years back exactly due to inflexibility of sexp??
lol. racket scheme lisp is not dead yet. one hundred people are still using it. i truly find this funny.
those of you #racket #scheme #lisp coders concerned about racket2, read these:
i have a honest question for lispers. Why can't #racket2 do a syntax layer, the original mexpr proposed by McCarthy? it's well done in Mathematica. This way, you have 100% regular syntax like the sexp, while have infix for flexibility.
every time lisp syntax comes up in past 20 years, lispers have no idea what Mathematica syntax works (a actual mexpr). I think i've only known maybe 5 person did. Wolfram himself, Richard Fateman, and others i forgot. Any who coded Mathematica and lisp for few years would know, and i suppose there are at least few thousand of them in the world. But online mouthers are typically not.
if i say, any who haven't coded both Mathematica and lisp for 5 years do not qualify to voice opinion about lisp syntax. That's not exactly right, because the syntax layer idea, need not drag in Mathematica. Most lispers, even with 20 years coding various lisps, do not know the mexp and Mathematica idea. I think what actually happens is that, lispers, like other people, those don't know talk a ton, those who know are rare to be seen. Common Lispers fanatics are the worst type of this lot.
For example of people i think surely know mexp idea and Mathematica syntax system well are: Daniel P. Friedman, Matthias Felleisen, Matthew Flatt. They must have written on this, but it'd be hard dig out or ask them for answer about mexp. It'd be some 40 hours research.
what is lisp m-exp? what's syntax sugar, really?
Note about mexp. m-expression, is the original idea by lisp inventor John McCarthy. It was never implemented. McCarthy's mexp is never specified in detail. Wikipedia today has some detail. The syntax system in Mathematica, is a actual mexp/sexp system. Mathematica is based on lisp and APL.
Many coder don't understand so called “syntax sugar” in detail. To them, anything similar is “syntax sugar”. In the most strick sense, syntax sugar is syntaxes that are syntactically equivalent.
For example,
JavaScript
x = 3 vs x=3, or lisp
(quote x) vs 'x, or
in Mathematica
Plus[3,4] vs 3+4.
Syntactic equivalence is such that any are 100% syntactically and semantically equivalent.
you can press a button in editor and whole source code can be transformed from one to the other.
In contrast, C's
x = x + 1 and
x++ are not syntactically equivalent, and not semantically equivalent.
Another example, in JavaScript
function f (x) {…}
vs
f = function (x) {…}
are not syntactically equivalent, and not semantically equivalent.
why don't lispers adopt the m-expression idea as in Mathematica? my hunch is that it's shouted down by loud lisp fanboys, as this is 1 of thing that identify lisp, identity crisis situation. Request for alt lisp syntax happen every year in past 20 years. purely nested syntax has practical issues. see LISP Infix Syntax Survey if haven't yet. 😁