Over the past 7 months or so, in addition to learning and using Common Lisp, I have been giving thought to the social issues affecting Common Lisp and its community. Over the past two months, I have been working on two projects, described below, to try to address the issues that most affect newcomers.

To gain perspective on the causes and social context surrounding these issues, I have been studying Common Lisp’s history, reading (among other things) the historical accounts of Richard Gabriel, Guy Steele, Kent Pitman, and others. (I can recommend, for example, The Evolution of Lisp, A Pattern of Language Evolution, A Critique of Common Lisp, Common Lisp: The Untold Story, and of course Lisp: Good News, Bad News, How to Win Big.) I have also been reading Common Lisp the Language, 2nd Edition and the Common Lisp HyperSpec, scrutinizing every inch of the language itself.

My research has given me a broader and more informed perspective than I had before. Before, I could see that the social climate surrounding CL was slowing its adoption rate and holding back progress, but I assumed its roots were healthy and strong. But now, it seems that CL is subsisting mainly on a fleeting aura of mystique, while other languages are surpassing it both technically and socially. That situation is untenable. Unless something changes, I don’t see how CL can have much of a future, except as an amusing curio for fans of antique computer technology.

If that was the only thing I had realized, it would have just strengthened my resolve to address the issues head-on. But, I also realized than CL is not the culmination of Lisp’s evolution, it is merely one box in a vast flowchart. It is a long-lived and popular dialect, and its creation was a significant milestone in Lisp history, but aside from historical interest CL is not an especially remarkable Lisp. It is a middle-of-the-road dialect, just as you would expect from something designed to be a compromise among numerous competing dialects.

CL is a fairly solid practical language, but it could have been much better. Inter-dialect politics, and the need to appease established interest groups, prevented the language from reaching its potential when it was created. Similar attitudes have prevented the core language from being cleaned up in the past 20 years, and likely will continue to do so indefinitely. There has been some activity building improved languages on top of CL, but there are better languages (particularly Scheme) to be foundations for that.

These realizations prompted a crisis for me. A month ago, I was ready and willing to dedicate myself to building a place for CL among a new generation of programmers. But now I’m not so sure. On the one hand, I have invested a lot of time into CL already, and it’s not a bad language, just not very exciting or inspiring. I could learn to live with it, and even smooth out most of its warts. On the other hand, there are certain attitudes that I find extremely demotivating and frustrating, and those attitudes seem to be more prevalent in the CL community than in many other languages’ communities.

To put it simply, the CL language and community both fall far below their potentials. It might be possible to shift the culture in a more positive direction, but that would require a sustained effort by many people, and would probably involve frequent stressful confrontrations with the “worst offenders” in the community. There are decades of inertia and ingrained attitudes that would not be easily changed.

The reward for all that effort would merely be preserving a lackluster language, a “fixer-upper” that would require further effort to turn it into something really worth using. And even if the core language were changed, CL would lose its mystique and become “just another Lisp”, unable to use longevity or historical significance as a crutch. It would have to compete with other languages solely on its actual merits, which nowadays are not as strong as CL proponents like to believe.

I enjoy a good challenge, but this seems more like a fool’s errand. There are other languages with healthier communities, more momentum, cleaner cores, and features on par with CL. So I have to ask myself, why bother with CL?

I would love to discover some factor I have overlooked, some redeeming quality that would make CL worth fighting for. But the harder I look, the less likely that seems.

The Next Generation of Common Lispers?

There are numerous issues affecting CL, but one that I thought I might be able to make an impact on was its low adoption rate. CL has a smaller user base than many other languages, as well as a smaller influx of new users, and (I suspect) a lower retention rate among those new users. This issue has many underlying causes, most of them social. There are a few technical issues, but they would be easily solved if the social issues did not prevent it.

(Some CL proponents insist that a low adoption rate is not a problem. The usual argument is that it is better to have a small number of highly skilled and intelligent users than a large number of mediocre users. But that is a false dichotomy, a cop-out to avoid actually addressing the problem. And even if it were logically valid, it is based on the assumption that CL is attracting and retaining users based on skill and intelligence. I would argue that the social climate surrounding CL merely favors people with certain personality traits, regardless of skill or intelligence.)

Many people in the CL community have long had an attitude of, “We do not need to make an effort to attract the next generation of programmers, or to make things easier for them. CL is so amazing that the best programmers will naturally seek it out and devote themselves to learning and using it.” That attitude may have been justified at some point in the past, but it is certainly not anymore.

These days, there are numerous languages whose technical merits are on par with CL, and many of them have more positive social climates, more forward-looking attitudes, and more effective ways of welcoming, supporting, and educating new users. CL is barely competitive from a technical standpoint, and significantly outclassed from a social standpoint.

Aside from a few unusual features, the only significant attraction CL seems to have left is the exaggerated mystique created by the writings of Paul Graham and other vocal Lisp advocates. Even that won’t draw new programmers to CL for much longer, especially with languages like Clojure and Racket offering the “Lisp Experience” with more useful and relevant built-in features, better community management, and more momentum.

“Handling Newbies Effectively”

The first project I planned to release was a paper highlighting and analyzing the various social issues affecting CL newbies, and offering practical advice for addressing those issues. These issues include not only the apathy (and occasionally outright hostility) that many oldbies show toward newbies, but also the inadequate social framework the community has in place for welcoming, orienting, supporting, and educating newbies.

For example, CL has no central location where newbies can find reliable information, documentation, or other learning resources. Instead, information is scattered across the web, with no way for newbies to know if it is up to date and reliable. (I’m sure many CL proponents are now itching to “correct my ignorance” and explain that CL is just a language standard with many different implementations, which is why there is no central website. Well, guess what? Newbies don’t know or care about that. They want reliable and helpful information, not excuses for why the community is so disorganized.)

There are also not many good places for CL newbies to get support from other Lispers. The only well-established venue I know of that caters to newbies is LispForum.com, and it does not seem to get much promotion or attention from the broader community. So instead of newbies finding someplace where they are welcome, they usually end up someplace where they are barely tolerated, such as the #lisp IRC channel or comp.lang.lisp newsgroup. Those are probably the two least hospitable places for newbies, yet they are the two most popular and widely promoted venues, so many newbies end up there anyway (much to the vexation of some oldbies). Granted, #lisp and comp.lang.lisp can be great if you need answers to specific technical questions, but they are really lousy if you need orientation, advice, encouragement, or other forms of “hand holding”.

I have also observed many negative social attitudes which, while certainly not universal among CL programmers (and not unique to the CL community), are prevalent enough that they affect the entire community, and make the social climate less hospitable to newbies. A few of the most common and most harmful attitudes are defensiveness, nitpicking, zealotry, arrogance, lack of empathy, and social dominance based on a narrow set of values. Social dominance is an especially significant factor, because it means that the “alphas” in the community cannot be censured for bad behavior, unless those behaviors are truly egregious. (For example, if an alpha were to provide a newbie with “sample code” that would in fact delete all files on the newbie’s computer, the worst reprimand the alpha can expect is a few people saying, “Tsk tsk, you know you shouldn’t do that”. But just as many people would consider it an amusing prank, albeit cliché. They might even congratulate the alpha on teaching the newbie a lesson, thereby negating any reprimand the alpha received.)

The paper I’ve been working on goes into much greater detail, and gives more examples of challenges newbies face. But, I have started to doubt that publishing it would help the CL community at all. Many of the most serious social issues that affect the CL community are older than CL itself. Many people over the years have pointed them out, yet the same issues remain. So what good would it do to point them out again? The people whose attitudes and behaviors are most harmful, are the least likely to listen. The people who would listen, have probably heard it before. Why bother?

Yet, it’s obvious that the situation cannot possibly improve if no one speaks out. Sweeping things under the rug doesn’t make them go away. As uncomfortable as it may be, the only hope for addressing the issues is to start discussing them openly. Indeed, perhaps the most important social issue affecting the CL community is widespread complacency and apathy, which prevent the other issues from being addressed.

I will probably publish my paper eventually, because the same issues affect many tech-oriented communities to some degree. The CL community may be too far gone, but my paper might do some good elsewhere.

CLUEL

The other project I have been working on is CLUEL, the Common Lisp Usability Enhancement Layer. Whereas my paper would highlight the social issues and challenges newbies face while learning Common Lisp, CLUEL would directly tackle the problems that make the language itself unnecessarily difficult to learn and use.

About two months ago, I described some of the historical baggage that makes CL harder to learn, and adds to the mental burden on all CL programmers without adding any real value to the language. Since that time, I have spent over 100 hours poring over the CL language (not including the time spent researching CL’s history, writing about social issues, or planning CLUEL’s project infrastructure). I have personally looked up the documentation for every one of the 987 symbols in CL. I certainly can’t remember every detail, but I can truthfully say that I have some knowledge of the purpose of every single function, macro, special operator, global variable, and constant in standard CL.

In the same way that researching Lisp history has given me a better understanding of CL’s origins, scrutinizing the language itself has given me a better understanding of how inconsistent and crufty it is. Depending on how forgiving you feel, there are between a few dozen and a few hundred examples of features whose names are inconsistent, obscure, misleading, or otherwise poorly chosen. There are numerous features that are redundant, obsolete, or rarely needed, yet they are given the same prominence as the most essential parts of the language, because everything is lumped together in a single package.

Frankly, CL is just not a well designed language. That fact is not a reflection of the people involved, who as individuals were quite intelligent and capable. It merely reflects the process used to design the language. Considering that CL was debated in committee for 10 years, with numerous competing interest groups each tugging in a different direction, I’m impressed that it didn’t turn out even worse than it did. (I’m even more impressed that the whole standardization process didn’t fall apart entirely. Although I sometimes wonder if that might have been preferable in the long run.) However, there is a big difference between a language being “not bad, given the circumstances” and it being “well designed”.

The most obvious solution to the problem of it not being well designed, would be to revise the language standard to make it better. But that would be a futile and misguided effort. Even if you could get enough people in the CL community to agree that revising the standard was worthwhile, there would be just as much debating, bickering, and nitpicking as there was during the original standardization process. A committee is just not a good mechanism for designing a coherent language (or much of anything else, for that matter).

A better approach, more practical and more likely to yield real improvements, would be to create a new package (or multiple packages) that can be used as a test bed for experimenting with the language. It’s easy to start, you don’t have to create widespread consensus, and it doesn’t disrupt anyone who doesn’t want to use your package. This approach has been recommended by quite a few knowledgeable people, including Peter Seibel at ILC 2010 (near the end of his talk). This also harkens back to the tradition of prototyping new dialects on top of existing dialects (although I do not consider CLUEL to be a new dialect).

That approach is exactly what I had planned for CLUEL. The process I used for creating the proof of concept was something like this, although with lots of jumping back and forth between steps:

1. Partition all standard CL symbols into packages based on a theme or concept. At last count, I had tentatively settled on 22 packages: core, general, types, cons, sequence, array, string, stream, file, read, write, symbol, package, clos, structure, hash-table, math, bitwise, condition, debug, host, and time. Some symbols belong to more than one package.
2. For each package, partition the symbols into layers based on how useful they are:
   • Symbols that are “often useful” stay in the main packages, which are available by default when using CLUEL.
   • Symbols that are “sometimes useful” are moved to the extra packages, which can be included when needed.
   • Symbols that are “rarely useful” are omitted from CLUEL entirely, but could still be accessed via the standard CL package if desired.

3. For each symbol in CLUEL, decide whether its name is sufficiently meaningful, memorable, and consistent with the rest of the language. If it is not, give it a new, better name. (I developed a way to easily create aliases for existing functions, macros, special operators, global variables, and constants.)

   This involved a lot of jumping back and forth, because renaming one symbol might affect the consistency of other symbols. Or, I might discover some pattern or naming convention that affects symbols I had already processed, so I would have to go back and rename them.

   This step also involved a lot of consulting the documentation to make sure I understood exactly what an operator did. That led to a few surprises. For example, I didn’t realize that ldiff and tailp are so closely related. Indeed, their names suggest that they are entirely unrelated, which is a sign that they are poor names (or at least, ldiff is). Those functions were likely invented independently, and they make sense individually, but when you put them together their names (and argument orders) are inconsistent.

4. Define a handful of new functions and macros to improve consistency, “fill in the gaps”, or improve usability in other ways. For example, there is no non-destructive sort function in standard CL, so I added a simple one. It may seem counterintuitive, but adding a new function in cases like this can reduce the cognitive load for users, because it makes the rules and patterns more consistent, with fewer exceptions to learn and remember.

   Another way to improve consistency and reduce cognitive load is to consolidate many similar functions into one generic function. For the proof of concept, I experimented with this for accessing containers (consolidating nth, elt, aref, svref, char, schar, and gethash) and counting their contents (consolidating length, array-total-size, and hash-table-count).

If that process seems to involve many subjective judgement calls, that’s because it did. But this was merely the proof of concept, to demonstrate that this sort of endeavor is feasible. In the long run, the initial design of CLUEL is irrelevant, because it was intended to be changed soon, with the involvement of more people. The more important aspects of CLUEL, its cultural values, emerged as I was planning the project and working on the proof of concept:

• A focus on the needs of programmers of many different experience levels. That means, among other things, going out of our way to get feedback from newbies. Contrast that with Common Lisp, which was designed mainly to satisfy the needs of the kinds of people on the committee: expert Lisp hackers with code bases written in earlier Lisp dialects, and the implementators of those dialects (who were also expert Lisp hackers). A well-designed programming language (or nearly any other kind of tool) can be suitable for newbies and experts alike. It is not necessary to “dumb it down” to make it approachable for newbies.
• An emphasis on user-centered design. Many of the same principles and techniques that are used to design doorknobs, toasters, electronic gadgets, and software user interfaces are just as applicable to designing programming languages. Some established usability testing techniques would be difficult to run, because of the time investment involved in learning a programming language, but other testing methods are applicable, and new testing methods can be invented to suit the new domain.
• An emphasis on iterative design and change. Good user-centered design requires the ability to field-test a design, gather feedback, and incorporate that feedback as changes into the next round of design. With most programming languages, the cost of changing the language (once it is out “in the wild”) are very high, so design iterations are usually measured in years or decades, if they occur at all. I planned CLUEL to support multiple API editions, so that each new API edition could potentially be quite different from past editions, without breaking any software that depended on past editions. (Once an API edition is released, it would never be changed in a way that breaks compatibility, only to fix bugs or improve performance.) And, I extended defpackage to make it very easy to define new editions based on older ones, so the costs to iterate are very low.
• A culture of collaboration, support, and respect. Beyond addressing purely technical issues, CLUEL would be a chance to lead by example in addressing social issues. Besides being a good thing on general principle, having a positive community atmosphere would also be a practical necessity for CLUEL. It is impossible to perform user-centered design in an atmosphere of self-centeredness, defensiveness, or resistance to change. You must be willing to look beyond your own needs and preferences, accept feedback and criticism of your design, and change whatever needs to be changed. This naturally means you must embrace diversity, respect other people’s viewpoints, and earnestly consider even ideas that you don’t yourself like. And, because newbies are included in the target demographic for CLUEL, you must give attention and effort to supporting them, mentoring them, and soliciting their feedback.

It’s important to point out that CLUEL was not designed to be a new dialect, or to ever exist separate from standard CL. Like it says in the name, CLUEL was a “usability enhancement layer”, a more usable and well-designed language interface for writing Common Lisp software. CLUEL would not add any new semantics or functionality to CL, just present (a subset of) the existing functionality in a better way.

A key feature of CLUEL was that code written in CLUEL would be fully compatible with code written in standard CL. So, for example, an application written in CLUEL could use libraries written in standard CL, and vice versa. (Unfortunately, that makes it impractical to address certain fundamental issues, such as the messy semantics of nil.)

I really poured myself into CLUEL. I estimate that over the past 2 months, I’ve spent over 100 hours studying the CL language and considering how to make it more usable, another 100 hours researching Lisp history and other dialects to see what has been tried before, 60 hours planning project organization and policies, and 20 hours developing infrastructure for packages, aliases, etc.

Indeed, the code for the first iteration of CLUEL, the proof of concept, is about 80% done, and the project planning is about 60% done. But I don’t plan on finishing it, or even releasing the in-progress code. It hurts to throw something away after spending so much time and effort on it, but at this point that seems to be the best course of action.

Why? Because CL doesn’t seem worth it anymore. I certainly don’t want to potentially spend a few years maintaining CLUEL and building up its community, shackling myself to a language I don’t really care about. Worse, I suspect that even handing off CLUEL for someone else to run it would be a mistake, because it might prolong CL’s decline, and delay a better Lisp dialect with a healthier community from rising to prominence.

I’m not sure what dialect that might be, but the two main contenders, Scheme and Clojure, are both promising. They both have cleaner core languages, more momentum, and (from my initial impressions, at least) healthier communities than CL. I don’t know yet how prevalent the negative attitudes I’ve observed in the CL community are in the Scheme and Clojure communities, but at least they do not seem to be so inhospitable to newbies, which is a good sign.

(I have heard some CL proponents insist that Scheme and Clojure are not really Lisp dialects, or at least are not as Lispy as CL. I’ve even given that impression myself on at least one occasion. But that is just ideology, with no grounding in reality. There have been many different Lisp dialects over the past 60 years, on many different architectures, with many different features and syntaxes. There is no rational basis for claiming that CL is a more “true”, “pure”, or “faithful” Lisp dialect than Scheme or Clojure.)

I don’t currently have any plans to create something like CLUEL for Scheme or Clojure, for the simple reason that they don’t seem to need it. Both languages are much more well-designed, consistent, and usable than CL is. But they are certainly flexible enough to implement a similar “usability enhancement layer”, should I happen to discover that implementing one would be beneficial.

What Now?

Despite my concerns about Common Lisp, I haven’t completely ruled it out as a language choice for Ambienome or future projects. I’m keeping my mind open to the possibility that CL has some redeeming characteristic or feature that makes it worth using, maybe even worth fighting to preserve.

Lately, I have been investigating other languages, including Clojure, Racket (a R6 Scheme implemention), and Chicken (a R5 Scheme implementation). Clojure is quite interesting, and there are a lot of cool things being created with it, but I’m concerned the emphasis on functional programming and immutable data structures might make an already ambitious game project even more challenging. Racket is very polished and featureful, with great documentation, but it is licensed under the LGPL. Ambienome would be a commercial game, and although the LGPL probably wouldn’t be a problem, there are certain ambiguities that make me hesitate to use it. Chicken is not as polished or featureful as Racket, but the community is quite friendly and welcoming, and its close integration with C might make it easier to use various game libraries or interface with the OS.

I am also considering non-Lisp languages, like CoffeeScript with Three.js (to make a browser game), or even going back to Ruby (perhaps JRuby with LWJGL). If possible, though, I’d like to keep using some Lisp dialect, because I have found Lisp to be conceptually powerful and enlightening.

Maybe, just maybe, I will start to miss something from Common Lisp, something that cannot be easily recreated in those other languages. But even if I don’t end up using CL, I don’t regret learning it. The experience has been very educational.

That’s no good, so I wanted to take some time to set my thoughts and ideas for Ambienome onto paper (so to speak). This will help me slide back into that mode of thinking, hone my ideas by putting them into words, leave a journal of how the game evolves, and maybe spark some interest in anyone reading this.

Caveat: you should not interpret any part of this as a promise or firm description of how the actual game will be. I’m just unloading my thoughts and ideas as they exist now. Everything is still subject to change, etc.

Creation vs Play

One thing I have not yet decided is how much focus will be on creating content, and how much focus will be on playing (or consuming) content. In other words, is Ambienome a game with a built-in creature creator, or a creative tool for designing interactive motion graphics and audio? Does the player create things to play with, or is the act of creation itself the gameplay? Where do I strike the balance?

I think it should lean more towards being a game that you can create custom content for. A typical user might spend two thirds of their time playing with user-created content, and one third creating new content. Naturally, individual users will have different interests and focuses, but I really want creating content to be accessible, highly interactive, and engaging. Spore,Minecraft, and Little Big Planet are all excellent examples of this.

As I’m envisioning it now, users would create scenes, which would have various creatures as well as level geometry (walls, etc.) and maybe some non-creature objects/items. The game would come with a variety of pre-built creatures, and the user could also build their own from scratch or by modifying an existing creature. Creatures could be saved and re-used between scenes, exported or imported as files, and easily shared online. Likewise, scenes could be exported, imported, and shared online.

Ambienome would likely come with a few dozen pre-made scenes that gradually introduce new creatures and gameplay mechanics, as puzzle games often do. Some of the scenes would have specific puzzles or goals to solve, while others would be “free play” mode like Electroplankton, where the only goal is to have fun playing with the creatures.

Gameplay Mechanics

There are many possible gameplay mechanics, ways in which you can interact with the scene, and creatures can interact with each other. Here are some that have been floating around in my head:

• Light/darkness (bioluminescence, light sensitivity)
• Heat/cold (energy transfer)
• Attraction/repulsion (magnetism)
• Falling/floating (gravity, buoyancy)
• Water/air currents (lines, splines, vortices)
• Bubbles (for air supply, or carrying objects)
• Sound pitch/harmony
• Playing a melody (correct notes in correct order)
• Physical contact (collision, bouncing, overlapping)
• Eating food or other creatures
• Life cycles (birth, growth, death)
• Controlling creature directly with keyboard, mouse, or joystick
• Creatures follow mouse pointer
• Clicking and dragging to move creatures and objects

These mechanics can be combined in interesting ways to create fun puzzles and free play scenes. For example, there might be a puzzle scene where you guide a creature to find some fruit and bring it back to the creature’s hungry offspring. But of course, there are obstacles in the way! First you have to bump into some singing sea cucumbers that each play a different note and glow a different color. When you play the correct melody, a hot magma stone appears. You carry the stone to a dormant hydrothermal vent, thereby heating up the vent. When the vent is hot, it produces a rising current which carries you up to reach some fruit hanging on a vine on the ceiling. You carry the fruit back to the hungry offspring, thus completing the scene. (This is all 100% scientifically accurate, of course. I saw it on the Discovery Channel. Honest!)

The game mechanics can also be used for free play scenes. One scene might have some jellies whose long tendrils ring like wind chimes when they move. When you move the mouse cursor, it creates a brief current, stirring the tendrils and making relaxing sounds. Perhaps there are schools of tiny fish hiding among some kelp, but they are attracted by the sound of the jellies’ tendrils, so they emerge and start swimming around, leaving colorful trails as they move. There’s no goal to accomplish or puzzle to solve, you can just relax and enjoy the sights and sounds for as long as you wish.

Naturally, all the creatures and game mechanics used in the scenes that came with the game would also be available for users to create their own scenes. And, players would be able to use those scenes and creatures as starting points for their own creations. Maybe you want to add some hydrothermal vents from that puzzle scene to create warm water currents that stir the jellies’ tendrils in the free play scene. Just press a button to switch to “create” mode, drag some vents into the scene, and voilà.

Ideally, creatures and scenes uploaded by other users would also be available for creating derivative works. But, that starts to get into some complex issues. How much control do users have over the license? Do all uploaded creations have to be under a certain license (perhaps some form of Creative Commons)? What about creations they don’t upload to a central service, just export as files? Can they choose “no derivatives”? Would the game try to enforce the license, for example by not allowing other users to modify scenes marked as “no derivatives”? These are some of the issues I’ll probably have to think about eventually. But, thankfully, not today.

I have approached CL from nearly 8 years of programming in Ruby. I mention this because my experience with Ruby has certainly colored my impressions of CL, and many of my observations are about how CL compares to Ruby.

Different Yet Similar

Overall, I’d rate Ruby and CL as being about equal, on my personal, subjective scale of programming language goodness. They differ in many ways, both good and bad, but it feels to me like the differences balance out. So, the two languages are different, but neither one is clearly superior.

On a conceptual level, the two languages are actually fairly similar. They are certainly more similar to each other than either one is to a language like C, for example. They both have an interactive prompt, automatic memory management and garbage collection, first-class functions, anonymous functions and closures, arrays and hash tables, mutable strings, arbitrarily large integers, namespaces (modules in Ruby, packages in CL), and class-based OOP systems that don’t force you into an OOP style. There are probably many other smaller similarities that I could think of if I took the time. (A dedicated Lisper would point out that many of these concepts originated in some Lisp dialect or another.)

Naturally, the details of many of these things are different. For example, CL’s packages work very differently than Ruby’s modules, despite fulfilling the same basic purpose. But in general, the concepts in each language are similar enough that knowing one language will make it significantly easier to learn the other.

Strengths (In Brief)

There are several aspects of Ruby that I feel are especially strong relative to Common Lisp. Stated briefly, they are:

• Community (active, supportive, sharing)
• Approachability (easy for someone new to get started)
• Convenience (many useful constructs and patterns are built in)
• Consistency of the object model (e.g. everything has a class)
• Wide variety of available libraries (both standard and installable)

Of course, Common Lisp has several areas where I feel it has the advantage:

• Flexibility (macros and read macros)
• Generic functions system
• Conditions and restarts
• IDEs, built-in documentation
• Optimization hints

I describe these in more detail near the end of this post.

Weaknesses

I’d like to be able to make a list of the areas each language is weak, but I’m too close to Ruby because of my years of use, yet also distant from it because I haven’t used it actively for perhaps a year. Both of those situations affect my perspective and make it hard to see Ruby’s real flaws.

Of course, Ruby has some weird bits of syntax (like {|args| ... } code blocks) and other quirks, but no more so than most languages, including CL. Ruby’s performance is traditionally a bit weak, but that varies across implementations, and is improving in leaps and bounds. The community is generally pretty great, although I find the “brogrammer” trend in some parts of the community to be distasteful and regressive.

My relationship to Common Lisp is a different situation. My eyes are still fresh and I’m actively using it, so I can see and describe Common Lisp’s shortcomings pretty clearly. But, I want to emphasize that just because I can still see the flaws in Common Lisp, that doesn’t mean there are no flaws in Ruby. I’m sure someone else could point out just as many problems in Ruby as I can in Common Lisp.

So, with that caveat, I’ll describe what I consider to be Common Lisp’s problem areas.

Minor Problem: Batteries Not Included

Ruby, Python, and other “batteries included” languages come with all kinds of nifty features (e.g. networking, regular expressions, XML/JSON/etc. support) right out of the box. A programmer coming to CL from one of those languages might expect CL to be the same way, but it’s not. Why?

The main reason is that CL has a formal ANSI standard, designed in the 1980s and early 1990s as a way to consolidate several different Lisp dialects. It took 10 years, countless man hours, and hundreds of thousands of dollars in direct expenses to finish the standard. Every detail was weighed and debated to make sure it was acceptable to the many people and groups with a vested interest in the process, such as the vendors who sold implementations of the old dialects.

So basically, only features that were acceptable to everybody were put in the standard. Things that they disagreed about, or that would put too much burden on the implementors, or that didn’t even exist back then, are not in the standard. Adding a new feature to the standard now would probably take many more years of debate and nitpicking, and would only succeed if practically everybody agreed that it was such a great feature that it is worth the effort. In other words, it would take a miracle and a half.

That is a very different scenario than you have with Ruby/Python/etc., where someone can submit a new feature to the most popular implementation, a core dev likes the features and commits it, and it’s released in the next version.

So, why is this not a huge deal in Common Lisp?

Well, most features can be written as libraries that you can install when you need them. CL is so flexible that even major changes to the language syntax can be installed as libraries! And with the advent of Quicklisp, installing libraries in CL is just as easy as installing gems in Ruby, or eggs in Python.

As for the features that can’t be written as libraries, individual CL implementations can provide them as extensions. For example, the implementation I use comes with a bunch of goodies that aren’t in the ANSI standard, such as threading, networking, code coverage and profiling tools, and a foreign function interface (FFI) for wrapping C libraries (such as OpenGL, as I’m using for Ambienome). If a feature is popular and many implementations provide it, someone will usually create a wrapper library that smooths out the differences between implementations, so that programmers can use the feature without being tied to one particular implementation.

(That’s an idealized scenario, of course. It doesn’t always work out so smoothly. Some implementations don’t add new features very often, if ever. Sometimes an implementation adds a feature in a way that isn’t compatible with other implementations, making it difficult or impossible to create a portable wrapper library. In such cases, if a programmer really needs the feature, they might target a specific implementation, rather than trying to keep their code portable. If they really need their code to be portable too, they can write code that acts differently for each implementation.)

That process of adding a feature to an implementation, letting other implementations copy it, and writing a wrapper library on top, is far from perfect. But, it’s faster, less onerous, and more flexible than trying to update the ANSI standard, and the results are usually satisfactory.

So, once you realize the implications of Common Lisp being formally standardized, and find out that there’s an easy way to install libraries, it’s not such a big deal that Common Lisp does not come with “batteries included”.

But, it’s still a minor problem that affects new Lispers coming from other languages. It might be possible to alleviate the problem through expectations management. If new Lispers understood from the start that many features are not part of the standard, but nevertheless are readily available as libraries, they wouldn’t be so surprised and disappointed.

I’m not sure who would be most effective at this sort of expectations management, though. Implementations? Book authors? Teachers? Bloggers? Wiki editors? IRC and newsgroup participants? I suspect all of the above are already doing it to some degree, yet many newcomers are still surprised. This is one area where having an “official website” for Common Lisp might make things easier.

Moderate Problem: Historical Baggage

Common Lisp has a lot of historical baggage: things that are done a certain way just because that’s how they were done in the past, even if they don’t make much sense nowadays. Perhaps that’s only natural for a programming language with such a long history. CL itself is nearly 20 years old (or 30, if you count from the first edition of Common Lisp the Language), and it inherited baggage from several older Lisp dialects, going all the way back to the original Lisp over 60 years ago.

That’s not to say that other languages don’t have some historical baggage of their own. For example, Ruby and many other languages inherit the bizarre old syntax of putting a 0 (zero) in front of a number to interpret it as octal form. (E.g. 10 means ten, yet 010 means eight. Surprise!) CL seems to have an especially large amount of baggage, though, and many Common Lispers seem to cling to that baggage rather tightly.

Historical baggage manifests itself in CL in many ways. One way is as individual quirks, like the setq operator. Setq was, I hear, originally invented as shorthand, so you could write (setq x 3) instead of (set (quote x) 3). But that was before the read macros were available; these days you can type (set 'x 3), which also means (set (quote x) 3) and is not any more characters than using setq. (Although, setq works a bit differently in CL than in earlier Lisps, so (setq x 3) isn’t quite the same as either (set (quote x) 3) or (set 'x 3) anymore.) Furthermore, CL has the setf operator, which can do everything setq can do and more, so there’s not really any reason to retain setq. It’s just historical baggage, kept around because that’s what they used in the old days. (Setq is still quite widely used today, usually with the rationale that it expresses the programmer’s intentions more clearly because it can do fewer things than setf.)

Historical baggage also manifests as old idioms that affect many parts of the language. Take for example the idiom of adding “p” to the names of predicate functions, functions that check something and return true or false (actually t or nil, another bit of historical baggage). For example, (evenp x) returns true if the value of x is an even number. It could have been (even? x), which I would argue is significantly clearer. But, Lispers used “p” in the old days, so (most, but not all) predicate functions in the CL standard use “p”, and therefore most Common Lispers still use “p” when they write their own predicate functions today.

Historical baggage also manifests itself as strange inconsistencies and curiosities in the underlying architecture. For example, Common Lisp has types, which it inherited from older Lisp dialects. It also has classes, which were added fairly late in the ANSI standardization process. Types and classes are very similar in many ways, but just different enough that they can’t be unified without breaking tons of old code. As a result, CL has both systems existing simultaneously, where each class has a corresponding type, but not all types have a corresponding class. Some features use types (like function and variable type declarations), while other features use classes (like generic function dispatching). I suppose the standards committee made the right choice given the circumstances back then, to avoid breaking all that old code. But, it is nevertheless historical baggage that all Common Lispers still carry nearly 20 years later.

Finally, historical baggage manifests itself as concepts and features that were useful many decades ago, but are pretty much obsolete today (and in some cases were already obsolete when the standard was written). For example, every symbol in Common Lisp has “properties”, a list of data that the symbol carries around in its guts. But nowadays, it would often be just as efficient to store that data in hash tables with the symbol as the key, and doing so would entirely avoid the possibility of two unrelated pieces of code coincidentally using the same property names. But, Lisp has had symbol properties since the very beginning, so Common Lisp has symbol properties too, along with the half-dozen functions used to manipulate them. (Symbol properties don’t seem to be used much anymore.)

These kinds of historical baggage aren’t a serious problem, because for the most part you can ignore the obsolete features, and either cover up or learn to live with the weird idioms and inconsistencies. But it is a moderate problem, because this historical baggage adds to the mental burden of every Common Lisp programmer in the world. (There are many, many other examples of historical baggage that I have omitted for the sake of brevity.)

It’s especially burdensome for new Lispers. Year after year, new Lispers have to go through a kind of rite of passage, learning the obscure idioms of years gone by, separating the useful features from the cruft, and trying to remember function names that seem arbitrary and inconsistent. Many of them give up and leave for other languages because of needless obstacles (this being just one of many they face). Some of them could have contributed a lot to the community over time, if only “the wall” hadn’t been built so high.

Serious Problem: The Community Atmosphere

That brings me to the most serious problem Common Lisp has: the community’s atmosphere, its prevailing moods and attitudes. This is such an important topic that it deserves its own post, but I’ll summarize the problem here.

I’ll admit that I started learning CL with the knowledge that many people (usually people who tried to join the community but were repelled) consider the community to be antagonistic, especially towards newcomers. So, I may be exhibiting some confirmation bias: seeing what I expected to see, and tending to ignore evidence to the contrary. But with issues like this, the widespread perception of a problem can be just as damaging as the reality of the problem itself.

I suspect that most people who use Common Lisp, as with any language, are probably decent folk who just want to write nifty code without a lot of fuss or drama. But these people are not very visible or active in community discussion venues (e.g. the comp.lang.lisp newsgroup or #lisp IRC channel). They’re off somewhere else, writing their nifty code in peace.

Alas, many of the people who are highly visible and active can best be characterized as “toxic”. These are people who, because of the nature of their personalities and attitudes, have a consistently negative emotional effect on the people they interact with. Without necessarily intending to do so, they have created a constant miasma of disrespect, nitpicking, defensiveness, discouragement, and intolerance. This toxic atmosphere permeates the entire culture, gradually driving less toxic people into seclusion or to other languages, or souring their moods such that they become toxic as well. This leaves an even higher concentration of toxicity, affecting even more people, on and on.

Is it possible to reverse this trend? I don’t know. It may be too late. If it can be reversed, I suspect the way to do it is for the quiet, decent folk to put on emotional hazmat suits and start being more active in the community. Participate in discussions, help new Lispers, and discourage toxic behavior by privately and tactfully informing people about the effect they have. Reducing the toxicity of the community atmosphere would be a major culture shift, but with a sustained effort it might be possible to create a more healthy and positive atmosphere.

I can’t help but compare this to the Ruby community’s notion of MINSWAN (Matz Is Nice So We Are Nice), and wish there were more positive role models in the CL community to set a good example. Most of the people who are revered in the CL community are quite intelligent and knowledgeable about CL and computer science, but are also very toxic. The motto for this community would be something along the lines of NaWTSWAT: Naggum Was Toxic So We Are Toxic. The community discussion venues have become an echo chamber, perpetuating and reinforcing the notion that it is okay to be derogatory and inflammatory, as long as you are intelligent or know a lot about CL. The people who disagree with this attitude tend to give up in disgust and either leave or stop talking, so you won’t hear many dissenting opinions about that from the people who remain.

This may seem like a gloomy assessment, but the situation is not all bad. There are some awesome people making cool things with Common Lisp, and many people who will try to help as best they can when you have a problem. It’s just a shame that the predominant attitude is so negative.

It’s Not All Bad!

I have spent many more words so far describing the weaknesses and problems of Common Lisp, than describing its strengths and interesting features. But despite its quirks and baggage and toxic people, Common Lisp is an incredibly flexible and powerful tool for creating software. Ruby may offer a more polished baseline experience, but you can’t take it as far as you can take CL.

The ANSI Common Lisp standard may stand still, but Common Lisp does not. It is constantly growing and evolving on top of the stable (albeit lumpy and uneven) foundation provided by the standard. For example, ASDF (which is a bit like Rake or Make) was created in 2001, and revolutionized the way CL libraries are defined and loaded. ASDF laid the groundwork for Quicklisp (CL’s analog to RubyGems), created in 2010, to revolutionize the way CL libraries are downloaded and installed. That in turn lays the groundwork for further development and progress.

Flexibility (Macros and Read Macros)

One reason CL can keep evolving without needing to change the standard, is the flexibility provided by features like macros and read macros. In addition to the usual small utility macros, I’ve created macros for Ambienome to implement a limited form of prototypal inheritance, and extensible object properties. They are somewhat longer than the average macro, but fairly mundane; they merely expand into a few formulaic functions with the details filled in. There are much more complex and interesting things you can do with macros, like the famous (or infamous) loop macro, which implements a specialized mini-language within CL, dedicated to making it easy to write fairly sophisticated code loops. Even the Common Lisp Object System (CLOS), which provides CL’s class-based OOP system, is largely implemented via some very complex macros. CLOS and loop are both defined in the standard, but they probably could have been implemented as separate libraries. (I’m guessing that having those things standardized enables implementations to optimize their performance. Or maybe they were just considered fundamental to the language.)

Read macros take flexibility and extensibility even further. They let you write code in CL that reprograms the way CL parses the text of your source code, potentially altering the language in radical ways. One fairly common and not-so-radical use for read macros is adding syntax sugar. For example, the CL standard doesn’t have a literal syntax for reading or printing hash tables, but thanks to read macros you can add Ruby’s hash table syntax to CL in 40 lines of code (or less if you don’t need the hashrockets and commas). Similarly, you can add literal syntax for regular expressions, even though regular expressions are provided by libraries like CL-PPCRE instead of being defined in the standard. I don’t know of any other language where you can think of some syntax that would make the language more expressive or powerful, then add it with just a couple of afternoon hacking sessions.

Generic Functions

Next to CL’s flexibility from macros and read macros, the thing I find most interesting in CL is the generic functions system. Classes in CL don’t (usually) have methods in the same sense that classes in Ruby do. Instead, CL has generic functions, and each method implements the behavior as it relates to instances of a certain class (or combination of classes). It’s quite different from Ruby or any other object-oriented language I’ve used, but very flexible and powerful. I’d like to write more about the CLOS object model and how it compares to Ruby’s, but this post is already rather long.

Conditions and Restarts

Another interesting feature of CL is its condition system. Conditions are analogous to Ruby’s exceptions, but much more flexible and powerful. (That seems to be a recurring theme.) Besides raising errors and warnings, you can use conditions to send any kind of signal up the call chain, with the lower-level code possibly providing some “restarts”, ways to proceed. Depending on how you set up the restarts, your higher-level code might interrupt the lower-level code like in Ruby, or ignore the signal and resume the lower-level code where it left off, or modify something and then resume, or pretty much anything else you can imagine. I haven’t had much opportunity to use conditions in a substantial way yet, but I’m sure they will prove useful as my code matures.

Integrated Development Environments

CL also has some nice IDEs. I use Emacs with SLIME, which is a big step up from my Ruby workflow. It offers the usual IDE amenities like tab-completion, parameter hints, jumping to function definitions, and looking up documentation (which is stored in each function/macro/etc., not derived from scanning comments like RDoc and YARD do). It also lets you inspect and modify the guts of most objects, which is pretty handy for debugging. And apparently even SLIME pales in comparison to the fancy IDEs provided by some commercial CL implementations.

Optimization Hints

Finally, CL has a system for declaring optimization hints to tweak the way the implementation compiles or interprets your code. (These are hints, so the implementation decides what to actually do. Some implementations might just ignore your hints.) Besides optimizing for computational speed, you can optimize for space (compiled code size, and runtime memory use), safety (run-time error checks), compilation speed, and/or ease of debugging. You can also optionally declare the argument and/or return types for functions, which can help the implementation generate even more efficient code. And finally, you can declare that a function can be compiled inline into other code that calls it, which can reduce or eliminate the overhead from function dispatch (this works well for short utility functions). I have read that with the right optimization hints, some implementations can compile number-crunching functions into code that is nearly as performant as C code. I’m pretty relaxed about performance (I used Ruby for over 7 years, after all!), but it’s nice to have extra tools available for dealing with bottlenecks.

Final Thoughts

So, those are my impressions of Common Lisp after 6 months of focused study and use. There are a lot of good things about CL, but also some really bad. It’s a pretty amazing language, but has some warts and flaws. The community is mostly decent people, but anyone who tries to participate in discussion ends up choking in the toxic atmosphere. The warts and flaws can be covered up or worked around pretty easily, but creating a more healthy community atmosphere seems nearly impossible. But I hope it can happen somehow, because it’s a shame to see the language being held back like this.

For Ambienome’s physics engine, I’m using SquirL, a Common Lisp port of the popular Chipmunk 2D game physics engine. I’m not far enough along yet to know how well it performs in my own game, but from running the demos, it certainly seems performant enough for my purposes. If it turns out to be somewhat sluggish, many Common Lisp implementations can perform some pretty aggressive optimization, given the right type declarations. As a last resort, I could create an FFI binding to the actual Chipmunk library, although that might not be any faster than a well-optimized Common Lisp port. One benefit of a binding to Chipmunk would be using its new features that have not been ported to SquirL (which hasn’t been updated in 2 years).

As I explained in the previous post, a creature in Ambienome is composed of a hierarchy of “parts”, or shapes. This hierarchical nature means that a child’s transformation (position, rotation, scale) are defined relative to the parent, and the parent’s transformation affects all the children (and the children’s children, and so on). But, SquirL/Chipmunk only has the concept of a body composed of a flat list of shapes; they don’t understand hierarchies. So, when generating the physical representation of a creature, the code must “walk” the hierarchy, converting each part’s parent-relative transformation to a root-relative transformation.

Actually, though, not every creature will be made of a single physical body. In SquirL/Chipmunk, each body is a rigid body, meaning its component shapes are not supposed to move relative to each other. Physical joints (springs, pivots, etc. — “constraints” in SquirL/Chipmunk parlance) only connect bodies, not shapes within a body. So, any creature that has parts connected via joints will have to be split up into multiple physical bodies. This would happen automatically behind the scenes, of course; Ambienome users don’t need to care about the underlying physical representation.

One potential pitfall is that if a part’s transformation is changed, the physical representation of it and everything lower in the hierarchy must be updated or regenerated. If a part is changing often, that could be a significant performance hit, and also cause problems with the physics engine. So, it may be that some creatures will have to be split up into multiple physical bodies, even if they don’t use explicit joints. A part that will move or rotate often could be automatically split into a separate physical body, then connected to the main body via a joint (or combination of joints) to hold it in place.

Unfortunately, SquirL/Chipmunk cannot handle scale changes in such a graceful way. So, it would still be necessary to regenerate the physical shapes after a scale change, with all the performance and simulation stability issues that entails. Of course, if that turns out to be a big problem, the ability for behaviors to change scale could be limited in some way. I might even disable it at first, then only add that ability later if it can be done without causing problems.

Two other things that I’ll need to address with regards to the physics engine, are static creatures and phantom creatures. Static creatures are creatures that don’t move, but other creatures can collide and bounce off of them. SquirL and Chipmunk both support fairly simple (but different) ways of accomplishing that, so the technical side is not a problem. But, I’ll have to give some thought to how this option is presented to users. For example, is the user able to make a static creature with non-static limbs, like an anemone whose base doesn’t move, but whose tendrils do? This has some implications about how creatures are built and structured; it may be necessary or desirable to explicitly divide creatures into segments (corresponding to SquirL/Chipmunk bodies), which are then divided into parts (corresponding to SquirL/Chipmunk shapes).

Phantom creatures are creatures that do not have a solid form, do not collide or bounce off other creatures, but can still move around. This can be accomplished easily in SquirL/Chipmunk by putting all phantom creatures on a designated layer (so they will not collide with non-phantom creatures), and putting them in the same group (so they will not collide with other phantom creatures). But, just like static creatures, the concept of phantom creatures has implications about how a creature is structured. Can some parts of the creature be phantom, while others are not? Ideally, this should be allowed, but it does suggest that my current concepts of creatures and parts needs to be reformed.

The final topic for this post is object selection via mouse picking. This isn’t always related to physics engine, but in this case, I’m planning to leverage the physics engine to make picking easier. SquirL and Chipmunk both have ways to check what shapes intersect a point, and Chipmunk additionally has ways to check for intersection with a line segment, axis-aligned bounding box, or shape.

This does mean, though, that all possibly selectable creatures must be added to the physics engine. If I used a different technique for mouse picking, only creatures that actually participate in the physics simulation would need to be added. But I don’t expect this to be much of a problem, because the only creatures that could have stayed out of the simulation would be creatures that are both phantom and static, which I think will be relatively rare.

There are several other picking techniques available. Two popular techniques for OpenGL applications are the GL_SELECTION rendering mode, and unique color rendering. These are certainly feasible, but implementing them would take time and effort I could spend on other things, and would add complexity to my code base. So, I’ll save them as backup plans, in case using the physics engine for mouse picking doesn’t work out.

Data structure simply means the way some data is structured or organized, either in memory while the program is running, or when the data is saved to a file or sent over a network. Is it stored in a hash table? An array? An instance of a class? What slots/members does the class have? How are different pieces of data related to each other? And so on.

Serialization is a topic very closely related to data structure. Serialization involves converting from one data structure to a simpler data structure, usually some human-redable text or a raw binary sequence with some specific order or pattern. The reverse process, deserialization, involves converting the simpler data structure back into the original structure (or a similar one). Serialization is most often used when saving data to a file or sending it over a network.

Personally, I consider binary serialization to be a measure of last resort, to be used only when performance is absolutely critical. Binary formats are notoriously fragile and difficult to extend, especially if they are poorly designed.

(I recently worked on a project where the previous programmer wrote code to “serialize” a C struct by copying a chunk of raw bytes from memory and sending it over the network. To “deserialize” it, they would — I kid you not — copy the bytes into memory and type-cast it to the struct type! They had given absolutely no thought to portability, extensibility, validation, or robustness. And the person who wrote it was ostensibly a professional programmer, and had been paid for the work.)

So, I use human-readable text formats whenever possible. Probably the two most popular formats for this kind of serialization are XML and JSON. But Lisp already has a simple, human-readable, portable, built-in serialization format: S-expressions, the same format used for Lisp code. I’m still experimenting with the structure, but here’s an example of a simple scene as it is serialized now:

(:instance :scene
 :props nil
 :creatures
 ((:instance :creature
   :name "Creature 1"
   :body
   (:instance :part
    :props
    ((:shape :label "Shape" :value :rectangle)
     (:angle :label "Angle" :type :angle :value 0)
     (:scale :label "Scale" :type :real :value 120)
     (:size :label "Size" :type :vec :value #C(1.0d0 1.0d0))
     (:pos :label "Position" :type :vec :value #C(75.0d0 150.0d0))
     (:depth :label "Depth" :type :real :value 0)
     (:color :label "Color" :type :color :value #(0.7 0.1 0.6))
     (:name :label "Name" :type :string :value nil))
    :children nil))))

It’s human-readable, very compact for how much information it expresses, and Lisp intrinsically knows how to read and print it. I might write a little bit of code to get rid of the colons (which indicate Lisp keyword symbols) to make it even easier for humans to read.

I should point out that even though the structure is serialized as lists, the data is actually stored in the program as class instances, arrays, and hash tables. They are converted to lists before being serialized, then inserted back into an array or hash table when deserialized. In addition to making the output easier for humans to read, adding this layer of abstraction means I can change the internal data structure while easily maintaining compatibility with older versions.

Besides decisions about whether to use lists, arrays, hash tables, or whatnot, I’ve also been experimenting with how the various kinds of objects relate to each other. As it is now, there are three main kinds of objects:

• Scene objects, which represent a particular scenario of creatures, scene-wide behaviors, and properties such as background color and the scene’s creator.
• Creature objects, which represent a collection of parts with some behaviors and properties such as its name and creator. A creature directly holds a main part, called the body, of which all the other shapes are “descendants”.
• Parts, which represent a shape, like a circle or triangle, with properties such as its size, color, density. A part can have “children”, meaning they are linked together in a hierarchy.

One noteworthy thing about these data structures is the way object properties are stored. Instead of storing color, position, size, etc. directly as slots/members of a class instance, they are all stored together in the object’s “props” hash table. This makes it more flexible, as props can be added or removed at any time. At some point, scene creators will be able to add their own props that are used by custom behaviors. For that same purpose, props are stored with metadata about their type, label, and so on.

There’s another kind of object that I may be adding soon: joints, which are used with the physics engine to connect parts that move independently, but are joined together with a spring, pivot, or whatnot. But, I might just make joints another type of part. I’ll talk about this and other physics engine stuff in my next post.

First, I spent quite a lot of time working on proto-slots. That may seem productive, but the amount of effort I put into polishing and documenting it was way out of proportion to the benefit I would get from it. I think I did a pretty good job on that project, but I also recognize now that I was using it as a way to avoid actually working on Ambienome.

It was about three weeks ago that I realized what was happening, so I tried to force myself to focus on Ambienome. That didn’t really work, and I just ended up with a serious creative block. I spent about two weeks making almost no progress. My code had been accruing unnecessary complexity, mostly due to exploring a lot of unfamiliar territory (Lisp, and OpenGL), which made it difficult to work on when my motivation was not very high. Each time I would try to focus on Ambienome, I’d run into some obstacle, grimace, and reflexively go find some way to distract myself. (In hindsight, it was probably not wise to start a challenging new project at the beginning of winter, since my energy levels and motivation always dip during the winter.)

(Those two weeks weren’t a complete loss, though; I did beat a lot of games! :P If you’re curious: Dungeons of Dredmor, The Binding of Isaac, Limbo, Glowfish, Blocks That Matter, and a couple others that I can’t recall at the moment.)

I finally broke through the creative block about a week ago, by unplugging the internet, shutting out all distractions, and just tackling my code head-on. I spent 3 days just simplifying and cleaning up my code to make further development feasible. All told, I managed to eliminate 9 classes:

• job, which acted as a central coordinating structure for the OpenGL rendering pipeline. The important functionality was transformed into methods of the program class, which is slightly lower-level coordinating structure for the rendering pipeline. The rest of the code was scrapped.
• uniform and attribute, which represented OpenGL shader uniforms and attributes. These classes were also reduced into methods of program, and the unnecessary code scrapped.
• rectangle, triangle, and circle, which represented instances of those geometric shapes. They were reduced into a render-shape generic function, with a method implemented for each shape based on a keyword. For example, to render a circle you’d do (render-shape :circle ...). Lisp’s method dispatching system is pretty handy sometimes!
• shape, which was the parent class of the three shapes mentioned above. Aside from the rendering code, which (as I said) became methods, this class also stored the shape’s transformation (position, rotation, scale), and other properties such as color. I moved this functionality to the part class, which represents a creature body part.
• group, which held multiple shapes that would move together, just like SVG’s g element. Instead, I decided that it’s simpler and more intuitive to allow body parts to be parented to other parts. To handle the case of an invisible pivot point, the parent shape can be made invisible, in which case it’s functionally equivalent to a group.
• prop, which represented a property of a shape, such as its color. The idea behind this class was that it would contain not only the data, but also some metadata (its type, human-readable name, etc.) that could be used later to generate an appropriate user interface to edit the property. For example, the color property would have some metadata indicating that the data was a color, so it should be edited using a color swatch widget. I decided to just store the data and metadata in hash tables instead.

My code is now much leaner and more function-oriented than before, and much more approachable and easier to work with. I actually eliminated (temporarily, I think) the need for my proto-slots library — which, after all the work I put into it, is either really funny or really frustrating.

Yesterday and today, I’ve been making progress (not a great deal, but enough) on new code, specifically the ability to serialize and deserialize creatures and their body parts, so that they can be saved and loaded from file. I’m nearly done with this, but I’m not sure yet what I’ll tackle next.

Perhaps I’ll try to plot out a more concrete roadmap, start to define and narrow the scope of the game, and lay out some specific tasks and goals. I’m sure my unorganized, exploratory approach is a significant cause of my slow progress. It would be good to have some concrete milestones to chase after, and maybe even some deadlines to push myself forward. What a radical notion!

proto-slots provides a macro for defining prototypal accessor methods so that CLOS instances will support protoypal inheritance. Prototypal inheritance means that an instance’s slots can inherit values from another instance, known in these docs as the “base object” (but more commonly known elsewhere as the “prototype”).

I wrote the first version of what eventually because the def-proto-slots macro to solve a problem I described in my previous post:

But, for each shape instance to have its own color (or other such data in the future), each shape instance must have its own job, because the color would be a uniform contained in that job. If I set the color in the base job, all instances of that kind of shape would have the same color. But if each instance has its own job, I lose the benefit of having the vertex data stored in one place.

I solved this by letting a job have a “parent”, from which it inherits anything it’s missing, such as a program or elements. For vertex attribs and uniforms, the inheritance works by merging the parent’s list of attribs/uniforms with the child’s list, with any of the child’s attribs/uniforms taking precedence when it has the same name as one of the parent’s.

I realized later that I was going to need prototypal inheritance for other parts of Ambienome, too. For example, I’m pretty sure the creature class will use this pretty heavily. So, I took my ad hoc macro, made it more flexible (it now supports different “inheritance strategies”), and polished it up. Along the way I decided to release it as a separate library, to maybe save someone else the trouble of implementing this same sort of thing.

proto-slots is available at GitHub, released under the X11/MIT license. If you find a bug, make an issue. To submit a patch, fork my repo and send a pull request.

First, I created a new transform class by abstracting and cleaning up the shape class’s position, angle, and size attributes and methods. I also created a transformable “mixin” class, which can be used by any class that has a transform. (Lisp doesn’t have mixins in the same sense that Ruby does, but since Lisp supports multiple inheritance, you can get more or less the same effect by designing a class in a mixin-ish style.) The transformable class provides accessors for the transform’s position, angle, and size, so that you can treat them as if they were direct slots of the object holding the transform.

Next, I decided to enhance the shape class so that shapes can have a color. This turned out to be somewhat challenging, because of how I had designed the OpenGL wrapper classes.

I have a job class that holds a program object (which is a collection of compiled shaders linked together) and the vertex attribs, uniforms, and elements used to render a certain thing (such as a shape). For memory efficiency, each kind of shape (rectangle, triangle, or circle) has a base job which holds the generated vertex data and elements.

But, for each shape instance to have its own color (or other such data in the future), each shape instance must have its own job, because the color would be a uniform contained in that job. If I set the color in the base job, all instances of that kind of shape would have the same color. But if each instance has its own job, I lose the benefit of having the vertex data stored in one place.

I solved this by letting a job have a “parent”, from which it inherits anything it’s missing, such as a program or elements. For vertex attribs and uniforms, the inheritance works by merging the parent’s list of attribs/uniforms with the child’s list, with any of the child’s attribs/uniforms taking precedence when it has the same name as one of the parent’s. It’s wonderfully easy to do that in Lisp:

(union (attribs parent) (own-attribs job)
       :key #'name :test #'string=)

This code takes the attribs list from the parent, including any attribs the parent itself inherits from its parent, and the list of attribs belonging directly to the job, and performs a set union of those two lists, which discards duplicates. But what is a “duplicate”? By providing the :key #'name, I’m telling it to call the name function on each attrib, and then check the names to see if they are duplicates. The :test #'string= part tells it to use the string= function, which performs case-sensitive string comparison, to decide if the two names are the same. The result is that if an attrib in the parent’s list and an attrib in the child’s list have the same name, only the child’s is used (since it is given as a later argument to the union function).

While doing this, I noticed that the inheritance code was exactly the same for attribs and uniforms, except for the word “attrib” or “uniform”. So, I wrote a fairly simple, albeit long, macro to keep things DRY. Then I wrote shorter macro to reduce the redundancy in the simpler inheritance behavior for the other slots, which only inherit from the parent if their slot is empty/null, but don’t have to perform any merging.

Now that jobs can have a parent, I can use each shape class base job (the job containing the vertex data and elements) as the parent for each shape instance’s job (the job containing the color and other instance-specific data). So, the instance’s job can inherit the vertex data and elements, while still having its own color. And later, I’ll be able to have base jobs at different levels of detail, and easily change the instances’ parents whenever I want to change their level of detail.

After a bit more work, I finally had support for different colors in each shape instance. Victory!

It was a long detour, but my code base is much better for all the work. Now I can finally get around to shape groups and creature components!

The triangle looks like it’s too small, but it fits perfectly in the circle, and the circle fits perfectly in the square. Yay, geometry! Of course, when building a creature, they can be scaled to whatever size you like, and the algorithms can produce meshes at any level of detail. Eventually, I’ll make it so the game automatically adjusts the level of detail based on the shape’s size on the screen.

As I mentioned in the previous post, these aren’t the only shapes that will be available, but they are enough to let me move on to other things. One of the core concepts of Ambienome is that I can program new shapes later, even after the game has been released. If I’m feeling really clever, I could probably leverage Lisp’s power to allow users to program new kinds of shapes, too. That’s not a high priority, though.

Today I’ll start code-sketching the concept of components and creatures. A creature is built from one or more components, which are basically shapes with some associated behavior. Components can be grouped to move together, and groups can contain other groups, so it’s essentially a “tree” (hierarchy) of transformation nodes, with each “leaf” being a shape. This is the same concept as grouping in SVG or Flash, or parenting in 3D software like Blender or Maya.

After weeks of just studying and debugging OpenGL, it sure is nice to be making visible progress again!

Even before starting Ambienome, I was already somewhat familiar with the old OpenGL “fixed function pipeline”, using glBegin/glEnd, glColor, glVertex, etc. Ambienome is going to be visually simple enough that I probably could have used the fixed function pipeline, but I decided to learn the shader pipeline to improve my knowledge and skills, to allow nicer visual effects, and to leverage the GPU’s number-crunching power as much as I can.

Grokking the shader pipeline was a challenge. There are many separate concepts to learn, and then you must understand how they fit together. Also, there are fewer learning resources for the shader pipeline than there are for the older fixed function pipeline. I relied mostly on Joe Groff’s “An intro to modern OpenGL” article series, but if I were doing it over again, I might try to find a good book. Unfortunately, the selection is somewhat thin. I’m considering OpenGL Shading Language (the so-called “Orange Book”) or the OpenGL ES 2.0 Programming Guide. (OpenGL ES 2.0 is conceptually very similar to the OpenGL shader pipeline, and also very similar to WebGL.) Unfortunately, the reviews suggest those two are not very well written, organized, or proofread. Real-Time Rendering looks like it might provide a good high-level understanding of shaders and rendering, but it’s not specific to OpenGL or GLSL, and I’m not really interested in advanced photorealistic shader effects. Maybe if there are any book stores still open in this town, I’ll see if any of these books are on the shelf so I can assess them before I buy.

Anyway, even after I had scraped together a modest understanding of the concepts behind the shader pipeline, applying that knowledge in practice took longer than I anticipated. It was fairly frustrating at times, partly due to some (minor) shortcomings of cl-opengl and lispbuilder-sdl, and partly due to me misinterpreting some vague parts of the OpenGL documentation. There were many times when I felt like putting the shader pipeline aside, and just using the fixed function pipeline for a while, so I could keep my momentum up. But I stubbornly stuck with it, and now I’ve got a working, reusable rendering framework built atop the shader pipeline. Huzzah!

My next task is to write code to algorithmically generate mesh data for several geometric primitives. Yesterday, I finished the code for a rectangle mesh built from triangle strips (well, actually one long strip with degenerate triangles connecting each row). Here’s a screenshot of a 10×10 rectangle mesh rendered in wireframe mode:

It’s not much to look at in itself. I could have achieved that in an afternoon using the fixed function pipeline! But that humble mesh, rendered here using the simplest possible vertex and fragment shaders, represents weeks of learning. Now that I have the basics working, I can write more interesting shaders to morph the shape and create cool visual effects. For example, Ambienome takes place underwater, so I’ll probably write a shader to make things sway and ripple as if they were being affected by a water current.

Next, I’ll be writing the code for a triangle mesh, and then a circle mesh. I have plans for more shapes, like teardrop, leaf, ring (circle with a hole in the middle), and tentacle (a Bézier curve with thickness controlled by another Bézier curve). But, I’m going to save those for later. Once I’ve got rectangle, triangle, and circle, I’m going to take a step back from the low-level foundation for a while, and start fleshing out higher-level concepts like components, creatures, and scenes.