That’s no good, so I wanted to take some time to set my thoughts and ideas for Ambienome onto paper (so to speak). This will help me slide back into that mode of thinking, hone my ideas by putting them into words, leave a journal of how the game evolves, and maybe spark some interest in anyone reading this.

Caveat: you should not interpret any part of this as a promise or firm description of how the actual game will be. I’m just unloading my thoughts and ideas as they exist now. Everything is still subject to change, etc.

Creation vs Play

One thing I have not yet decided is how much focus will be on creating content, and how much focus will be on playing (or consuming) content. In other words, is Ambienome a game with a built-in creature creator, or a creative tool for designing interactive motion graphics and audio? Does the player create things to play with, or is the act of creation itself the gameplay? Where do I strike the balance?

I think it should lean more towards being a game that you can create custom content for. A typical user might spend two thirds of their time playing with user-created content, and one third creating new content. Naturally, individual users will have different interests and focuses, but I really want creating content to be accessible, highly interactive, and engaging. Spore,Minecraft, and Little Big Planet are all excellent examples of this.

As I’m envisioning it now, users would create scenes, which would have various creatures as well as level geometry (walls, etc.) and maybe some non-creature objects/items. The game would come with a variety of pre-built creatures, and the user could also build their own from scratch or by modifying an existing creature. Creatures could be saved and re-used between scenes, exported or imported as files, and easily shared online. Likewise, scenes could be exported, imported, and shared online.

Ambienome would likely come with a few dozen pre-made scenes that gradually introduce new creatures and gameplay mechanics, as puzzle games often do. Some of the scenes would have specific puzzles or goals to solve, while others would be “free play” mode like Electroplankton, where the only goal is to have fun playing with the creatures.

Gameplay Mechanics

There are many possible gameplay mechanics, ways in which you can interact with the scene, and creatures can interact with each other. Here are some that have been floating around in my head:

• Light/darkness (bioluminescence, light sensitivity)
• Heat/cold (energy transfer)
• Attraction/repulsion (magnetism)
• Falling/floating (gravity, buoyancy)
• Water/air currents (lines, splines, vortices)
• Bubbles (for air supply, or carrying objects)
• Sound pitch/harmony
• Playing a melody (correct notes in correct order)
• Physical contact (collision, bouncing, overlapping)
• Eating food or other creatures
• Life cycles (birth, growth, death)
• Controlling creature directly with keyboard, mouse, or joystick
• Creatures follow mouse pointer
• Clicking and dragging to move creatures and objects

These mechanics can be combined in interesting ways to create fun puzzles and free play scenes. For example, there might be a puzzle scene where you guide a creature to find some fruit and bring it back to the creature’s hungry offspring. But of course, there are obstacles in the way! First you have to bump into some singing sea cucumbers that each play a different note and glow a different color. When you play the correct melody, a hot magma stone appears. You carry the stone to a dormant hydrothermal vent, thereby heating up the vent. When the vent is hot, it produces a rising current which carries you up to reach some fruit hanging on a vine on the ceiling. You carry the fruit back to the hungry offspring, thus completing the scene. (This is all 100% scientifically accurate, of course. I saw it on the Discovery Channel. Honest!)

The game mechanics can also be used for free play scenes. One scene might have some jellies whose long tendrils ring like wind chimes when they move. When you move the mouse cursor, it creates a brief current, stirring the tendrils and making relaxing sounds. Perhaps there are schools of tiny fish hiding among some kelp, but they are attracted by the sound of the jellies’ tendrils, so they emerge and start swimming around, leaving colorful trails as they move. There’s no goal to accomplish or puzzle to solve, you can just relax and enjoy the sights and sounds for as long as you wish.

Naturally, all the creatures and game mechanics used in the scenes that came with the game would also be available for users to create their own scenes. And, players would be able to use those scenes and creatures as starting points for their own creations. Maybe you want to add some hydrothermal vents from that puzzle scene to create warm water currents that stir the jellies’ tendrils in the free play scene. Just press a button to switch to “create” mode, drag some vents into the scene, and voilà.

Ideally, creatures and scenes uploaded by other users would also be available for creating derivative works. But, that starts to get into some complex issues. How much control do users have over the license? Do all uploaded creations have to be under a certain license (perhaps some form of Creative Commons)? What about creations they don’t upload to a central service, just export as files? Can they choose “no derivatives”? Would the game try to enforce the license, for example by not allowing other users to modify scenes marked as “no derivatives”? These are some of the issues I’ll probably have to think about eventually. But, thankfully, not today.

I have approached CL from nearly 8 years of programming in Ruby. I mention this because my experience with Ruby has certainly colored my impressions of CL, and many of my observations are about how CL compares to Ruby.

Different Yet Similar

Overall, I’d rate Ruby and CL as being about equal, on my personal, subjective scale of programming language goodness. They differ in many ways, both good and bad, but it feels to me like the differences balance out. So, the two languages are different, but neither one is clearly superior.

On a conceptual level, the two languages are actually fairly similar. They are certainly more similar to each other than either one is to a language like C, for example. They both have an interactive prompt, automatic memory management and garbage collection, first-class functions, anonymous functions and closures, arrays and hash tables, mutable strings, arbitrarily large integers, namespaces (modules in Ruby, packages in CL), and class-based OOP systems that don’t force you into an OOP style. There are probably many other smaller similarities that I could think of if I took the time. (A dedicated Lisper would point out that many of these concepts originated in some Lisp dialect or another.)

Naturally, the details of many of these things are different. For example, CL’s packages work very differently than Ruby’s modules, despite fulfilling the same basic purpose. But in general, the concepts in each language are similar enough that knowing one language will make it significantly easier to learn the other.

Strengths (In Brief)

There are several aspects of Ruby that I feel are especially strong relative to Common Lisp. Stated briefly, they are:

• Community (active, supportive, sharing)
• Approachability (easy for someone new to get started)
• Convenience (many useful constructs and patterns are built in)
• Consistency of the object model (e.g. everything has a class)
• Wide variety of available libraries (both standard and installable)

Of course, Common Lisp has several areas where I feel it has the advantage:

• Flexibility (macros and read macros)
• Generic functions system
• Conditions and restarts
• IDEs, built-in documentation
• Optimization hints

I describe these in more detail near the end of this post.

Weaknesses

I’d like to be able to make a list of the areas each language is weak, but I’m too close to Ruby because of my years of use, yet also distant from it because I haven’t used it actively for perhaps a year. Both of those situations affect my perspective and make it hard to see Ruby’s real flaws.

Of course, Ruby has some weird bits of syntax (like {|args| ... } code blocks) and other quirks, but no more so than most languages, including CL. Ruby’s performance is traditionally a bit weak, but that varies across implementations, and is improving in leaps and bounds. The community is generally pretty great, although I find the “brogrammer” trend in some parts of the community to be distasteful and regressive.

My relationship to Common Lisp is a different situation. My eyes are still fresh and I’m actively using it, so I can see and describe Common Lisp’s shortcomings pretty clearly. But, I want to emphasize that just because I can still see the flaws in Common Lisp, that doesn’t mean there are no flaws in Ruby. I’m sure someone else could point out just as many problems in Ruby as I can in Common Lisp.

So, with that caveat, I’ll describe what I consider to be Common Lisp’s problem areas.

Minor Problem: Batteries Not Included

Ruby, Python, and other “batteries included” languages come with all kinds of nifty features (e.g. networking, regular expressions, XML/JSON/etc. support) right out of the box. A programmer coming to CL from one of those languages might expect CL to be the same way, but it’s not. Why?

The main reason is that CL has a formal ANSI standard, designed in the 1980s and early 1990s as a way to consolidate several different Lisp dialects. It took 10 years, countless man hours, and hundreds of thousands of dollars in direct expenses to finish the standard. Every detail was weighed and debated to make sure it was acceptable to the many people and groups with a vested interest in the process, such as the vendors who sold implementations of the old dialects.

So basically, only features that were acceptable to everybody were put in the standard. Things that they disagreed about, or that would put too much burden on the implementors, or that didn’t even exist back then, are not in the standard. Adding a new feature to the standard now would probably take many more years of debate and nitpicking, and would only succeed if practically everybody agreed that it was such a great feature that it is worth the effort. In other words, it would take a miracle and a half.

That is a very different scenario than you have with Ruby/Python/etc., where someone can submit a new feature to the most popular implementation, a core dev likes the features and commits it, and it’s released in the next version.

So, why is this not a huge deal in Common Lisp?

Well, most features can be written as libraries that you can install when you need them. CL is so flexible that even major changes to the language syntax can be installed as libraries! And with the advent of Quicklisp, installing libraries in CL is just as easy as installing gems in Ruby, or eggs in Python.

As for the features that can’t be written as libraries, individual CL implementations can provide them as extensions. For example, the implementation I use comes with a bunch of goodies that aren’t in the ANSI standard, such as threading, networking, code coverage and profiling tools, and a foreign function interface (FFI) for wrapping C libraries (such as OpenGL, as I’m using for Ambienome). If a feature is popular and many implementations provide it, someone will usually create a wrapper library that smooths out the differences between implementations, so that programmers can use the feature without being tied to one particular implementation.

(That’s an idealized scenario, of course. It doesn’t always work out so smoothly. Some implementations don’t add new features very often, if ever. Sometimes an implementation adds a feature in a way that isn’t compatible with other implementations, making it difficult or impossible to create a portable wrapper library. In such cases, if a programmer really needs the feature, they might target a specific implementation, rather than trying to keep their code portable. If they really need their code to be portable too, they can write code that acts differently for each implementation.)

That process of adding a feature to an implementation, letting other implementations copy it, and writing a wrapper library on top, is far from perfect. But, it’s faster, less onerous, and more flexible than trying to update the ANSI standard, and the results are usually satisfactory.

So, once you realize the implications of Common Lisp being formally standardized, and find out that there’s an easy way to install libraries, it’s not such a big deal that Common Lisp does not come with “batteries included”.

But, it’s still a minor problem that affects new Lispers coming from other languages. It might be possible to alleviate the problem through expectations management. If new Lispers understood from the start that many features are not part of the standard, but nevertheless are readily available as libraries, they wouldn’t be so surprised and disappointed.

I’m not sure who would be most effective at this sort of expectations management, though. Implementations? Book authors? Teachers? Bloggers? Wiki editors? IRC and newsgroup participants? I suspect all of the above are already doing it to some degree, yet many newcomers are still surprised. This is one area where having an “official website” for Common Lisp might make things easier.

Moderate Problem: Historical Baggage

Common Lisp has a lot of historical baggage: things that are done a certain way just because that’s how they were done in the past, even if they don’t make much sense nowadays. Perhaps that’s only natural for a programming language with such a long history. CL itself is nearly 20 years old (or 30, if you count from the first edition of Common Lisp the Language), and it inherited baggage from several older Lisp dialects, going all the way back to the original Lisp over 60 years ago.

That’s not to say that other languages don’t have some historical baggage of their own. For example, Ruby and many other languages inherit the bizarre old syntax of putting a 0 (zero) in front of a number to interpret it as octal form. (E.g. 10 means ten, yet 010 means eight. Surprise!) CL seems to have an especially large amount of baggage, though, and many Common Lispers seem to cling to that baggage rather tightly.

Historical baggage manifests itself in CL in many ways. One way is as individual quirks, like the setq operator. Setq was, I hear, originally invented as shorthand, so you could write (setq x 3) instead of (set (quote x) 3). But that was before the read macros were available; these days you can type (set 'x 3), which also means (set (quote x) 3) and is not any more characters than using setq. (Although, setq works a bit differently in CL than in earlier Lisps, so (setq x 3) isn’t quite the same as either (set (quote x) 3) or (set 'x 3) anymore.) Furthermore, CL has the setf operator, which can do everything setq can do and more, so there’s not really any reason to retain setq. It’s just historical baggage, kept around because that’s what they used in the old days. (Setq is still quite widely used today, usually with the rationale that it expresses the programmer’s intentions more clearly because it can do fewer things than setf.)

Historical baggage also manifests as old idioms that affect many parts of the language. Take for example the idiom of adding “p” to the names of predicate functions, functions that check something and return true or false (actually t or nil, another bit of historical baggage). For example, (evenp x) returns true if the value of x is an even number. It could have been (even? x), which I would argue is significantly clearer. But, Lispers used “p” in the old days, so (most, but not all) predicate functions in the CL standard use “p”, and therefore most Common Lispers still use “p” when they write their own predicate functions today.

Historical baggage also manifests itself as strange inconsistencies and curiosities in the underlying architecture. For example, Common Lisp has types, which it inherited from older Lisp dialects. It also has classes, which were added fairly late in the ANSI standardization process. Types and classes are very similar in many ways, but just different enough that they can’t be unified without breaking tons of old code. As a result, CL has both systems existing simultaneously, where each class has a corresponding type, but not all types have a corresponding class. Some features use types (like function and variable type declarations), while other features use classes (like generic function dispatching). I suppose the standards committee made the right choice given the circumstances back then, to avoid breaking all that old code. But, it is nevertheless historical baggage that all Common Lispers still carry nearly 20 years later.

Finally, historical baggage manifests itself as concepts and features that were useful many decades ago, but are pretty much obsolete today (and in some cases were already obsolete when the standard was written). For example, every symbol in Common Lisp has “properties”, a list of data that the symbol carries around in its guts. But nowadays, it would often be just as efficient to store that data in hash tables with the symbol as the key, and doing so would entirely avoid the possibility of two unrelated pieces of code coincidentally using the same property names. But, Lisp has had symbol properties since the very beginning, so Common Lisp has symbol properties too, along with the half-dozen functions used to manipulate them. (Symbol properties don’t seem to be used much anymore.)

These kinds of historical baggage aren’t a serious problem, because for the most part you can ignore the obsolete features, and either cover up or learn to live with the weird idioms and inconsistencies. But it is a moderate problem, because this historical baggage adds to the mental burden of every Common Lisp programmer in the world. (There are many, many other examples of historical baggage that I have omitted for the sake of brevity.)

It’s especially burdensome for new Lispers. Year after year, new Lispers have to go through a kind of rite of passage, learning the obscure idioms of years gone by, separating the useful features from the cruft, and trying to remember function names that seem arbitrary and inconsistent. Many of them give up and leave for other languages because of needless obstacles (this being just one of many they face). Some of them could have contributed a lot to the community over time, if only “the wall” hadn’t been built so high.

Serious Problem: The Community Atmosphere

That brings me to the most serious problem Common Lisp has: the community’s atmosphere, its prevailing moods and attitudes. This is such an important topic that it deserves its own post, but I’ll summarize the problem here.

I’ll admit that I started learning CL with the knowledge that many people (usually people who tried to join the community but were repelled) consider the community to be antagonistic, especially towards newcomers. So, I may be exhibiting some confirmation bias: seeing what I expected to see, and tending to ignore evidence to the contrary. But with issues like this, the widespread perception of a problem can be just as damaging as the reality of the problem itself.

I suspect that most people who use Common Lisp, as with any language, are probably decent folk who just want to write nifty code without a lot of fuss or drama. But these people are not very visible or active in community discussion venues (e.g. the comp.lang.lisp newsgroup or #lisp IRC channel). They’re off somewhere else, writing their nifty code in peace.

Alas, many of the people who are highly visible and active can best be characterized as “toxic”. These are people who, because of the nature of their personalities and attitudes, have a consistently negative emotional effect on the people they interact with. Without necessarily intending to do so, they have created a constant miasma of disrespect, nitpicking, defensiveness, discouragement, and intolerance. This toxic atmosphere permeates the entire culture, gradually driving less toxic people into seclusion or to other languages, or souring their moods such that they become toxic as well. This leaves an even higher concentration of toxicity, affecting even more people, on and on.

Is it possible to reverse this trend? I don’t know. It may be too late. If it can be reversed, I suspect the way to do it is for the quiet, decent folk to put on emotional hazmat suits and start being more active in the community. Participate in discussions, help new Lispers, and discourage toxic behavior by privately and tactfully informing people about the effect they have. Reducing the toxicity of the community atmosphere would be a major culture shift, but with a sustained effort it might be possible to create a more healthy and positive atmosphere.

I can’t help but compare this to the Ruby community’s notion of MINSWAN (Matz Is Nice So We Are Nice), and wish there were more positive role models in the CL community to set a good example. Most of the people who are revered in the CL community are quite intelligent and knowledgeable about CL and computer science, but are also very toxic. The motto for this community would be something along the lines of NaWTSWAT: Naggum Was Toxic So We Are Toxic. The community discussion venues have become an echo chamber, perpetuating and reinforcing the notion that it is okay to be derogatory and inflammatory, as long as you are intelligent or know a lot about CL. The people who disagree with this attitude tend to give up in disgust and either leave or stop talking, so you won’t hear many dissenting opinions about that from the people who remain.

This may seem like a gloomy assessment, but the situation is not all bad. There are some awesome people making cool things with Common Lisp, and many people who will try to help as best they can when you have a problem. It’s just a shame that the predominant attitude is so negative.

It’s Not All Bad!

I have spent many more words so far describing the weaknesses and problems of Common Lisp, than describing its strengths and interesting features. But despite its quirks and baggage and toxic people, Common Lisp is an incredibly flexible and powerful tool for creating software. Ruby may offer a more polished baseline experience, but you can’t take it as far as you can take CL.

The ANSI Common Lisp standard may stand still, but Common Lisp does not. It is constantly growing and evolving on top of the stable (albeit lumpy and uneven) foundation provided by the standard. For example, ASDF (which is a bit like Rake or Make) was created in 2001, and revolutionized the way CL libraries are defined and loaded. ASDF laid the groundwork for Quicklisp (CL’s analog to RubyGems), created in 2010, to revolutionize the way CL libraries are downloaded and installed. That in turn lays the groundwork for further development and progress.

Flexibility (Macros and Read Macros)

One reason CL can keep evolving without needing to change the standard, is the flexibility provided by features like macros and read macros. In addition to the usual small utility macros, I’ve created macros for Ambienome to implement a limited form of prototypal inheritance, and extensible object properties. They are somewhat longer than the average macro, but fairly mundane; they merely expand into a few formulaic functions with the details filled in. There are much more complex and interesting things you can do with macros, like the famous (or infamous) loop macro, which implements a specialized mini-language within CL, dedicated to making it easy to write fairly sophisticated code loops. Even the Common Lisp Object System (CLOS), which provides CL’s class-based OOP system, is largely implemented via some very complex macros. CLOS and loop are both defined in the standard, but they probably could have been implemented as separate libraries. (I’m guessing that having those things standardized enables implementations to optimize their performance. Or maybe they were just considered fundamental to the language.)

Read macros take flexibility and extensibility even further. They let you write code in CL that reprograms the way CL parses the text of your source code, potentially altering the language in radical ways. One fairly common and not-so-radical use for read macros is adding syntax sugar. For example, the CL standard doesn’t have a literal syntax for reading or printing hash tables, but thanks to read macros you can add Ruby’s hash table syntax to CL in 40 lines of code (or less if you don’t need the hashrockets and commas). Similarly, you can add literal syntax for regular expressions, even though regular expressions are provided by libraries like CL-PPCRE instead of being defined in the standard. I don’t know of any other language where you can think of some syntax that would make the language more expressive or powerful, then add it with just a couple of afternoon hacking sessions.

Generic Functions

Next to CL’s flexibility from macros and read macros, the thing I find most interesting in CL is the generic functions system. Classes in CL don’t (usually) have methods in the same sense that classes in Ruby do. Instead, CL has generic functions, and each method implements the behavior as it relates to instances of a certain class (or combination of classes). It’s quite different from Ruby or any other object-oriented language I’ve used, but very flexible and powerful. I’d like to write more about the CLOS object model and how it compares to Ruby’s, but this post is already rather long.

Conditions and Restarts

Another interesting feature of CL is its condition system. Conditions are analogous to Ruby’s exceptions, but much more flexible and powerful. (That seems to be a recurring theme.) Besides raising errors and warnings, you can use conditions to send any kind of signal up the call chain, with the lower-level code possibly providing some “restarts”, ways to proceed. Depending on how you set up the restarts, your higher-level code might interrupt the lower-level code like in Ruby, or ignore the signal and resume the lower-level code where it left off, or modify something and then resume, or pretty much anything else you can imagine. I haven’t had much opportunity to use conditions in a substantial way yet, but I’m sure they will prove useful as my code matures.

Integrated Development Environments

CL also has some nice IDEs. I use Emacs with SLIME, which is a big step up from my Ruby workflow. It offers the usual IDE amenities like tab-completion, parameter hints, jumping to function definitions, and looking up documentation (which is stored in each function/macro/etc., not derived from scanning comments like RDoc and YARD do). It also lets you inspect and modify the guts of most objects, which is pretty handy for debugging. And apparently even SLIME pales in comparison to the fancy IDEs provided by some commercial CL implementations.

Optimization Hints

Finally, CL has a system for declaring optimization hints to tweak the way the implementation compiles or interprets your code. (These are hints, so the implementation decides what to actually do. Some implementations might just ignore your hints.) Besides optimizing for computational speed, you can optimize for space (compiled code size, and runtime memory use), safety (run-time error checks), compilation speed, and/or ease of debugging. You can also optionally declare the argument and/or return types for functions, which can help the implementation generate even more efficient code. And finally, you can declare that a function can be compiled inline into other code that calls it, which can reduce or eliminate the overhead from function dispatch (this works well for short utility functions). I have read that with the right optimization hints, some implementations can compile number-crunching functions into code that is nearly as performant as C code. I’m pretty relaxed about performance (I used Ruby for over 7 years, after all!), but it’s nice to have extra tools available for dealing with bottlenecks.

Final Thoughts

So, those are my impressions of Common Lisp after 6 months of focused study and use. There are a lot of good things about CL, but also some really bad. It’s a pretty amazing language, but has some warts and flaws. The community is mostly decent people, but anyone who tries to participate in discussion ends up choking in the toxic atmosphere. The warts and flaws can be covered up or worked around pretty easily, but creating a more healthy community atmosphere seems nearly impossible. But I hope it can happen somehow, because it’s a shame to see the language being held back like this.

For Ambienome’s physics engine, I’m using SquirL, a Common Lisp port of the popular Chipmunk 2D game physics engine. I’m not far enough along yet to know how well it performs in my own game, but from running the demos, it certainly seems performant enough for my purposes. If it turns out to be somewhat sluggish, many Common Lisp implementations can perform some pretty aggressive optimization, given the right type declarations. As a last resort, I could create an FFI binding to the actual Chipmunk library, although that might not be any faster than a well-optimized Common Lisp port. One benefit of a binding to Chipmunk would be using its new features that have not been ported to SquirL (which hasn’t been updated in 2 years).

As I explained in the previous post, a creature in Ambienome is composed of a hierarchy of “parts”, or shapes. This hierarchical nature means that a child’s transformation (position, rotation, scale) are defined relative to the parent, and the parent’s transformation affects all the children (and the children’s children, and so on). But, SquirL/Chipmunk only has the concept of a body composed of a flat list of shapes; they don’t understand hierarchies. So, when generating the physical representation of a creature, the code must “walk” the hierarchy, converting each part’s parent-relative transformation to a root-relative transformation.

Actually, though, not every creature will be made of a single physical body. In SquirL/Chipmunk, each body is a rigid body, meaning its component shapes are not supposed to move relative to each other. Physical joints (springs, pivots, etc. — “constraints” in SquirL/Chipmunk parlance) only connect bodies, not shapes within a body. So, any creature that has parts connected via joints will have to be split up into multiple physical bodies. This would happen automatically behind the scenes, of course; Ambienome users don’t need to care about the underlying physical representation.

One potential pitfall is that if a part’s transformation is changed, the physical representation of it and everything lower in the hierarchy must be updated or regenerated. If a part is changing often, that could be a significant performance hit, and also cause problems with the physics engine. So, it may be that some creatures will have to be split up into multiple physical bodies, even if they don’t use explicit joints. A part that will move or rotate often could be automatically split into a separate physical body, then connected to the main body via a joint (or combination of joints) to hold it in place.

Unfortunately, SquirL/Chipmunk cannot handle scale changes in such a graceful way. So, it would still be necessary to regenerate the physical shapes after a scale change, with all the performance and simulation stability issues that entails. Of course, if that turns out to be a big problem, the ability for behaviors to change scale could be limited in some way. I might even disable it at first, then only add that ability later if it can be done without causing problems.

Two other things that I’ll need to address with regards to the physics engine, are static creatures and phantom creatures. Static creatures are creatures that don’t move, but other creatures can collide and bounce off of them. SquirL and Chipmunk both support fairly simple (but different) ways of accomplishing that, so the technical side is not a problem. But, I’ll have to give some thought to how this option is presented to users. For example, is the user able to make a static creature with non-static limbs, like an anemone whose base doesn’t move, but whose tendrils do? This has some implications about how creatures are built and structured; it may be necessary or desirable to explicitly divide creatures into segments (corresponding to SquirL/Chipmunk bodies), which are then divided into parts (corresponding to SquirL/Chipmunk shapes).

Phantom creatures are creatures that do not have a solid form, do not collide or bounce off other creatures, but can still move around. This can be accomplished easily in SquirL/Chipmunk by putting all phantom creatures on a designated layer (so they will not collide with non-phantom creatures), and putting them in the same group (so they will not collide with other phantom creatures). But, just like static creatures, the concept of phantom creatures has implications about how a creature is structured. Can some parts of the creature be phantom, while others are not? Ideally, this should be allowed, but it does suggest that my current concepts of creatures and parts needs to be reformed.

The final topic for this post is object selection via mouse picking. This isn’t always related to physics engine, but in this case, I’m planning to leverage the physics engine to make picking easier. SquirL and Chipmunk both have ways to check what shapes intersect a point, and Chipmunk additionally has ways to check for intersection with a line segment, axis-aligned bounding box, or shape.

This does mean, though, that all possibly selectable creatures must be added to the physics engine. If I used a different technique for mouse picking, only creatures that actually participate in the physics simulation would need to be added. But I don’t expect this to be much of a problem, because the only creatures that could have stayed out of the simulation would be creatures that are both phantom and static, which I think will be relatively rare.

There are several other picking techniques available. Two popular techniques for OpenGL applications are the GL_SELECTION rendering mode, and unique color rendering. These are certainly feasible, but implementing them would take time and effort I could spend on other things, and would add complexity to my code base. So, I’ll save them as backup plans, in case using the physics engine for mouse picking doesn’t work out.

Data structure simply means the way some data is structured or organized, either in memory while the program is running, or when the data is saved to a file or sent over a network. Is it stored in a hash table? An array? An instance of a class? What slots/members does the class have? How are different pieces of data related to each other? And so on.

Serialization is a topic very closely related to data structure. Serialization involves converting from one data structure to a simpler data structure, usually some human-redable text or a raw binary sequence with some specific order or pattern. The reverse process, deserialization, involves converting the simpler data structure back into the original structure (or a similar one). Serialization is most often used when saving data to a file or sending it over a network.

Personally, I consider binary serialization to be a measure of last resort, to be used only when performance is absolutely critical. Binary formats are notoriously fragile and difficult to extend, especially if they are poorly designed.

(I recently worked on a project where the previous programmer wrote code to “serialize” a C struct by copying a chunk of raw bytes from memory and sending it over the network. To “deserialize” it, they would — I kid you not — copy the bytes into memory and type-cast it to the struct type! They had given absolutely no thought to portability, extensibility, validation, or robustness. And the person who wrote it was ostensibly a professional programmer, and had been paid for the work.)

So, I use human-readable text formats whenever possible. Probably the two most popular formats for this kind of serialization are XML and JSON. But Lisp already has a simple, human-readable, portable, built-in serialization format: S-expressions, the same format used for Lisp code. I’m still experimenting with the structure, but here’s an example of a simple scene as it is serialized now:

(:instance :scene
 :props nil
 :creatures
 ((:instance :creature
   :name "Creature 1"
   :body
   (:instance :part
    :props
    ((:shape :label "Shape" :value :rectangle)
     (:angle :label "Angle" :type :angle :value 0)
     (:scale :label "Scale" :type :real :value 120)
     (:size :label "Size" :type :vec :value #C(1.0d0 1.0d0))
     (:pos :label "Position" :type :vec :value #C(75.0d0 150.0d0))
     (:depth :label "Depth" :type :real :value 0)
     (:color :label "Color" :type :color :value #(0.7 0.1 0.6))
     (:name :label "Name" :type :string :value nil))
    :children nil))))

It’s human-readable, very compact for how much information it expresses, and Lisp intrinsically knows how to read and print it. I might write a little bit of code to get rid of the colons (which indicate Lisp keyword symbols) to make it even easier for humans to read.

I should point out that even though the structure is serialized as lists, the data is actually stored in the program as class instances, arrays, and hash tables. They are converted to lists before being serialized, then inserted back into an array or hash table when deserialized. In addition to making the output easier for humans to read, adding this layer of abstraction means I can change the internal data structure while easily maintaining compatibility with older versions.

Besides decisions about whether to use lists, arrays, hash tables, or whatnot, I’ve also been experimenting with how the various kinds of objects relate to each other. As it is now, there are three main kinds of objects:

• Scene objects, which represent a particular scenario of creatures, scene-wide behaviors, and properties such as background color and the scene’s creator.
• Creature objects, which represent a collection of parts with some behaviors and properties such as its name and creator. A creature directly holds a main part, called the body, of which all the other shapes are “descendants”.
• Parts, which represent a shape, like a circle or triangle, with properties such as its size, color, density. A part can have “children”, meaning they are linked together in a hierarchy.

One noteworthy thing about these data structures is the way object properties are stored. Instead of storing color, position, size, etc. directly as slots/members of a class instance, they are all stored together in the object’s “props” hash table. This makes it more flexible, as props can be added or removed at any time. At some point, scene creators will be able to add their own props that are used by custom behaviors. For that same purpose, props are stored with metadata about their type, label, and so on.

There’s another kind of object that I may be adding soon: joints, which are used with the physics engine to connect parts that move independently, but are joined together with a spring, pivot, or whatnot. But, I might just make joints another type of part. I’ll talk about this and other physics engine stuff in my next post.

First, I spent quite a lot of time working on proto-slots. That may seem productive, but the amount of effort I put into polishing and documenting it was way out of proportion to the benefit I would get from it. I think I did a pretty good job on that project, but I also recognize now that I was using it as a way to avoid actually working on Ambienome.

It was about three weeks ago that I realized what was happening, so I tried to force myself to focus on Ambienome. That didn’t really work, and I just ended up with a serious creative block. I spent about two weeks making almost no progress. My code had been accruing unnecessary complexity, mostly due to exploring a lot of unfamiliar territory (Lisp, and OpenGL), which made it difficult to work on when my motivation was not very high. Each time I would try to focus on Ambienome, I’d run into some obstacle, grimace, and reflexively go find some way to distract myself. (In hindsight, it was probably not wise to start a challenging new project at the beginning of winter, since my energy levels and motivation always dip during the winter.)

(Those two weeks weren’t a complete loss, though; I did beat a lot of games! :P If you’re curious: Dungeons of Dredmor, The Binding of Isaac, Limbo, Glowfish, Blocks That Matter, and a couple others that I can’t recall at the moment.)

I finally broke through the creative block about a week ago, by unplugging the internet, shutting out all distractions, and just tackling my code head-on. I spent 3 days just simplifying and cleaning up my code to make further development feasible. All told, I managed to eliminate 9 classes:

• job, which acted as a central coordinating structure for the OpenGL rendering pipeline. The important functionality was transformed into methods of the program class, which is slightly lower-level coordinating structure for the rendering pipeline. The rest of the code was scrapped.
• uniform and attribute, which represented OpenGL shader uniforms and attributes. These classes were also reduced into methods of program, and the unnecessary code scrapped.
• rectangle, triangle, and circle, which represented instances of those geometric shapes. They were reduced into a render-shape generic function, with a method implemented for each shape based on a keyword. For example, to render a circle you’d do (render-shape :circle ...). Lisp’s method dispatching system is pretty handy sometimes!
• shape, which was the parent class of the three shapes mentioned above. Aside from the rendering code, which (as I said) became methods, this class also stored the shape’s transformation (position, rotation, scale), and other properties such as color. I moved this functionality to the part class, which represents a creature body part.
• group, which held multiple shapes that would move together, just like SVG’s g element. Instead, I decided that it’s simpler and more intuitive to allow body parts to be parented to other parts. To handle the case of an invisible pivot point, the parent shape can be made invisible, in which case it’s functionally equivalent to a group.
• prop, which represented a property of a shape, such as its color. The idea behind this class was that it would contain not only the data, but also some metadata (its type, human-readable name, etc.) that could be used later to generate an appropriate user interface to edit the property. For example, the color property would have some metadata indicating that the data was a color, so it should be edited using a color swatch widget. I decided to just store the data and metadata in hash tables instead.

My code is now much leaner and more function-oriented than before, and much more approachable and easier to work with. I actually eliminated (temporarily, I think) the need for my proto-slots library — which, after all the work I put into it, is either really funny or really frustrating.

Yesterday and today, I’ve been making progress (not a great deal, but enough) on new code, specifically the ability to serialize and deserialize creatures and their body parts, so that they can be saved and loaded from file. I’m nearly done with this, but I’m not sure yet what I’ll tackle next.

Perhaps I’ll try to plot out a more concrete roadmap, start to define and narrow the scope of the game, and lay out some specific tasks and goals. I’m sure my unorganized, exploratory approach is a significant cause of my slow progress. It would be good to have some concrete milestones to chase after, and maybe even some deadlines to push myself forward. What a radical notion!

proto-slots provides a macro for defining prototypal accessor methods so that CLOS instances will support protoypal inheritance. Prototypal inheritance means that an instance’s slots can inherit values from another instance, known in these docs as the “base object” (but more commonly known elsewhere as the “prototype”).

I wrote the first version of what eventually because the def-proto-slots macro to solve a problem I described in my previous post:

But, for each shape instance to have its own color (or other such data in the future), each shape instance must have its own job, because the color would be a uniform contained in that job. If I set the color in the base job, all instances of that kind of shape would have the same color. But if each instance has its own job, I lose the benefit of having the vertex data stored in one place.

I solved this by letting a job have a “parent”, from which it inherits anything it’s missing, such as a program or elements. For vertex attribs and uniforms, the inheritance works by merging the parent’s list of attribs/uniforms with the child’s list, with any of the child’s attribs/uniforms taking precedence when it has the same name as one of the parent’s.

I realized later that I was going to need prototypal inheritance for other parts of Ambienome, too. For example, I’m pretty sure the creature class will use this pretty heavily. So, I took my ad hoc macro, made it more flexible (it now supports different “inheritance strategies”), and polished it up. Along the way I decided to release it as a separate library, to maybe save someone else the trouble of implementing this same sort of thing.

proto-slots is available at GitHub, released under the X11/MIT license. If you find a bug, make an issue. To submit a patch, fork my repo and send a pull request.

First, I created a new transform class by abstracting and cleaning up the shape class’s position, angle, and size attributes and methods. I also created a transformable “mixin” class, which can be used by any class that has a transform. (Lisp doesn’t have mixins in the same sense that Ruby does, but since Lisp supports multiple inheritance, you can get more or less the same effect by designing a class in a mixin-ish style.) The transformable class provides accessors for the transform’s position, angle, and size, so that you can treat them as if they were direct slots of the object holding the transform.

Next, I decided to enhance the shape class so that shapes can have a color. This turned out to be somewhat challenging, because of how I had designed the OpenGL wrapper classes.

I have a job class that holds a program object (which is a collection of compiled shaders linked together) and the vertex attribs, uniforms, and elements used to render a certain thing (such as a shape). For memory efficiency, each kind of shape (rectangle, triangle, or circle) has a base job which holds the generated vertex data and elements.

But, for each shape instance to have its own color (or other such data in the future), each shape instance must have its own job, because the color would be a uniform contained in that job. If I set the color in the base job, all instances of that kind of shape would have the same color. But if each instance has its own job, I lose the benefit of having the vertex data stored in one place.

I solved this by letting a job have a “parent”, from which it inherits anything it’s missing, such as a program or elements. For vertex attribs and uniforms, the inheritance works by merging the parent’s list of attribs/uniforms with the child’s list, with any of the child’s attribs/uniforms taking precedence when it has the same name as one of the parent’s. It’s wonderfully easy to do that in Lisp:

(union (attribs parent) (own-attribs job)
       :key #'name :test #'string=)

This code takes the attribs list from the parent, including any attribs the parent itself inherits from its parent, and the list of attribs belonging directly to the job, and performs a set union of those two lists, which discards duplicates. But what is a “duplicate”? By providing the :key #'name, I’m telling it to call the name function on each attrib, and then check the names to see if they are duplicates. The :test #'string= part tells it to use the string= function, which performs case-sensitive string comparison, to decide if the two names are the same. The result is that if an attrib in the parent’s list and an attrib in the child’s list have the same name, only the child’s is used (since it is given as a later argument to the union function).

While doing this, I noticed that the inheritance code was exactly the same for attribs and uniforms, except for the word “attrib” or “uniform”. So, I wrote a fairly simple, albeit long, macro to keep things DRY. Then I wrote shorter macro to reduce the redundancy in the simpler inheritance behavior for the other slots, which only inherit from the parent if their slot is empty/null, but don’t have to perform any merging.

Now that jobs can have a parent, I can use each shape class base job (the job containing the vertex data and elements) as the parent for each shape instance’s job (the job containing the color and other instance-specific data). So, the instance’s job can inherit the vertex data and elements, while still having its own color. And later, I’ll be able to have base jobs at different levels of detail, and easily change the instances’ parents whenever I want to change their level of detail.

After a bit more work, I finally had support for different colors in each shape instance. Victory!

It was a long detour, but my code base is much better for all the work. Now I can finally get around to shape groups and creature components!

The triangle looks like it’s too small, but it fits perfectly in the circle, and the circle fits perfectly in the square. Yay, geometry! Of course, when building a creature, they can be scaled to whatever size you like, and the algorithms can produce meshes at any level of detail. Eventually, I’ll make it so the game automatically adjusts the level of detail based on the shape’s size on the screen.

As I mentioned in the previous post, these aren’t the only shapes that will be available, but they are enough to let me move on to other things. One of the core concepts of Ambienome is that I can program new shapes later, even after the game has been released. If I’m feeling really clever, I could probably leverage Lisp’s power to allow users to program new kinds of shapes, too. That’s not a high priority, though.

Today I’ll start code-sketching the concept of components and creatures. A creature is built from one or more components, which are basically shapes with some associated behavior. Components can be grouped to move together, and groups can contain other groups, so it’s essentially a “tree” (hierarchy) of transformation nodes, with each “leaf” being a shape. This is the same concept as grouping in SVG or Flash, or parenting in 3D software like Blender or Maya.

After weeks of just studying and debugging OpenGL, it sure is nice to be making visible progress again!

Even before starting Ambienome, I was already somewhat familiar with the old OpenGL “fixed function pipeline”, using glBegin/glEnd, glColor, glVertex, etc. Ambienome is going to be visually simple enough that I probably could have used the fixed function pipeline, but I decided to learn the shader pipeline to improve my knowledge and skills, to allow nicer visual effects, and to leverage the GPU’s number-crunching power as much as I can.

Grokking the shader pipeline was a challenge. There are many separate concepts to learn, and then you must understand how they fit together. Also, there are fewer learning resources for the shader pipeline than there are for the older fixed function pipeline. I relied mostly on Joe Groff’s “An intro to modern OpenGL” article series, but if I were doing it over again, I might try to find a good book. Unfortunately, the selection is somewhat thin. I’m considering OpenGL Shading Language (the so-called “Orange Book”) or the OpenGL ES 2.0 Programming Guide. (OpenGL ES 2.0 is conceptually very similar to the OpenGL shader pipeline, and also very similar to WebGL.) Unfortunately, the reviews suggest those two are not very well written, organized, or proofread. Real-Time Rendering looks like it might provide a good high-level understanding of shaders and rendering, but it’s not specific to OpenGL or GLSL, and I’m not really interested in advanced photorealistic shader effects. Maybe if there are any book stores still open in this town, I’ll see if any of these books are on the shelf so I can assess them before I buy.

Anyway, even after I had scraped together a modest understanding of the concepts behind the shader pipeline, applying that knowledge in practice took longer than I anticipated. It was fairly frustrating at times, partly due to some (minor) shortcomings of cl-opengl and lispbuilder-sdl, and partly due to me misinterpreting some vague parts of the OpenGL documentation. There were many times when I felt like putting the shader pipeline aside, and just using the fixed function pipeline for a while, so I could keep my momentum up. But I stubbornly stuck with it, and now I’ve got a working, reusable rendering framework built atop the shader pipeline. Huzzah!

My next task is to write code to algorithmically generate mesh data for several geometric primitives. Yesterday, I finished the code for a rectangle mesh built from triangle strips (well, actually one long strip with degenerate triangles connecting each row). Here’s a screenshot of a 10×10 rectangle mesh rendered in wireframe mode:

It’s not much to look at in itself. I could have achieved that in an afternoon using the fixed function pipeline! But that humble mesh, rendered here using the simplest possible vertex and fragment shaders, represents weeks of learning. Now that I have the basics working, I can write more interesting shaders to morph the shape and create cool visual effects. For example, Ambienome takes place underwater, so I’ll probably write a shader to make things sway and ripple as if they were being affected by a water current.

Next, I’ll be writing the code for a triangle mesh, and then a circle mesh. I have plans for more shapes, like teardrop, leaf, ring (circle with a hole in the middle), and tentacle (a Bézier curve with thickness controlled by another Bézier curve). But, I’m going to save those for later. Once I’ve got rectangle, triangle, and circle, I’m going to take a step back from the low-level foundation for a while, and start fleshing out higher-level concepts like components, creatures, and scenes.

Even though I have seven years of experience with Ruby, and my own game library, Rubygame, already available and ready to use, I’ve decided not to write Ambienome in Ruby. Instead, I’m writing it in Common Lisp, using OpenGL and OpenAL for graphics and audio.

Libraries

OpenGL and OpenAL were natural choices, since they’re free, cross-platform, and widely supported with pretty much any programming language. I would have chosen these no matter what language I settled on.

OpenGL doesn’t provide window management or input event processing, though, so I’ll need another library to handle that. The main choices for that are SDL, GLFW, and GLUT.

GLUT is a very popular and widely supported framework, but I don’t like how you have to shape your entire program structure around GLUT’s expectations, so I quickly dismissed it. SDL is also popular, but my years of experience with it have made me keenly aware of its shortcomings. So, I decided to try GLFW first.

GLFW is pretty nice, like a less obtrusive version of GLUT. I played around with it long enough to create an abstraction layer for opening and closing the window, but then discovered that GLFW currently has no API for setting the window’s icon. That may seem like a minor detail, but it was enough to nudge me into reconsidering SDL.

Two other points in SDL’s favor are that I’m already familiar with it — unlike everything else about this project! — and that SDL 1.3 will have support for multi-touch devices, and possibly multiple mice, both of which could provide some cool ways of interacting with the game. SDL 1.3 hasn’t been released yet (after many years…), but I might try taking a snapshot of the development repository, and working with that.

So, I ported the window abstraction layer from GLFW to SDL (1.2), and added an input event abstraction layer. I’m not entirely happy with it, so I might reevaluate GLFW (or even GLUT, since I’ve read that freeglut and Apple’s GLUT let you avoid the glutMainLoop function). But for now, SDL will be enough while I figure out the challenging new stuff with OpenGL and OpenAL.

One final library worth mentioning is Clutter, which I’m considering using to create the in-game GUI. When it gets to that point, I’ll try Clutter, then look into alternative libraries if Clutter doesn’t work out. As a last resort, I could just roll my own GUI widgets, but I’d rather avoid that if I can.

Lisp

OpenGL and OpenAL are pretty popular, “safe” choices of technology for game development. There are countless web articles, tutorials, books, and other learning resources available, and many experienced programmers I could turn to for help if needed. Both libraries are available on Linux, Mac, and Windows, and have bindings for many programming languages. There’s nothing particularly surprising or unusual about using OpenGL or OpenAL.

The choice to use Common Lisp, on the other hand, is likely to raise a few eyebrows. After all, it’s secret alien technology powered by excessive parentheses!

Lisp, despite its elegance and power, is not nearly as commonly used for game programming as the C language family (C++, C#, Java, etc.), Python, or probably even Ruby. If I had to speculate as to why Lisp has a low adoption rate among game programmers, I’d cite the usual factors that affect Lisp’s adoption rate in general: difficulty getting started (choosing and installing a Lisp implementation and IDE, finding good tutorials, etc.), the “strange” syntax, Lisp’s reputation as being more academic than practical, and the Lisp community’s reputation (whether deserved or not) for being somewhat elitist and unfriendly. Also, many game programmers are obsessed with raw performance, as if brute-force number crunching was the key ingredient in making a fun game. Those programmers will usually stick with C or C++, and dismiss anything else as being “too slow”.

But, I don’t particularly care about Lisp’s popularity or why other people like or dislike it. I care about having an interesting, powerful language that fits the project well.

• Like Ruby, Python, etc., Lisp promotes rapid prototyping with concise and flexible code, and sustains creative flow in a way “stop-and-compile” languages like C cannot.
• Common Lisp can be compiled to native executables, and I can declare optimizations for those situations where number crunching efficiency is needed, such as generating custom audio data.
• Lisp’s notion of “code as data” should work well with the concept of expressing behaviors as objects, and saving each creature’s behavior in a file.
• Lisp’s macro and package systems should make it really easy to create a safe, simple, and efficient domain-specific language to allow users to define new creature parts and behaviors.
• Lisp is a novel language to me, so this project is sure to be an interesting learning experience!

Before settling on Common Lisp, I also considered CoffeeScript and Clojure. CoffeeScript (using HTML5 Canvas and Audio elements) was very tempting because of the ease of deployment and cross-platform compatibility. But, HTML5 does not yet have a workable API for generating and manipulating audio data, and running in a browser would also limit my options for player-to-player file sharing and multiplayer gameplay.

As for Clojure: it’s a really interesting Lisp, and the ability to use Java libraries and compile to Java bytecode could be useful. But, learning to write a game using only immutable data structures and STM/agents would be a bit too much of a novel learning experience, and this project is going to be a lot to wrap my head around already. Also, Clojure’s strengths, such as safe multithreading, don’t provide much of a practical benefit to this particular project.

So far, I’m really enjoying Common Lisp. I’ve been reading Practical Common Lisp, which is a really excellent resource for learning Common Lisp, and available online at no cost. I’m certainly not a Lisp guru yet, but I can already feel the amazing expressive power and freedom Lisp provides. This is sure to be an interesting project!