First, I spent quite a lot of time working on proto-slots. That may seem productive, but the amount of effort I put into polishing and documenting it was way out of proportion to the benefit I would get from it. I think I did a pretty good job on that project, but I also recognize now that I was using it as a way to avoid actually working on Ambienome.

It was about three weeks ago that I realized what was happening, so I tried to force myself to focus on Ambienome. That didn’t really work, and I just ended up with a serious creative block. I spent about two weeks making almost no progress. My code had been accruing unnecessary complexity, mostly due to exploring a lot of unfamiliar territory (Lisp, and OpenGL), which made it difficult to work on when my motivation was not very high. Each time I would try to focus on Ambienome, I’d run into some obstacle, grimace, and reflexively go find some way to distract myself. (In hindsight, it was probably not wise to start a challenging new project at the beginning of winter, since my energy levels and motivation always dip during the winter.)

(Those two weeks weren’t a complete loss, though; I did beat a lot of games! :P If you’re curious: Dungeons of Dredmor, The Binding of Isaac, Limbo, Glowfish, Blocks That Matter, and a couple others that I can’t recall at the moment.)

I finally broke through the creative block about a week ago, by unplugging the internet, shutting out all distractions, and just tackling my code head-on. I spent 3 days just simplifying and cleaning up my code to make further development feasible. All told, I managed to eliminate 9 classes:

• job, which acted as a central coordinating structure for the OpenGL rendering pipeline. The important functionality was transformed into methods of the program class, which is slightly lower-level coordinating structure for the rendering pipeline. The rest of the code was scrapped.
• uniform and attribute, which represented OpenGL shader uniforms and attributes. These classes were also reduced into methods of program, and the unnecessary code scrapped.
• rectangle, triangle, and circle, which represented instances of those geometric shapes. They were reduced into a render-shape generic function, with a method implemented for each shape based on a keyword. For example, to render a circle you’d do (render-shape :circle ...). Lisp’s method dispatching system is pretty handy sometimes!
• shape, which was the parent class of the three shapes mentioned above. Aside from the rendering code, which (as I said) became methods, this class also stored the shape’s transformation (position, rotation, scale), and other properties such as color. I moved this functionality to the part class, which represents a creature body part.
• group, which held multiple shapes that would move together, just like SVG’s g element. Instead, I decided that it’s simpler and more intuitive to allow body parts to be parented to other parts. To handle the case of an invisible pivot point, the parent shape can be made invisible, in which case it’s functionally equivalent to a group.
• prop, which represented a property of a shape, such as its color. The idea behind this class was that it would contain not only the data, but also some metadata (its type, human-readable name, etc.) that could be used later to generate an appropriate user interface to edit the property. For example, the color property would have some metadata indicating that the data was a color, so it should be edited using a color swatch widget. I decided to just store the data and metadata in hash tables instead.

My code is now much leaner and more function-oriented than before, and much more approachable and easier to work with. I actually eliminated (temporarily, I think) the need for my proto-slots library — which, after all the work I put into it, is either really funny or really frustrating.

Yesterday and today, I’ve been making progress (not a great deal, but enough) on new code, specifically the ability to serialize and deserialize creatures and their body parts, so that they can be saved and loaded from file. I’m nearly done with this, but I’m not sure yet what I’ll tackle next.

Perhaps I’ll try to plot out a more concrete roadmap, start to define and narrow the scope of the game, and lay out some specific tasks and goals. I’m sure my unorganized, exploratory approach is a significant cause of my slow progress. It would be good to have some concrete milestones to chase after, and maybe even some deadlines to push myself forward. What a radical notion!

proto-slots provides a macro for defining prototypal accessor methods so that CLOS instances will support protoypal inheritance. Prototypal inheritance means that an instance’s slots can inherit values from another instance, known in these docs as the “base object” (but more commonly known elsewhere as the “prototype”).

I wrote the first version of what eventually because the def-proto-slots macro to solve a problem I described in my previous post:

But, for each shape instance to have its own color (or other such data in the future), each shape instance must have its own job, because the color would be a uniform contained in that job. If I set the color in the base job, all instances of that kind of shape would have the same color. But if each instance has its own job, I lose the benefit of having the vertex data stored in one place.

I solved this by letting a job have a “parent”, from which it inherits anything it’s missing, such as a program or elements. For vertex attribs and uniforms, the inheritance works by merging the parent’s list of attribs/uniforms with the child’s list, with any of the child’s attribs/uniforms taking precedence when it has the same name as one of the parent’s.

I realized later that I was going to need prototypal inheritance for other parts of Ambienome, too. For example, I’m pretty sure the creature class will use this pretty heavily. So, I took my ad hoc macro, made it more flexible (it now supports different “inheritance strategies”), and polished it up. Along the way I decided to release it as a separate library, to maybe save someone else the trouble of implementing this same sort of thing.

proto-slots is available at GitHub, released under the X11/MIT license. If you find a bug, make an issue. To submit a patch, fork my repo and send a pull request.

First, I created a new transform class by abstracting and cleaning up the shape class’s position, angle, and size attributes and methods. I also created a transformable “mixin” class, which can be used by any class that has a transform. (Lisp doesn’t have mixins in the same sense that Ruby does, but since Lisp supports multiple inheritance, you can get more or less the same effect by designing a class in a mixin-ish style.) The transformable class provides accessors for the transform’s position, angle, and size, so that you can treat them as if they were direct slots of the object holding the transform.

Next, I decided to enhance the shape class so that shapes can have a color. This turned out to be somewhat challenging, because of how I had designed the OpenGL wrapper classes.

I have a job class that holds a program object (which is a collection of compiled shaders linked together) and the vertex attribs, uniforms, and elements used to render a certain thing (such as a shape). For memory efficiency, each kind of shape (rectangle, triangle, or circle) has a base job which holds the generated vertex data and elements.

But, for each shape instance to have its own color (or other such data in the future), each shape instance must have its own job, because the color would be a uniform contained in that job. If I set the color in the base job, all instances of that kind of shape would have the same color. But if each instance has its own job, I lose the benefit of having the vertex data stored in one place.

I solved this by letting a job have a “parent”, from which it inherits anything it’s missing, such as a program or elements. For vertex attribs and uniforms, the inheritance works by merging the parent’s list of attribs/uniforms with the child’s list, with any of the child’s attribs/uniforms taking precedence when it has the same name as one of the parent’s. It’s wonderfully easy to do that in Lisp:

(union (attribs parent) (own-attribs job)
       :key #'name :test #'string=)

This code takes the attribs list from the parent, including any attribs the parent itself inherits from its parent, and the list of attribs belonging directly to the job, and performs a set union of those two lists, which discards duplicates. But what is a “duplicate”? By providing the :key #'name, I’m telling it to call the name function on each attrib, and then check the names to see if they are duplicates. The :test #'string= part tells it to use the string= function, which performs case-sensitive string comparison, to decide if the two names are the same. The result is that if an attrib in the parent’s list and an attrib in the child’s list have the same name, only the child’s is used (since it is given as a later argument to the union function).

While doing this, I noticed that the inheritance code was exactly the same for attribs and uniforms, except for the word “attrib” or “uniform”. So, I wrote a fairly simple, albeit long, macro to keep things DRY. Then I wrote shorter macro to reduce the redundancy in the simpler inheritance behavior for the other slots, which only inherit from the parent if their slot is empty/null, but don’t have to perform any merging.

Now that jobs can have a parent, I can use each shape class base job (the job containing the vertex data and elements) as the parent for each shape instance’s job (the job containing the color and other instance-specific data). So, the instance’s job can inherit the vertex data and elements, while still having its own color. And later, I’ll be able to have base jobs at different levels of detail, and easily change the instances’ parents whenever I want to change their level of detail.

After a bit more work, I finally had support for different colors in each shape instance. Victory!

It was a long detour, but my code base is much better for all the work. Now I can finally get around to shape groups and creature components!

The triangle looks like it’s too small, but it fits perfectly in the circle, and the circle fits perfectly in the square. Yay, geometry! Of course, when building a creature, they can be scaled to whatever size you like, and the algorithms can produce meshes at any level of detail. Eventually, I’ll make it so the game automatically adjusts the level of detail based on the shape’s size on the screen.

As I mentioned in the previous post, these aren’t the only shapes that will be available, but they are enough to let me move on to other things. One of the core concepts of Ambienome is that I can program new shapes later, even after the game has been released. If I’m feeling really clever, I could probably leverage Lisp’s power to allow users to program new kinds of shapes, too. That’s not a high priority, though.

Today I’ll start code-sketching the concept of components and creatures. A creature is built from one or more components, which are basically shapes with some associated behavior. Components can be grouped to move together, and groups can contain other groups, so it’s essentially a “tree” (hierarchy) of transformation nodes, with each “leaf” being a shape. This is the same concept as grouping in SVG or Flash, or parenting in 3D software like Blender or Maya.

After weeks of just studying and debugging OpenGL, it sure is nice to be making visible progress again!

Even before starting Ambienome, I was already somewhat familiar with the old OpenGL “fixed function pipeline”, using glBegin/glEnd, glColor, glVertex, etc. Ambienome is going to be visually simple enough that I probably could have used the fixed function pipeline, but I decided to learn the shader pipeline to improve my knowledge and skills, to allow nicer visual effects, and to leverage the GPU’s number-crunching power as much as I can.

Grokking the shader pipeline was a challenge. There are many separate concepts to learn, and then you must understand how they fit together. Also, there are fewer learning resources for the shader pipeline than there are for the older fixed function pipeline. I relied mostly on Joe Groff’s “An intro to modern OpenGL” article series, but if I were doing it over again, I might try to find a good book. Unfortunately, the selection is somewhat thin. I’m considering OpenGL Shading Language (the so-called “Orange Book”) or the OpenGL ES 2.0 Programming Guide. (OpenGL ES 2.0 is conceptually very similar to the OpenGL shader pipeline, and also very similar to WebGL.) Unfortunately, the reviews suggest those two are not very well written, organized, or proofread. Real-Time Rendering looks like it might provide a good high-level understanding of shaders and rendering, but it’s not specific to OpenGL or GLSL, and I’m not really interested in advanced photorealistic shader effects. Maybe if there are any book stores still open in this town, I’ll see if any of these books are on the shelf so I can assess them before I buy.

Anyway, even after I had scraped together a modest understanding of the concepts behind the shader pipeline, applying that knowledge in practice took longer than I anticipated. It was fairly frustrating at times, partly due to some (minor) shortcomings of cl-opengl and lispbuilder-sdl, and partly due to me misinterpreting some vague parts of the OpenGL documentation. There were many times when I felt like putting the shader pipeline aside, and just using the fixed function pipeline for a while, so I could keep my momentum up. But I stubbornly stuck with it, and now I’ve got a working, reusable rendering framework built atop the shader pipeline. Huzzah!

My next task is to write code to algorithmically generate mesh data for several geometric primitives. Yesterday, I finished the code for a rectangle mesh built from triangle strips (well, actually one long strip with degenerate triangles connecting each row). Here’s a screenshot of a 10×10 rectangle mesh rendered in wireframe mode:

It’s not much to look at in itself. I could have achieved that in an afternoon using the fixed function pipeline! But that humble mesh, rendered here using the simplest possible vertex and fragment shaders, represents weeks of learning. Now that I have the basics working, I can write more interesting shaders to morph the shape and create cool visual effects. For example, Ambienome takes place underwater, so I’ll probably write a shader to make things sway and ripple as if they were being affected by a water current.

Next, I’ll be writing the code for a triangle mesh, and then a circle mesh. I have plans for more shapes, like teardrop, leaf, ring (circle with a hole in the middle), and tentacle (a Bézier curve with thickness controlled by another Bézier curve). But, I’m going to save those for later. Once I’ve got rectangle, triangle, and circle, I’m going to take a step back from the low-level foundation for a while, and start fleshing out higher-level concepts like components, creatures, and scenes.

Even though I have seven years of experience with Ruby, and my own game library, Rubygame, already available and ready to use, I’ve decided not to write Ambienome in Ruby. Instead, I’m writing it in Common Lisp, using OpenGL and OpenAL for graphics and audio.

Libraries

OpenGL and OpenAL were natural choices, since they’re free, cross-platform, and widely supported with pretty much any programming language. I would have chosen these no matter what language I settled on.

OpenGL doesn’t provide window management or input event processing, though, so I’ll need another library to handle that. The main choices for that are SDL, GLFW, and GLUT.

GLUT is a very popular and widely supported framework, but I don’t like how you have to shape your entire program structure around GLUT’s expectations, so I quickly dismissed it. SDL is also popular, but my years of experience with it have made me keenly aware of its shortcomings. So, I decided to try GLFW first.

GLFW is pretty nice, like a less obtrusive version of GLUT. I played around with it long enough to create an abstraction layer for opening and closing the window, but then discovered that GLFW currently has no API for setting the window’s icon. That may seem like a minor detail, but it was enough to nudge me into reconsidering SDL.

Two other points in SDL’s favor are that I’m already familiar with it — unlike everything else about this project! — and that SDL 1.3 will have support for multi-touch devices, and possibly multiple mice, both of which could provide some cool ways of interacting with the game. SDL 1.3 hasn’t been released yet (after many years…), but I might try taking a snapshot of the development repository, and working with that.

So, I ported the window abstraction layer from GLFW to SDL (1.2), and added an input event abstraction layer. I’m not entirely happy with it, so I might reevaluate GLFW (or even GLUT, since I’ve read that freeglut and Apple’s GLUT let you avoid the glutMainLoop function). But for now, SDL will be enough while I figure out the challenging new stuff with OpenGL and OpenAL.

One final library worth mentioning is Clutter, which I’m considering using to create the in-game GUI. When it gets to that point, I’ll try Clutter, then look into alternative libraries if Clutter doesn’t work out. As a last resort, I could just roll my own GUI widgets, but I’d rather avoid that if I can.

Lisp

OpenGL and OpenAL are pretty popular, “safe” choices of technology for game development. There are countless web articles, tutorials, books, and other learning resources available, and many experienced programmers I could turn to for help if needed. Both libraries are available on Linux, Mac, and Windows, and have bindings for many programming languages. There’s nothing particularly surprising or unusual about using OpenGL or OpenAL.

The choice to use Common Lisp, on the other hand, is likely to raise a few eyebrows. After all, it’s secret alien technology powered by excessive parentheses!

Lisp, despite its elegance and power, is not nearly as commonly used for game programming as the C language family (C++, C#, Java, etc.), Python, or probably even Ruby. If I had to speculate as to why Lisp has a low adoption rate among game programmers, I’d cite the usual factors that affect Lisp’s adoption rate in general: difficulty getting started (choosing and installing a Lisp implementation and IDE, finding good tutorials, etc.), the “strange” syntax, Lisp’s reputation as being more academic than practical, and the Lisp community’s reputation (whether deserved or not) for being somewhat elitist and unfriendly. Also, many game programmers are obsessed with raw performance, as if brute-force number crunching was the key ingredient in making a fun game. Those programmers will usually stick with C or C++, and dismiss anything else as being “too slow”.

But, I don’t particularly care about Lisp’s popularity or why other people like or dislike it. I care about having an interesting, powerful language that fits the project well.

• Like Ruby, Python, etc., Lisp promotes rapid prototyping with concise and flexible code, and sustains creative flow in a way “stop-and-compile” languages like C cannot.
• Common Lisp can be compiled to native executables, and I can declare optimizations for those situations where number crunching efficiency is needed, such as generating custom audio data.
• Lisp’s notion of “code as data” should work well with the concept of expressing behaviors as objects, and saving each creature’s behavior in a file.
• Lisp’s macro and package systems should make it really easy to create a safe, simple, and efficient domain-specific language to allow users to define new creature parts and behaviors.
• Lisp is a novel language to me, so this project is sure to be an interesting learning experience!

Before settling on Common Lisp, I also considered CoffeeScript and Clojure. CoffeeScript (using HTML5 Canvas and Audio elements) was very tempting because of the ease of deployment and cross-platform compatibility. But, HTML5 does not yet have a workable API for generating and manipulating audio data, and running in a browser would also limit my options for player-to-player file sharing and multiplayer gameplay.

As for Clojure: it’s a really interesting Lisp, and the ability to use Java libraries and compile to Java bytecode could be useful. But, learning to write a game using only immutable data structures and STM/agents would be a bit too much of a novel learning experience, and this project is going to be a lot to wrap my head around already. Also, Clojure’s strengths, such as safe multithreading, don’t provide much of a practical benefit to this particular project.

So far, I’m really enjoying Common Lisp. I’ve been reading Practical Common Lisp, which is a really excellent resource for learning Common Lisp, and available online at no cost. I’m certainly not a Lisp guru yet, but I can already feel the amazing expressive power and freedom Lisp provides. This is sure to be an interesting project!

I’ll be blogging about my development progress, both to solidify my ideas by putting them into words, and to keep myself “accountable” for maintaining progress. I’m starting off with a two-part announcement and introduction to the project. In this part, I’ll describe Ambienome’s concept and a tentative list of progress milestones. In the second part, I’ll write about the tools (programming language and library) I’m using, and why I chose them.

Concept

Ambienome will be a freeform creative toy/playground, where players design and interact with colorful, musical, AI-driven aquatic creatures. The concept is heavily inspired by Toshio Iwai’s Electroplankton, and by Will Wright’s Spore.

I loved Electroplankton, but wished there was a way to create new types of creatures, to have multiple types of creatures in the same scene, and to save and share scenes with my friends. Spore’s easy-to-use method for creating new creatures out of parts, and integrated system for saving, uploading, and downloading creations online, suggested a workable model for creating and sharing Ambienome creatures and scenes.

Roadmap

This is a very novel kind of project for me, and it will have many challenges to overcome and problems to solve. I can’t even guess how long it might take to finish, so I’m not going to bother speculating about a project timeline. But, I have plotted out some of the major milestones for the project, in terms of functionality to be implemented:

1. Players can interact with a variety of built-in creatures, and watch/hear them interact with each other.
2. Players can assemble new types of creatures from a variety of built-in parts.
3. Players can import and export creations (creatures and scenes), to share with other players.
4. Players can upload and download creations from a central site (a la Sporepedia), and rate and comment on other players’ creations.
5. Players can design custom parts, and share them with other players.
6. Multiple players can build and interact with the same scene simultaneously, over the network.

I consider 1-3 to be essential, and the rest to be icing on the cake. I’ll do at least one release per milestone. I haven’t decided yet whether this will be an open source project or not. If it seems to be turning out well, I might try to make some money off it.

Currently, I’m still laying the foundation, implementing low-level functionality. For reasons I’ll discuss in the next post, I’ve chosen not to use Ruby or Rubygame for this project. So, in many ways I’m starting from square one, reimplementing the basics: opening a window, handling input events, drawing to the screen, managing the game loop, etc. Well, not exactly square one — I’m building atop other libraries, and I can borrow the basic concepts from Rubygame, so I don’t have to solve those problems all over again.

In the next post, I’ll be writing about the programming language and libraries I’ll be using, and why I chose them.

Before I get to that, though, a brief review of Sup. What first attracted me to Sup was the fact that it’s written in Ruby (with fairly clean code, from what I’ve seen), and has a fairly extensive system of hooks for customizing and extending the client behavior (the hooks are also written in Ruby). Also, it has a nice label-based filing system and powerful search utilities (a la Gmail), rather than the tired old folders metaphor. It runs in the console and is entirely keyboard driven, the workflow is pretty quick and easy, and you compose messages in the text editor of your choice (emacs, vim, etc.).

Those are the strong points, for me. Unfortunately, there are a number of issues that made me decide to look elsewhere. Foremost is the console-based interface using Curses, which holds it back in a variety of ways. One issue I ran into was that there’s no way to define keybindings that use the Alt key; Curses simply doesn’t support that. Sup also doesn’t seem to be able to render non-Latin characters, which is occasionally an issue for me. And naturally, being console-based, modern luxuries like HTML mail and previews of attached images are out of the question.

So, Sup turns out not to be my cup of tea after all. But, maybe it is yours, so I’ll share the fruits of my brief foray.

• The first obstacle was the fact that Sup, like many Unix-style “mail user agents”, has no facilities for the actual fetching or sending of email over a network. Instead, it leverages programs like fetchmail, offlineimap, and sendmail to handle those things. This Unix-style mail system was something I had no prior experience in, so I had a lot to learn. After a lot of research and a fair bit of frustration, my advice is to forget the traditional fetchmail and sendmail, and instead use getmail and msmtp. They are a lot simpler to set up, and do everything most people need.

• After I got my email imported into Sup, I wanted to change some key bindings to be more comfortable. In particular, being an Emacs user, I am used to Ctrl-n/Ctrl-p to move the cursor up and down lines, while Sup defaults to the vi-style hjkl bindings (which make no sense for me, using the Dvorak keyboard layout). Unfortunately, I could find very little documentation about how to change the keybindings, other than the fact that there was a “keybindings.rb” hook that I could write… if I knew what to write in it.

  Undeterred, I dove into the source code and learned all about the keybindings system. Then, I turned that knowledge into a comprehensive CustomizingKeys wiki page, so that the next guy wouldn’t have to dig through the code like I did. That was a fun afternoon project.

• Next, I realized I needed a system to filter my incoming mail, automatically apply appropriate labels, and so on. Sup doesn’t have a filtering system like that built in, but it’s fairly straightforward to create one by taking advantage of Sup’s hook system. There is some documentation about this on the AutoAddLabelsToNewMessages wiki page, but as I started writing my filters, I began to realize that, although writing ruby code for every filter gives you a lot of flexibility and power, it also involves a lot of redundant code and is not very easy to read or maintain.

  So, I started writing my own filter system, which loads rules from a YAML file (see the example filters file). I’ve posted the result to my sup-filters repository on GitHub. Actually, what I’ve posted there is even more polished and better documented than what I used myself. Even after I had decided I wasn’t going to use Sup myself, I spent a couple days cleaning it up for other people, to call it a job well done. That’s Just How I Roll ™.

Anyway, if you use Sup, I hope you will find this stuff useful. As for myself, I will miss Sup’s labels and easy searching, but it’s time to continue my quest for a well-fitting email client. Next on my list to try is Wanderlust. If you have any recommendations for other clients to try, leave a comment!

Version 0.4 fixes several major bugs related to finding and loading dynamic libraries:

• PathSet now properly expands globs (“*”) in file and directory names, like it was supposed to.
• PathSet now works even when a directory path has no trailing separator (“/” or “\”).
• Fixed a typo which caused NameError when using load_library with the default paths.

You can get Nice-FFI:

• Via RubyGems: gem install nice-ffi
• Via Github: http://github.com/jacius/nice-ffi/
• As a tarball or zip.

Please send bug reports and feature requests to the issue tracker on Github.

Version 0.3 has a few changes:

• The default PathSet has a less restrictive filename search pattern, so it is better at detecting libraries.
• Struct.new can now accept FFI::Buffers. Buffers are like MemoryPointers, but with improved performance on JRuby in some situations.
• Struct.new will now create a Buffer instead of a MemoryPointer when you create a new struct instance by value from an array, hash, bytestring, or another struct.

Also, since I never posted an announcement for 0.2, here are the changes for that version:

• Added automatic memory management to the Struct class, if you define a “release” class method. It can be disabled per-instance by passing :autorelease => false to MyStruct.new.
• Added new OpaqueStruct class. This class is useful for when the struct layout is not declared in the C library header. It also features automatic memory management like Struct.
• Added Struct.typed_pointer, as a convenient alternative to TypedPointer.new(MyStruct). You can also pass :autorelease => false to disable memory management for all structs created via that TypedPointer, which is useful when using the TypedPointer as a function return type.
• Structs can now be dumped to a bytestring (Struct#to_bytes), and also created from a bytestring (by passing a String to Struct.new).
• Revamped the PathSet class. It now has separate patterns for paths (directories to search) and files (filename patterns), to be more flexible and easy to customize (e.g. you can add a new directory without caring about the filename pattern).
• Removed Pathset#<< and Pathset#>>. Use append and prepend instead.
• Renamed Library::DEFAULT_PATHS to PathSet::DEFAULT.
• Added Library.optional, for when you want to continue loading the library even if a certain function or variable doesn’t exist in the library (e.g. because it’s not defined in old versions). It takes a code block, and will rescue and emit a warning if there is a FFI load error.
• Also added Library.opt_func, which is the same as doing attach_function within an optional block.
• Added some convenience aliases in Library: func = attach_function, var = attach_variable.
• Nice-FFI no longer depends on the “need” library. We don’t need no stinking need!

You can get Nice-FFI:

• Via RubyGems: gem install nice-ffi
• Via Github: http://github.com/jacius/nice-ffi/
• As a tarball or zip.

Please send bug reports and feature requests to the issue tracker on Github.