Friday, August 31, 2007

Music Control Tools: Python-Based

segno
In the previous part of this series of articles I looked at dedicated programming languages for music creation. But why invent a new language for this one special domain? Surely it makes more sense to use one of the many existing languages, provide libraries for required protocols and interface with the correct hardware?

Well, that is not necessarily true. As we saw in our short look at ChucK, new paradigms of coding ("strictly-timed") may enhance development in a specific domain.

However, the advantages of adopting an existing language are obvious. The developer need not work out their own grammar, syntax and tools, but can rather concentrate on the actual audio part of the problem.

In this article I will look at audio development tools based in my language of choice, Python. These will all have the advantages of that language: clear syntax, pragmatic mix of functional, object-oriented and imperative models, strong OOP implementation, etc.

PyGame, based on the Simple DirectMedia Layer (SDL), has lots of graphics functionality, but also plays back MPG files and accesses devices like CD-ROM drives and joysticks. It can queue and play music streams, and contains sound sample manipulation, but is not robust enough in this regard for our purposes. For instance, it can handle only one audio stream at a time and has no DSP functionality.

pySonic is a wrapper for the FMOD library and provides sound file playback and recording, MIDI, 3D sound and support for many formats (wav, aiff, mp3, ogg, mod, etc.). FMOD is not free or open source, so one would have to abide by their license restrictions.

PyPortMidi provides MIDI I/O, although the latest binary is for only Python 2.4. Likewise here's a module for OpenSoundControl (OSC) client functionality.

PySndObj is a wrapper for the SndObj Sound Object Library which provides realtime audio IO and MIDI input (though not on all platforms). Currently this is too immature to recommend for the task at hand.

PyMedia is a library for sound file playback and recording which supports wav, mp3, ogg, avi, divx, dvd, cdda etc. It also has some DSP functionality, namely resampling and frequency analysis.

athenaCL is an interactive command line program specifically designed for algorithmic composition and pitch models studies. It has a host of tools in this regard, outputting results to Csound, MIDI, audio file, XML and text formats. While not requiring Csound, it is tightly integrated with that programme, containing Csound format instrument. (Without it, results can nonetheless be rendered to MIDI files.) This programme looks excellent within the domain it is targeting.

Finally, MusicKit includes serial and MIDI I/O, scheduling and synchronization, real-time (or non-realtime) synthesis (FM, wavetable, physical modeling etc.) and DSP. It supports quadraphonic sound, MP3 and Ogg/Vorbis plus multiple inputs and outputs. Furthermore, the package comes with high-end tools including a sampler, sequencer and score player.

Of all the packages here, it potentially provides the most robust toolkit for sound creation and manipulation. Unfortunately, not only is the GUI on Windows incomplete, but so is MIDI support and the DSP functions. Critically, MIDI and DSP are also missing from the LINUX version. (MusicKit had its genesis on the NeXT and so works just fine in OpenStep... in case that helps you.) There has been no version since May 2005, so it is fair to say this tool is ripe for salvage.

The conclusion of this article must therefore be that there is no Python-based system that meets our needs at present. And while it would be possible to scrape one together from bits and pieces of the packages listed here, that would be far more work than could be justified.

Reference: The wiki entry PythonInMusic contains tools I have not covered here, such as those for media playback and cataloging.

I thank the Arts Council for their support in this research.
Thursday, August 30, 2007

Music Control Languages: Specialised Tools

segno
In the first article in this series, I set the criteria for my examination of audio control languages. In this second part I examine the specialised tools available for the purpose. The series will then be concluded with a look at Python-specific solutions.

Though the last few items are subjective and optional, all products discussed meet the first four criteria unless otherwise mentioned:
* open source
* free of charge
* cross-platform (Mac, Windows, LINUX)
* programmatic control with OOP paradigm
* mature implementation
* efficient resource handling
* powerful, expressive, clear syntax
* strong user community

A good amount of software has emanated from IRCAM in France, possibly the foremost institution of electro-acoustic music in the world. Everyone from Pierre Boulez to Aphex Twin has been shaped by IRCAM. So it makes sense to start our examination of software with their graphical audio software Max/MSP. This is popular, but not free. jMax can also be quickly ruled out, since it is a non-OOP visual programming environment for building interactive real-time music applications. Though it is free and open source, the Windows version is only a beta and might not be robust enough for critical use.

OpenMusic is another IRCAM package, also graphical, running only on Macintosh (Linux port is on the way). I include it here for completeness only.

Pure Data (PD) gets the thumbs up from its users. It too derives from IRCAM work on Max. PD may be used for audio and MIDI processing, but also graphics and video (it has a fast dedicated openGL library, GEM, for this purpose). The forum is active and there are many resources on the web, though the home page does link to much dead content.

However, PD is a "patcher" programming language, in which one graphically links together blocks that perform different functions. It is not object-oriented but is important enough to consider, especially as regards the large number of available extensions. For example, RTC-lib, the Real Time Composition Library, provides high-level compositional algorithms. There are also mature interfaces to devices such as the Arduino.

So how do PD and Max/MXP differ? The latter has better documentation, more bundled extensions and a smoother usability experience. Plus it has some programmatic niceties, at least according to those who know it well. For example, the order of processing is from left to right in the Max flowchart, whereas in PD it depends on when you created the items. That is, items you added to the flowchart first when building will execute first when running! That makes zero sense.

Moving across the water we next look at Processing, a programming language which sprang from the MIT Media Lab. Designed specifically for interactive application it is written on top of Java, and has the distinct advantage of being able to run directly in a web browser. There is a good amount of support on sites like Processing Hacks. A MIDI library is available, as is the Sonia Library for advanced capabilities like multiple sample playback, realtime sound synthesis and analysis.

However Processing is still in beta, and this shows in its inefficient implementation. According to this thread it's not quite ready for prime time in its performance and memory handling. But if your emphasis is on visually interactive applications (perhaps for your mobile phone) you should give it a look.

SuperCollider is an interpreted object-oriented language that has two components: a client which communicates with a realtime sound synthesis server via OSC. This architecture is flexible enough to permit "live coding", changing code in real-time as a performance tool. Code objects and process declarations can also be shared and modified over a network.

Because SuperCollider has support for MIDI and serial port I/O (LINUX only), it can be used to control, or controlled by, pretty well any hardware device. So you can scratch with your Wii remote, should that be your thing.

The Windows version, known as Psycollider, is in beta. The fact that the last version update was over a year ago is not encouraging.

ChucK is a "strongly-timed" audio language optimised for real-time synthesis, composition, performance and analysis. It supports MIDI, OSC and multi-channel audio through a command-line or Integrated Development Environment (IDE). Live coding is supported.

Several of the programming paradigms in ChucK are novel. Unlike other audio systems that have a separate audio signal and control signal, ChucK has a unified timing mechanism. One of the built-in types is the "shred", a non-preemptive process with its own namespace. (However, other than this and the public space, there is no other control for namespaces.) Shreds can be sporked (I kid you not) which essentially means instantiated in the virtual machine. Shreds are automatically synchronized and may be scheduled (though ChucK calls it "shreduled" -- silly rabbits!).

Garbage collection is by reference counting, but this is still being implemented. In fact in many places in the documentation there's the phrase "not completely implemented" or even "will leak memory". The last version was 23 August 2007. With further active development I hope ChucK will be ready for prime time soon.

Csound is the grand-daddy of computer programming languages, its genesis reaching back before the nineties to Music360 in 1957. There are vast resources available, but Csound is closer to a lifestyle than a programme, requiring a dense incremental development cycle. The language itself is limited, a fact acknowledged by the Csound community, which has responded by embedding a Python interpreter. This creates a somewhat complicated schizoid syntax for coding.

Python can also be used on the "outside" of Csound, to manipulate CSound files, speeding the creation of sounds. The article Sound synthesis with Csound contains the required module.

A number of front-ends have been developed to ease development. For example, Cabel provides a GUI with Python script control.

While I am sure that a given collection of tools might make work in Csound highly efficient, determining what that might be is the work of months. Unless one's full-time job is computational music, it's difficult to recommend Csound. (I can see the flames igniting from here.)

So, conclusions?

* Pure Data is a rough-hewn many-faceted system that is the best of the free graphical solutions. If you want to spend money get Max/MXP instead.

* SuperCollider has a solid open architecture that suits networked and multi-user solutions, so long as one is willing to forget Windows.

* ChucK is still a work in progress but its "strongly-timed" declarative syntax is an innovation. As is some of the nomenclature! Watch this one.

* I will use Processing to hack my phone into a musical instrument, but not use it for mission-critical work as of yet.

* Csound is what you should use once you get employed by a university to make sounds.

Addendum: Sometime after I first wrote this document I came across the Wikipedia "Comparison of audio synthesis environments" here. An exhaustive list of software for algorithmic composition is located at algorithm.net. This includes many products not on my list but none that meet my criteria.

I thank the Arts Council for their support in this research.
Tuesday, August 28, 2007

Music Control Languages: Criteria

segno

A few months ago I investigated control languages for MIDI and OSC data as part of an arts research project. I thought it might be useful to share this work with anyone else interested in programming audio.

Currently I use Reaktor to create my own software instruments, sequences, audio control signals and music generators. But in this instance I require an open source and free product, so that any of my development can be easily shared. I require support for my primary operating system, Windows XP, butalso needed compatibility with LINUX and Mac OS X for the widest deployment possibilities. With these criteria in mind, I was ready to evaluate possible solutions.

In order not to get buried under hundreds of possibilities I decided to disallow graphical-only products. The reason is that I need fine-grained control and the ability to replicate objects programmatically. Reaktor and other graphics-based applications are deficient in this regard. Though you can easily group any number of functions as an "object", and subsequently duplicate these, you have a problem if you need to make a change. Rather than make a change to one object and have it reflected in all instances, all copies must be changed manually.

In other words I need an object-oriented environment, not just something that looks like objects but does not fully partake of the OOP paradigm.

My goal is to build a control system for a sound installation. This needs to process environmental data in real time, converting this to some sort of signal flow (MIDI, OSC, serial data stream). Hence any product that is not mature and stable enough to run for days unattended does not qualify. Efficiency is also a concern. The fewer system resources (RAM, drive space, processor speed) required, the better.

All of the products discussed in this series of articles meets these criteria unless otherwise noted. (Though I remark in passing that the stability and efficiency requirements are harder to pin down than the others.)

In addition, there were further criteria in the "nice to have" category.

The language should be powerful, expressive and easy to read. I am used to Python, which reads more or less like pseudo-code. Anything too abstruse now seems like a waste of time. Remembering that the end product is an aesthetic product, I do not want to use any tool that too severely shunts aside the artistic side of my creativity.

A strong user community with helpful documentation, examples, and so on helps a newbie not only get started but continue in the right direction without reinventing the wheel.

I investigated two broad classes of solutions. The first are specialised languages built especially for audio. The second are packages or libraries that give a general-purpose language strong abilities in this specific domain. Since my high-level language of choice is Python, I focused my attention on modules for that language. If I preferred Java or Lisp or Forth or some other language, there would be a myriad of additional solutions.

It turns out that there is some overlap between the two solution categories, since Python can be embedded in specialised tools or used as a higher-level control structure for same. (As it commonly is with Csound, for example.)

In the next article I will look at the specialised audio control languages.

I thank the Arts Council for their support in this research.
Sunday, August 19, 2007

Wasp Templating: An Introduction

Wasp
Templating involves putting a text document (the template) though a series of transformations to produce a finished document. In word processing this is handy for mail merge operations or the insertion of boilerplate text. For the web, it is often used to automate the creation of HTML files. Templating provides several advantages, the most obvious being a decrease in repetitive markup and enhanced organisation.

There are many different approaches to web templating, as illustrated by the variety of Python tools devoted to the task.

Wasp follows the following design precepts:
  • Editing XML-based templates is too much work

  • Adding descriptors to existing HTML tags is a muddle

  • Use your usual HTML editor for the job

  • Be friendly to web designers

  • Use Python syntax so there's nothing new to learn

  • Minimal clean implementation

  • No performance bottlenecks


Wasp 1.x took a minimal approach that did not allow arbitrary Python code in HTML files, though it did provide an execute tag for running custom functions. While I still believe that a separation of markup and programme code is the best way, I acknowledge that it is almost impossible to write a web app without mixing the two in some fashion. Either your Python writes HTML or your HTML embeds Python. Either can result in a code base that is difficult to maintain.

The upcoming Wasp 2.0 does away with execute, instead allowing arbitrary Python code in embedded tags. This is much more powerful, and proved to be easy to implement.

You can now have markup that defines and uses functions, like so:
<?w def row(arg):
y = arg + 1
prn('* hello %d' % y) ?>

Here is a list:
<?w for j in range(x):
row(j) ?>

If this template is compiled with x=4 the result is:
Here is a list:
* hello 1
* hello 2
* hello 3
* hello 4

(This illustrates that Wasp templating need not be used only for HTML.)

The additional Wasp tags have been implemented anew as Python functions. Thus their syntax has changed. Where before you included a separate file with the include tag, like so:
<?w i:header('title') ?>
some stuff in between
<?w i:footer ?>

You now do it this way, the name of the file becoming the first argument to the function:
<?w i('header', 'title') ?>
some stuff in between
<?w i('footer') ?>

I have kept two useful features: the arbitrary nesting (include a file that includes a file that...) and the fact that arguments can be passed to the included bits. So in the previous example the header include file would receive an argument title which would get inserted where you use the tag <?w 1 ?>. If there was a second argument it would be interpolated where the code said <?w 2 ?> and so on up to 9. (I must say that if you are passing nine arguments there is likely a better way to do things!)

Finally, to expand a macro into its full text (as defined in the configuration file), use this function:
My name is: <?w m('my_name') ?>

If you were a famous surrealist, this might output:
My name is: Max Ernst

That's it. I've done away with all the other tags. The special tags that worked only in batch mode (seq and index) will be zapped. If people need them I am sure they can be created through plugins. The conditionals (if, endif) will go too, since I think that with full Python code in place they'll be irrelevant.

There is one other major change that you might have noticed. In Wasp 1.x you output text to the web page from within your code by simply using the print command. In Wasp 2.x you must use functions for this task as well. prnz(s) outputs only the string whereas prn(s) adds a line break in accordance with how print works.

The reason for this change has to do with the architecture of the server code. Previously I was redirecting standard output in order to capture the print statements. This works very badly in a multi-threaded application!

If you want more than what Wasp templating provides, I recommend Cheetah. I hope to provide hooks to slot in external templating packages in a version after 2.0.

Does Wasp templating do anything radical or new? No. Does it do things simpler than any other tool? Yes. For me that was a good enough reason to roll my own solution to this much-solved problem.
Sunday, August 19, 2007

Free Books Online

Here is a list of sites which offer books, magazines and other documents for free download. Rather than being comprehensive, I am trying to highlight the best or largest sources. You should find enough here to while away many an hour reading on your computer... or PDA or whatever.

There are two main types of resources, those which are protected by copyright but which are nonetheless free of charge, and those which are in the public domain. In the USA any book published before 1923 is now free of copyright, but rules differ in other countries.

Public domain works are not only free as in beer but free as in speech.

Project Gutenberg is the grand-daddy of copyright-free e-texts, with over 20,000 titles available in plain text. For example, check out The Notebooks of Leonardo Da Vinci or The Devil's Dictionary by Ambrose Bierce (still good fun). They also have a number of affiliates, providing even more.

For example, Project Gutenberg of Australia follows the copyright laws of that country, which makes public works of authors who died before 1955. This 32 year difference translates into a good number of additional documents.

The University of Pennsylvania indexes 25,000 Books Online, most in PDF and text. Amazingly this collection has been managed by a single individual since 1993. It includes not only public domain books but those that are free for personal noncommercial use.

Bartleby has a large collection of reference works, including the Columbia Encyclopedia and American Heritage Dictionary. This was once impressive, but with so many other references online, the need to get an electronic copy of a printed document is less compelling. The site houses what is apparently the largest Internet quotation collection (86,000) as well as fiction, non-fiction and over 10,000 poems. The files are accessible through over a third of a million web pages.

Asiaing has a selection of books and magazines, freely available in PDF. Start with Frankenstein and Alice's Adventures in Wonderland before working your way up to The Complete Works of Friedrich Nietzsche. Why this site exists and who runs it is a mystery.

There are also a number of sites which exist to publish new works.
For example, Booksie allows you to post fiction, non-fiction and poetry, and read similar works from others, all without fee.

If you have any other suggestions, please comment!
Sunday, August 19, 2007

Online Books: Spotbit

Want to read free magazines? A site called Spotbit, apparently operating from Malaysia, hosts a good number of such in PDF and their own executable format. The problem is wading through all the crap since the interface is not that search-friendly.

Artzmania hosts their magazine there, along with the special issue Best collection of Super Web 2.0. Why you'd want to download a large PDF to view images of websites you could go to in one click I don't know. But their regular issues are well worth a look.

Of potential interest to publishers, the Spotbit software can be freely used to create your own e-book from PDFs, word processor documents, or even screenshots. The problem is, I don't think it's a great idea distributiong EXE files... they look like viruses.

Besides, it took ages for one of the files to load. When it did, the document was broken.

Spotbit also acts as a publishing house for documents that authors want to charge for. But their info pages do not say what their royalty rates are, and they don't pay until the balance hits 100 dollars. So much for micro-economics.

Spotbit hit the news a little over a year ago for offering free copies of copyright magazines... but now they've conformed to the law.

In conclusion: their ethics is questionable, the economic model is flawed, the technology substandard, the acceptance factor low and the website itself poorly implemented. I recommend you grab a copy of Artzmania and leave it at that.

I will look at other eBook sites shortly.
Sunday, August 12, 2007

"Young Hearts Fail": The Death Of Tony Wilson

"I used to say some people make money and some make history - which is very funny until you find you can't afford to keep yourself alive."

Tony Wilson is gone. A man who did so much for civic pride in Manchester, basically re-inventing the city as a hip place to be in the eighties and nineties, has died. In his last days he could not afford 3500 pounds a month for the drug to treat his cancer, and was denied coverage for same under the UKs National Health Service.

Yes, this is the true face of capital.

Working as band manager and label owner of iconic Factory Records would seem an odd profession for a man unable to bring himself to make a proper contract with his bands.

"The musicians own everything. The company owns nothing. All our bands have the freedom to fuck off."

So read the only agreement, famously signed in his own blood.

In his book 24 Hour Party People he conflates myth and reality. "Between the truth and the legend, print the legend" he quotes someone as saying. (Yes, Tony, it was from "The Man Who Shot Liberty Valence".)

This blurring of the principle that separates truth from fiction makes him great in our imaginings. But we can put the "entertainment" aside (if we like) and look at some facts.

Tony Wilson was the first to put the Sex Pistols on television, one of the first to start an independent label in Britain, released the first single from OMD, let Peter Saville recreate graphic design for an era, and was possibly the biggest Vini Reilly fan ever.

Furthermore he allowed Joy Division to be who they wanted to be. Against all expectations and logic.

Somewhere I have that first release on his new label. A Factory Sample was four artists with two songs each: the influential Cabaret Voltaire, the odd John Dowie, the only recordings from the original Durutti Column band, and two tracks that would change every life they came into contact with.

"I need you here today.
Don't ever fade away
Don't ever fade away
Don't ever fade away
Don't ever fade away
Fade away
Fade away
Fade away
Fade away
Fade away
Fade away
Fade away"

That's what Ian sang on the first song, "Digital".

A line from the second, "Glass", I quote in the title of this entry. Perhaps this glass cuts too close to the bone. But then again sometimes bone is all we have.

Tony Wilson died 10 August 2007 at age 57, of a heart attack. He was a young heart always who did what he could for his city.

I've never even been to Manchester. But I will remember.
Friday, August 10, 2007

Line Of Code Counter

Someone was asking for a way to count lines of code, so I had a look in my toolbox and found the following. By providing this I do not wish to support such code metrics in any but the most general sense.

For what can be learned from counting lines of code? Should we assume that the programmer who produces more bulk is more productive. Or that the language that produces less is more efficient? I wouldn't want to (re)start any of those flame wars!

This function uses a variant on my directory tree walker, as published in the Python Cookbook. It's a straightforward but nonetheless helpful function that wraps os.walk with some useful extra functionality.

The LOC function itself has some basic logic to skip comment lines. This could get confused by the use of triple quotes for multiline strings. To prevent that, do not start such a line with the quotes.

import os
import fnmatch

def Walk(root='.', recurse=True, pattern='*'):
"""
Generator for walking a directory tree.
Starts at specified root folder, returning files
that match our pattern. Optionally will also
recurse through sub-folders.
"""
for path, subdirs, files in os.walk(root):
for name in files:
if fnmatch.fnmatch(name, pattern):
yield os.path.join(path, name)
if not recurse:
break

def LOC(root='', recurse=True):
"""
Counts lines of code in two ways:
maximal size (source LOC) with blank lines and comments
minimal size (logical LOC) stripping same

Sums all Python files in the specified folder.
By default recurses through subfolders.
"""
count_mini, count_maxi = 0, 0
for fspec in Walk(root, recurse, '*.py'):
skip = False
for line in open(fspec).readlines():
count_maxi += 1

line = line.strip()
if line:
if line.startswith('#'):
continue
if line.startswith('"""'):
skip = not skip
continue
if not skip:
count_mini += 1

return count_mini, count_maxi


And here's how you use it on the current directory:

print '%d : %d' % LOC('.', False)


I know there are common cross-language tools to do this, but whipping up a pure-Python implementation was quick and saves me installing YAT (Yet Another Tool). Also, this code is freely available under the Python license. It has also been published as recipe 527746.
Wednesday, August 08, 2007

Friendly Readable ID Strings

One often needs unique ID strings to tag processes, threads, files or anything else that might be created in quantity. Traditionally these are created based on PIDs or similar system values. But it is not easy to visually recognise these strings, which makes debugging more difficult than it need be. This recipe creates readable and pronounceable ID strings that are much easier to work with.

I have submitted this to the Python Cookbook but thought I would write it up here as well. The code is freely available under the Python license.

The inspiration for this recipe is the fact that I hate debugging systems by trying to track IDs that look like "kSbT73oQ". It's much easier on the brain to scan strings like "levokosi". These are also pronounceable, so one can communicate with other members on the development team. "Hey dude, what's up with file sadofabi -- it appears to be locked."

I many cases one does not need billions of IDs, so restricting the strings to alternating consonants and vowels, and a reasonable length, gives us plenty of scope. The function GetFriendlyID() generates these of length 8, using the built-in Python pseudo-random generator. If this is not good enough, increase the length or use a more random module. But before you do so, realise that the algorithm here allows over 40 million unique strings.

(Of course it is sometimes useful to have ID strings based on PIDs or whatever, so they can be tied back to the originating process. In those cases this recipe is not applicable.)

The GetUniqueFriendlyID() function sketches out a typical implementation, where we want to ensure the generated IDs are, in fact, unique.

The module-level code does a simple test so we can inspect some resulting strings. Baby name generator, anyone?

from random import choice

def GetFriendlyID():
"""
Create an ID string we can recognise.
(Think Italian or Japanese or Native American.)
"""
v = 'aeiou'
c = 'bdfghklmnprstvw'

return ''.join([choice(v if i%2 else c) for i in range(8)])

def GetUniqueFriendlyID(used_ids):
"""
Return an ID that is not in our list of used IDs.
"""
# trying infinitely is a bad idea
LIMIT = 1000

count = 0
while count < LIMIT:
id = GetFriendlyID()
if id not in used_ids:
break
count += 1
id = ''
return id

if __name__ == '__main__':
from sets import Set

print 'some sample unique IDs:'
used_ids = Set()
for i in xrange(50):
id = GetUniqueFriendlyID(used_ids)
if not id:
print 'something broke'
break
used_ids.add(id)
print id


Edit 10 August: Fixed resetting ID string in GetUniqueFriendlyID(). Fixed import. Changed implementation example to use a set to store IDs.

Edit 8 August: I've simplified the algorithm into what is basically a one-liner. This version requires Python 2.5 as it uses conditional expressions.
Saturday, August 04, 2007

Implementing Comments + A Lightweight CMS Data Format

So I followed Lee's suggestion from my last article, and had a look at the Tim Bray article on implementing a non-database post comment system. This describes a file-based approach with three folders where comment fragments live: incoming, rejected, and accepted.

My initial reaction was that this is massive overkill for the task. On further reflection I have come to the same conclusion. :-)

Instead I think I might write all of the contents for a given post to a single file. Since editing and deletion of comments will be uncommon activities, contention should not be an issue. Maybe you wouldn't want to run Slashdot on such a system but it should do for any normal volume use.

When a reader contributes a comment it'll be threaded in the appropriate physical place in the file, but marked with the status "incoming". This action might optionally send the editor an email. An editorial function will be available to view all such comments. They can be "approved" or "rejected", the status marked accordingly. When the article is built for output to normal (non-editor) readers, only approved posts will be gathered.

Physically, comments will be stored like this, using my previous file example:
/article/2007/07/21/My_New_Article.dat
/article/2007/07/21/Another_Article.dat
/article/2007/07/21/Another_Article.comment.dat
/article/2007/07/22/Short_Dissertation.dat


I'd rather have 30 comments in a single file, regardless of their status, than to have them in 30 different files scattered over three folders.

As far as formatting goes, I followed up on Helge's recommendation for YAML. Believe it or not, this looked too heavyweight for me... may as well use XML. But if you do need a pure-Python YAML implementation get PyYAML. If you prefer a wrapper around the C library for same, use PySyck, which provides Windows binaries.

A much simpler format is JSON which provides only two data structures: lists and dictionaries. But that's enough for pretty well anything I can think of. It is minimal, well-formed, readable and open. Not to mention the parsers are about a thousand times simpler and a hundred times faster than those for XML. (Yes, I am picking these numbers out of thin air.)

This comparison of the Python implementations reveals that Python-cjson is way fast but demjson is more tolerant of edge cases. The latter should be fine for the relatively small data payloads I'm needing, and has the benefit of being a pure-Python implementation.

So it looks like I'm heading down the rosy JSON path. I still reserve the right to use XML in the content of the JSON data structures. We'll see if that is necessary.
Friday, August 03, 2007

Ideas For Blog System File Storage


How should one best store the data for a simple, open blog system? One approach is to use a relational database and SQL queries. That is easy for those familiar with such technology, and allows one to depend on the database layer for concurrency, etc. Others prefer to stick more-or-less plain files into the local filesystem. This is easy for people who don't have database experience or access.

Before I made a decision on this I thought I'd better get a good idea of what I wanted to accomplish. I sketched out an entity-relationship diagram. Since I didn't have the back of a napkin handy, I used paper.

I realised that right away I'd need several tables. POST will hold the articles themselves, but could also be used for news, want ads, or other categories of postings, as described in CATEGORY. PERSON is the user of the system, the article author, etc. To describe what sort of a person, I added a ROLE table. Each role allowed one or more RIGHTs, allowing people to author articles, edit other people's articles, add comments, perform admin functions, and so on.

What is a post without COMMENTs? These attach themselves to a given article and also a person, as author. They must also thread and so require a link back to a previous comment. In order to enable useful searching and grouping, articles need to be tagged with different descriptive labels. However I don't think this is best accomplished with a new table.

With six tables in the database, this seemed too complex to manage in flat files. And what would the advantages of that be, anyway?
1. Works anywhere.
2. Low impact.
3. Simple to extend and diagnose.
4. Human readable.

But on the downside:
1. Concurrency must be manually handled.
2. Performance?
3. Manual search routines required.
4. Potentially messy organisation.

I don't think concurrency is a big issue for the articles themselves, which would rarely be changing. A single-user lock system similar to that used for wikis would be sufficient. Comments might also be handled similarly, but here there is more chance of contention since in a file-based system it would be difficult to handle each comment as an atomic unit. (Unless each comment was in a new file... no thanks.)

When retrieving one article at a time for reading, performance should not be an issue. However aggregate searches ("Give me all of Jean's articles for May") would require some sort of a globbing search over the directory structure. Still, there are fast ways to do that as well.

The need to write the searches in the development language (Python) could be seen as a bonus, since it does away with the need to comprehend SQL. We wouldn't be getting any automatic optimisations, but we wouldn't be needing them with the relatively simple text searches envisaged.

The organisational issue is down to design as much as anything else. Managing a separate database server, configuration files, etc. can get messy too.

So at this point I'm leaning towards a flat-file system, just to keep things very very simple. But this isn't going to work without rethinking the database and simplifying the data model. Third normal form is not going to cut it... we need to get flatter.

First of all I can get rid of CATEGORY and just store that as a text string in POST. For any given site it'll only be one of three or four values. Likewise RIGHT can be incorporated directly into ROLE. I'll abbreviate each right as a single letter. We need PERSON since we have to have a user table in any system that requires a login. And COMMENT must remain as a logical unit.

How will we store these four logical tables? COMMENTs are always read as affixed to the article they are for. Perhaps we can have one file for POST / COMMENT? In order to facilitate searching and to provide pretty URLs, I'd like to have these in folders like these:
/article/2007/07/21/My_New_Article.dat
/article/2007/07/21/Another_Article.dat
/article/2007/07/22/Short_Dissertation.dat
/news/2007/07/19/Welcome.dat
/news/2007/07/21/My_New_Layout.dat

The first element of the path corresponds to the category, the next three to the post date, and the final element is the file name (also article title). The file stat itself can be read for the last modification timestamp. This leaves the author, tags, description and content itself to be formatted somehow within the file.

ROLE is simple and fixed enough (once a system is initiated) to store in code as a data structure. That leaves only PERSON to deal with. I suppose it would be consistent to do this, embedding roles as part of the path:
/person/author/Max_Ernst.dat
/person/author/Jean_Arp.dat
/person/author/Man_Ray.dat
/person/editor/Andre_Breton.dat


But I'm not sure if that isn't one giant mess for hundreds of users. Maybe a single file with one line per person makes more sense.

And what format for these files? Some sort of an XML-based RSS-friendly format would save time. Atom 1.0 supports relative URIs, XML namespaces, the definition of an XML schema and so on, whereas RSS 2.0 does not. So let's make these files Atom fragments.

At this point I get the sinking feeling I'm merely reinventing a database. Does this design make any sense for a mini-CMS? Are the advantages of a database-free solution worth the disadvantages? What decisions have been made by others facing the same task? Comments please!
Thursday, August 02, 2007

How You Can Help

My Javascript went missing in action when I closed down one of my service providers. Now the Oblique Strategies are back, giving you a new approach to each day's problems. And the Amazon recommendations are back as well, new and improved. Which brings me to the topic of this entry.

There are lots of ways you can show you appreciate what I do here and help me continue.

1. Check out the aforementioned Amazon buttons in the sidebar. I've updated them so that now each recommended item is explicitly mentioned, with a link to the article in which it is discussed. Remember that you don't need to buy my recommended DVD or book or CD. Anything you buy after clicking on a link sends a trickle of revenue back to me. I get a buck instead of Amazon; you pay the same either way.

2. You can donate using my PayPal link. Maybe what you'd spend on a coffee. Especially a Starbucks coffee, say a Caramel Machiatto. Yum! (Though I am rather conflicted about Starbucks and in fact am happy there are none here to tempt me.)

3. You can leave a comment to say how much I delight you, annoy you, or get you thinking about something new. That makes me happy. And sometimes angry. But that's what it's all about.

4. You can hire me to write something for money. Then I can buy my own coffee. And food.

5. You can hire Parmar Development to build your next website.

6. If you like broken beats and noisy ambiance you might invite escalation746 to play your next wedding.

In any case I will not put any ads on this site. I just don't think it's appropriate.

Alright, over to you.