Interrupt Service Routine: LMMS
Time for another update. It is depressing how fast the time goes by. By now, I was hoping that my first game would be at least in alpha and I will be getting first feedback from testers, slowly inching towards a release. But as always, the classic formula for how much time software development takes (make an estimate, multiply it by two, and add zero to the end) proved to be true. Some portions of the project turned out to be much more difficult than I expected, and combined with my tendency to procrastinate it eventually lead me to quite an unexpected side-quest.
Content trouble
I should start with what happened after the previous post. I mentioned a new project I was working on: a VR rhythm game that motivated me to refresh my aging Blender skills and allowed me to get some hands-on experience with game development in a big, full-featured game engine. But while building and testing the prototype was great fun, I eventually started running into issues with content. And that's where all previous plans went out the window.
As I explained previously, handling everything from code, through graphics, to SFX, music and other assets, is a pretty hard job for a single developer. I pretty much abandoned the idea of making everything from scratch and hoped to save a great amount of time by using 3rd-party 3D models and music as much as I can.
That worked quite well for 3D models, since most licences don't stop you from customizing the assets. That means I could buy something that's "close enough" to my needs, open it in Blender and fix or improve whatever part I did not like. But with music that did not work at all. The majority of stock music is sold "as is", a single audio file containing the song, or maybe several tracks rendered separately if you are lucky.
Since RRA! is a rhythm game, music is obviously quite an important part of it. It would be nearly impossible to create a good soundtrack from the sort of material that is available in stock libraries. The quality and style varies too much, so without the ability to make changes it would be very hard to make anything that sounds cohesive and consistent. Even if the perfect combination of stock songs did exist, the libraries tend to have way too much "search noise" for my liking, making the whole search process extremely frustrating. Many authors add all the damned tags available to all their songs, so in the end, everything ends up on a single massive pile, with no meaningful way to navigate it.
So no, thanks. Even though I could technically make a half-decent soundtrack this way, I want more control and flexibility. That leaves me with two options: hire somebody to produce the complete soundtrack for me, or do it myself.
Hiring an experienced producer seems like the best and logical choice: they know what they are doing, so the quality would be good and the project could be finished in a reasonable time-frame. But it also comes with a higher up-front cost and a big commitment. I can't just say "Yeah, by the way, it's just a side project I may never finish. I'll maybe pay you in the next decade or so." I have to show up with cash, and I need to decide on the budget now. Do I offer $100 for a two-minute song? Or a $1000? What if I go cheap, the game is a success and the only complaint is "a crap soundtrack"? Or if I go expensive, invest my savings into it, and the game completely flops or I never finish it?
Apart from the budget, there are also "project management" considerations. If I decide to go with a one-off contract, I would need to get the whole story and gameplay ready in advance. That could make it hard to do any substantial changes later in the development, since the producer may not be available anymore to update the soundtrack. At the same time, planning everything in advance would be even harder due to the fact that I would have no soundtrack to use for prototyping. So one-off contract is not a great deal for multiple reasons.
On the other hand, if the composer is a team member working full time on the soundtrack, they could start with a quick draft and then continuously improve it as the project moves forward. But that brings an even higher commitment: I would have to work on the project full time as well. Otherwise it's not much better than the one-off contract: the composer would finish the first iteration, and by the time I make enough progress to provide any meaningful feedback, they could be busy with another project. Unless I pay them for all the idle time they spend waiting for me, which is obviously not very economical.
Perhaps some middle ground exists and a long-term cooperation spanning several years could be arranged. But still, I don't want to drag someone into my project if I'm not sure I can actually keep up and make consistent progress. I also like my current job and the consistent income it provides, so I do not plan to switch to working full-time on my pet projects any time soon. That really only leaves me with the second option: learn how to produce music, git gud, and make the soundtrack myself.
Getting to know LMMS
In the past I tried to experiment with music production a little bit, but I always gave up after a while. Unlike 3D objects displayed in Blender, sound only exists in one moment. Without the constant "big picture" overview, I can't easily start from a rough shape and cut away pieces until it resembles something I imagine in my mind. At least not in the beginning, without the necessary skills acquired by training.
I found 3D modelling much more intuitive and fun as a beginner. Even with a very low skill level, you have a clear feedback all the time. It is easy to see progress, which kept me motivated. On the contrary, I always found music production boring and demotivating. Everything I make as a beginner sounds like crap, and it is not immediately obvious to me what is wrong or missing. And so I quickly lost motivation and never learned enough to break through this entry-level barrier.
But with the new motivation gained from the music-oriented project, I decided to give it another try. I took a more methodical approach this time, and finally I realized that a "full" Digital Audio Workstation (or DAW for short) is much better suited for my needs than trackers, which I experimented with previously. Trackers are pretty cool and minimalistic, but their features are quite limited. So I went with a "real DAW" this time.
Naturally, I looked for a free, open source DAW first. No reason to spend money on proprietary software at this stage; I could upgrade later if needed, after I get some experience and I know what I'm actually looking for. As the title suggests, I went with LMMS. I'm not sure how I missed Ardour, but I think it seemed to me too focused on recorded audio, as opposed to sound synthesis.
My original fascination with 3D graphics and games came from the fact, that you can use a computer to take take something that exits only in your mind, and give it a form. Bring it into existence, out of nothing. I suppose it is also the reason why I got into 3D printing relatively early, when printers still weren't that widely available or easy to use. Or why I like to tinker with electronics, and why I work as an FPGA developer. I just like to play close to the interface between reality and the virtual world.
So, anyway, to get to the point. LMMS seemed like a good fit for what I needed and for what I liked. So I installed the latest version and started to learn and play around. And again, I found music production to be harder than I expected. But as I now expected some difficulties to arise, this time I was determined to not let them stop me. To analyze what exactly am I missing and to keep moving forward step by step.
Well, it kinda worked, and kinda did not. As you could guess from the status update in the beginning, after three years I still can't make a decent song to save my life. But I definitely made some progress, just in a different area than I intended.
Spectrum Analyzer and Vectorscope
Unlike trackers, where you tend to work with pre-recorded or pre-generated samples, LMMS offers a variety of synthesizers: modules that generate sound on the fly, based on several supplied parameters. Synthesizers can be extremely flexible and can generate many kinds of sound, from simple tones, through complex lead instruments with rich textures, to percussion or special effects. As you can guess, that fits my interests nicely, so I started my quest to learn music production by experimenting with sound design.
After grasping the basics, I wanted to see if I can make something that sounds realistic. My thinking was that if I can't take a reference and imitate it closely in a synthesizer, it probably means I'm still missing some knowledge or skill. If that is the case, I would be unaware of any mistakes I made when not trying to imitate a real instrument. So I tried to make a relatively simple "church bell" instrument in ZynAddSubFX (an old, but powerful open source synthesizer). And the result sounded nothing like the reference.
Trying to figure out where did I make mistakes, I quickly ran into a problem: by listening, I could easily tell that there is a problem, but I could not tell where the problem is, or how to fix it. If there only was some way to visualize— oh, wait, there is! I spent quite a few days playing around with arss years ago, I used the waterfall display in quisk to spot interesting signals, and I use the spectrogram in Audacity all the time. I need a spectrum analyzer!
So I go to the list of LMMS plugins, locate the spectrum analyzer plugin, put it into my project and. Oh. What the hell is that? It looked like something I would expect to see on a two decades old Hi-Fi stereo system. Needless to say it did not help me at all. I tried to export the sounds and analyze them in Audacity instead, but doing this after every little change was extremely slow and inefficient.
Eventually I discovered the new native Equalizer plugin, added in version 1.2.0. It has a working spectrum analyzer, who knew! To my amusement, it was much better than the dedicated Spectrum Analyzer plugin. It was definitely an improvement, but it didn't take long before I reached its limits as well. It only showed loud signals, and it was not really helpful for analyzing fast-changing sounds like percussion, or specifically the bell hit I was trying to replicate.
After being stuck on this for some time and not making any progress, my determination started to fail. Procrastination took over. But this time, luckily, it was a productive kind of procrastination. Instead of trying to learn something new, I procrastinated by doing what I already knew: I started poking around the code. First I tried to see if I can simply improve the Equalizer plugin to better suit my needs. After all, I have been using open-source software for the last 15 years or so, and knowing that I can always change or fix anything if I really need to was one of the things I liked about it the most. It was about time I finally started making use this advantage in practice.
While looking through the code, I happened to notice a small error in the formula for a FFT windowing function. That turned into my first pull request. And that in turn led me to finally realize: I don't have to just make small tweaks for myself. Open source projects are built by volunteers: programmers like me, who do it in their own spare time. If I need a good spectrum analyzer to make progress, why don't I make one? So, I did.
It was also my first experience with a PR review process. It took way more time than I expected, but I also learned a lot from more experienced developers, so the time was definitely not wasted. It also made the final result much better than what I would have created if I was just aiming to hack together something that barely fits my needs. So as a bonus, I now also benefit from the higher quality.
And by the way, I have to say that knowing it will be used possibly by millions of people around the world,1 seeing the PR finally merged felt better than getting paid on my job. :)
1 The latest number I heard was around 2 million downloads for the stable 1.2.0 version. But it's hard to say how many of that are re-downloads, and how many more installs came from Linux distro repositories or 3rd party websites.
As I started using the new spectrum analyzer to continue my experiments with sound synthesis, I quickly discovered new limitations. So I did not stop there and I soon made a batch of upgrades, mostly aimed at adding advanced settings, and improving temporal resolution to help with analyzing fast transitions. That again took a long time to review, but again it felt very worth-while in the end.
Now that I started procrastinating, it was hard to stop. So I kept going. It was around the time when I started experimenting with the stereo image, and inspired by a YouTube video about "oscilloscope music", I figured I probably need a vectorscope plugin. So I made one.
It also took relatively long time to review, but the code wasn't nearly as complex compared to the spectrum analyzer, so it wasn't as much work as before.
At any rate, by the time it was merged, I spent about a year on developing plugins for LMMS. I learned a lot along the way, but I felt I'm still nowhere close to making an actual song. One that someone would listen to even without good headphones and a large amount of duct tape. So I re-evaluated what I know and what I do not, and I figured out it is probably a good time to learn some music theory.
Microtonality and xenharmonic scales
As you can probably guess from the title, that's where it went wrong again.
At first, I found music theory incredibly confusing. I did not want to memorize a list of chords, list of scales, or some arbitrary rules that revolve around those lists and so on. I wanted to understand music theory, learn how it works and why are things the way they are. But in most material I encountered, these inner workings were carefully hidden behind layers and layers of misleading terms and historical cruft.
Take the simple concept of a third, or a perfect fifth. A third of what? Nobody bothers to tell you the name is completely unrelated to its associated interval (like 5/4 or 6/5), and that it is in fact a mutilated adjective used as a noun. It is just assumed you know it means "the third piano key, counting from the first key of the scale, but assuming you skip the keys that are not in the key you are using, and by the way yes, key and key are two completely different things".
That being said, once I figured out what really is an "interval", and how they (and chords) map to frequencies, it all became much clearer. This video (along with the previous episodes leading to it) is probably the one that helped me the most to tear through the fossilized layer of confusing terminology. But the sudden increase in understanding was also a gateway drug leading to the world of Just Intonation, microtonality and xenharmonic music.
Not all of the stories I am working on are based in western culture and history. So I thought: why on Earth would I use western tuning and western scales to compose a song that's supposed to represent a civilization with completely unrelated culture? Sure: they would come to the same conclusions regarding some basic rules, but a large part of music theory is made of cultural preferences and traditions. Just compare western music with traditional Arabic or Japanese music as an example. And now imagine the civilization is separated not only by geography, but also by thousands of years in time.
Perhaps I worry too much about pointless details that nobody would ever notice, but I like my stories to be self-consistent, and researching these sort of details is actually one of the things I enjoy about world-making the most. You know where this is leading. In order to continue with my quest to learn music production, I needed LMMS to have support for xenharmonic scales. So I made it.
This time not only I added a lot of new code, but I dared to start fiddling around with things close to the LMMS core. So, naturally, the PR review took even longer than all the others before. In fact, it has now been around year and a half, and as of writing this article, the PR is still waiting for the final stage of the review. The main issue this time is simply the scarcity of experienced developers who are able to review it and have the time to do so.
Developers come and go, and not everyone is able or willing to review pull requests. This leads to accumulation of PRs over time, and many of them stay incomplete or forgotten as their authors give up and move on. I imagine the growing pile of work may further discourage other developers from investing their free time to reviews. The work just never ends. This creates a feedback loop that could slowly erode the motivation of everyone involved. But what's the solution? Light the match and set fire to the mountain of unfinished, yet potentially useful contributions? I do not know..
People who review pull requests really deserve as much gratitude as the authors themselves. Reviewers not only ensure the contribution meets quality standards, provide feedback and suggest improvements, they do so at the expense of their own free time, which they would possibly like to spend on programming their own features instead. So thanks again to everyone who donated their time to the review process on my PRs! And to any other PRs for that matter.
So, what's the plan
At this point, I think LMMS has most of what I need, so it is time I stopped procrastinating and went back to learning. There are still some new features I wish LMMS had, and some improvements I could make to my previous contributions, but I would like to start focusing again on music production. If I hope to improve in music production, I have to start producing music, first of all. Or at least try. And fail. And learn from that.
That does not mean I'm completely stopping all work on LMMS; I will still maintain the code I contributed so far, and perhaps make some new small contributions here and there. But I am now consciously avoiding starting any new complex projects, since I know they would probably take much more time than I'm currently willing to "donate". LMMS development is fun, but with the recent PR backlog slowdowns it is not as fun as before, and I really need to start putting more work into the CC series if I hope to finish it within my lifetime.