Raspberry Pi (2 or 3) MIDI Synthesizer Approach

[JovianPyx]

This thread is about helping others who might want to make a MIDI synthesizer using a Raspberry Pi 2 or 3. For this thread, when I write "Raspberry Pi", I mean a "model" 2 or 3.

My research helped me conclude that a Linux environment that supports ALSA might be a good way to start. ALSA provides many different ways to talk to the sound card driver. I chose a method that allows me to write directly into the DMA RAM that the sound card will use (and I think this is true regardless of the hardware if the drivers are respectful).

I used the asound library system for my work. The programs are developed and compiled on the target Raspberry Pi (project compiles take only 2 or 3 seconds) using gcc. In this environment, the NEON FPU is enabled and compiles in.

The goal for me has always been to keep MIDI to sound latency as low as possible. I've measured results in the range of 1 to 2 milliseconds.

I started by visiting the ALSA project website where I found example C files.

I downloaded those files and found them to be too old to simply compile right off, so I had to screw around finding out why. I eventually got one demo file to work - a program that would play a sinewave at I think one kilohertz. This was the starting point and is the basis for all of my current synth code.

I would love to just post the C file here, but I know squat about copyright and the code I started with is a modification of the code I downloaded from the ALSA project.

The thing is - this one particular file is the key to the rest of what I call my "synthesizer framework". It has evolved into a set of several synths which include effects. It is also supported by other C files which contain mostly immutable ALSA function code.

Please understand that my approach here is that it has to work on my computer and that YMMV.

[JovianPyx]

For my synth projects, ALSA was a critical decision. I looked at a bare metal approach first because I'd been accustomed to working with FPGAs and dsPIC devices. Without an operating system, those devices allowed me to totally control what happens, cycle by cycle. However, the ARM chip on the Raspberry Pi runs much faster than a dsPIC and also outperforms the somewhat old FPGAs I have. Bare metal on a Raspberry Pi would mean either finding or coding my own bootstrap system to get whatever synth program up and running. Booting the Rpi is a process that involves writing specialized code for the board's GPU. It would also mean working with the RAM's hardware that controls CPU RAM access in coordination with DMA capable devices. I realized I would need expert level understanding of the more primitive aspects of the very complex ARM CPU, GPU and RAM. I finally decided that it was worthwhile investigating what it would take to access the sound board I have with Linux as a lightweight operating system. For this, ALSA seemed the best choice since it has rather direct access to the sound board hardware. ALSA can set the board to do any of it's sample rates, any of it's data types and provides a connection to a function that is called whenever the system needs more data for the DAC.

To learn ALSA, I started here: http://www.alsa-project.org/main/forums.html/Main_Page

I rummaged around there and eventually found example files written in plain old C which was nice because I've never liked C++ nor any object oriented programming language. I'm fairly certain C++ could be used though.

Among those files, the most interesting to me was a program that outputs a sinewave. Of course, when I tried to compile, there were some errors that needed to be resolved. Some were because gcc had changed a little and required adding a header file or two, but Google helped me find what I needed and eventually the program compiled. This is when I started to learn about "buffers" and "periods", two terms with which I had not been familiar. Buffer is the larger amount of RAM and is meant to prevent glitches in playback due to variable CPU demand. Period is a small number of samples and is the smallest unit of time (or RAM) that can be processed by ALSA. Whenever the program needs more data for the DAC, it will request another period of samples. The way I have ALSA set up gives me an 8 sample period which is the smallest I could configure. It amounts to approx. 181.4 microseconds. The program allowed me to set the board's sample rate, period time and buffer time and I could experiment with the effects of these settings. Buffer was interesting because while a larger buffer allowed glitch free playing through times of heavy CPU use, it also added latency. My use of the Rpi as a synthesizer meant (to me) that as little CPU activity should be present, so I eliminated any services and other processes that weren't absolutely necessary. I also experimented with different transfer methods. I settled on writing directly to the RAM allocated for the sound board's DMA. There are some 7 or so transfer methods, some more efficient than others and some "prettier" than others. In my view, the direct write transfer method wasted the fewest CPU cycles.

Once I was satisfied with how the sinewave program worked, I began work on doing other things with it and building the framework I have today. The transfer method function I use now is a modified version of the one I downloaded as an example file from the ALSA project website.

[JovianPyx]

MIDI CONTROLLER and VOICE ASSIGNER

Part of this is rather weird, but it works. Why I did it is a long enough story that I'll just say I had numerous problems with the buffered serial driver that Linux uses. This resulted in me disabling the UART interrupt so that the Linux driver will never do anything for the hardware UART. I had already managed to get the UART to run at 31.250 kbaud. Since the driver is disabled, I had to write my own. Essentially a loop that polls the UART 'data ready' bit, grabbing the received byte, process the byte and go back to the polling loop. Access to the UART device registers is done with mmap (stands for memory map). mmap allows you to store and retrieve information into the device registers as if they were memory locations.

This actually resulted in significant improvement in MIDI note on received and sound happening. I run the MIDI controller in it's own core. The voice assigner does exactly what the name implies, an application dependent algorithm is used to determine which voice plays which note and implement voice theft when no voices are available.

This is a multi threaded application. Communication between threads is accomplished with global memory definitions. The MIDI controller talks to a synth module which is running in a different thread (and core) by depositing required values computed from MIDI data received into the locations where the synth.c code will look for it.

[JovianPyx]

The MIDI synthesizers I've designed all use the same basic structure. In order to get the best performance from the Rpi, I needed to use more than one CPU core. This is accomplished by doing two things: 1) Create separate threads that will allow parallel processing and 2) assign each thread to it's own CPU core.

Reading about Linux threads showed me that the actual implementation of launching separate threads was not difficult. It involves creating a thread "handle" and calling a function. Assignment of the thread to a core is another simple function call. All of this information can be found on the web using a search engine. The assignment of threads to cores can be enhanced by using core isolation. This amounts to a single parameter on the startup line in /boot/cmdline.txt. The parameter is "isolcpus=n,m,p" where n, m and p are numbers from 0 to 3 specifying the cores to reserve. 1, 2 or 3 cores can be isolated this way. An isolated core is said to be a core that the kernel will not assign work to. I have read, but not confirmed that this isn't precisely true (though I've not found proof in documentation, this comes only from people posting in fora), that in fact the Linux kernel can or does assign some kind of work to all the cores whether isolated or not. However, if I watch the top command display, I never say anything but zeroes associated with the isolated cores. If what I read is actually true, then the amount of work assigned to the isolated cores is so small that it is insignificant. Linux can easily run on a single core on the Rpi2 or Rpi3.

In my designs, 3 cores are isolated. One core runs the MIDI controller, voice assigner and polling serial driver. A second core runs the actual synthesizer process and the process that feeds the synth's data into the DAC DMA RAM. The third core runs an audio special effect which so far has been a rudimentary but functional reverb. Synchronization is implemented with simple software flags. Communication between the cores is provided by global memory which is accessible by all of the threads started by the main program.

If you are interested in this, reading about pthread for Linux is a good thing to do.

[JovianPyx]

The structure of ALSA for audio playback is rather simple. It's root is in what is called the transfer method function. In my designs, I decided to us the mmap direct write transfer method. This means that the transfer method function will write data directly into the DMA RAM locations for the DAC and the kernel driver uses whatever is put there.

The transfer method function is essentially an infinite loop. Inside the loop, a sample generation function is called as long as it's pointer into the DAC's RAM is behind the kernel's stop pointer. Errors generated here will cause either a recovery call (for a retry) or if the error is bad enough, the loop will exit. Things that can cause problems here include a sample generation function (your function) that takes too long. I've been able to discover this limit for each synth by experimentation mainly. Note that I've found that if it's way out, it can cause the audio system to lock up and require a reboot. In most of my designs, I've been able to achieve 32 voice polyphony.

Your sample generation function can do whatever DSP you can think of and must supply exactly one period of samples. Within my sample generation function, a function called 'synth()' is called for each sample of the period currently being constructed. This makes it easier to think in terms of per sample synthesis. Latency can accumulate here, which is why it is good to keep the period size as small as possible. I try to keep it at 8 or fewer samples per period, though I've not been able to get it below 5. The sample generator function is passed a pointer into "areas" (a RAM section which holds the sample values to place into DMA RAM) and the period size (which is measured in frames - a frame is either a stereo set of 2 side-by-side samples or one mono sample, but ultimately it represents one sample time). Your synth() function should operate as if it were to create a constant stream of samples, even if they are silent.

The transfer method function I use came from the PCM sinewave demo program at the ALSA project site and is unmodified except for a change to the name of the sample generation function. The sample generation function I use is a modified version of the one from the same project.

[JovianPyx]

TOOLS
I do the development for my synths directly on the Raspberry Pi. The synth programs usually take around 2 seconds to compile and link so the development cycle of edit-compile-test is very fast. My main tool (besides my editor of joice) is gcc. However, there can be problems. One problem that I recently encountered was in trying to use ASAN (stands for address sanitizer). ASAN is a library that can help find problems with array indexes going out of bounds. gcc can compile in a bare-bones mode where all checks for "bad" things a programmer can do are turned off. ASAN is needed to have the target program check itself for an array index that goes wandering off into never-never-land. I usually leave such checks turned off because I want my code to run as fast as possible. My intention with ASAN was to discover the problem, fix it and then turn it off. ASAN, however, in the Rpi world (at the time of this writing) appears to have some problems. Arch Linux doesn't even supply the package, so apparently those folks knew about it and didn't include it. It is available, however, for Raspbian, but it isn't much good in that with reasonably complex code (like a MIDI synth), it declares bugs in programs where they don't really exist and can cause a programmer's wild goose chase.

I've not tried it, but it may be possible to get it to work by doing cross-compile on another system like a PC or a Mac.

[ian-s]

[JovianPyx]

[JovianPyx]

gcc COMPILER

When working with synthesizer code, I've been using gcc as a C compiler. One of the issues I have with the version of gcc available for the Rpi is that it does not by default check for out of bounds access to an array. That is, if an array is declared like:

[gdavis]

This isn't so much a gcc issue, it's standard ansi C behavior. Arrays aren't much more than pointers in C. When you use array[x], this is essentially the same as *(array + x). Variables declared as pointers or arrays can be accessed with both array index notation or pointer notation interchangebly.

Anyway, there are two things that need to be taken into careful consideration when using arrays in C.

1. This is a general code style thing but it's referred to as "magic numbers". You should never use a hard coded constants (except maybe 0 or 1) in your code. All other constants should be #defines (with comment) to make it clear what the number means.

This then leads to:

2. Boundary conditions. Your code needs to be rigorous about boundaries. Good programming instructors usually pound this into you early. So before accessing an array, you should always check that you're within the array boundaries. In C the lower boundary is always 0. The upper boundary is the array size you defined - 1. So for C it's very common to say:

[JovianPyx]

Oh of course that can be done, but it gets complex and weighty with all of that and it becomes a computational load I'd prefer to avoid in a realtime application.

I'd prefer to use ASAN during troubleshoot, get it right and then remove ASAN for the final product.

[gdavis]

Everyone would prefer to avoid it, but I hope you're starting to see why it's important

[blue hell]

For really time critical code I will make the boundary checking controlled by #ifdef __DEBUG__ or something .. so that its checked in debug compiles and not for the release.

[JovianPyx]

yes, but it still makes messy code

but I may have to do that...

[JovianPyx]

[blue hell]

You could all wrap it up in a couple of macros .. like array_def( name, size) and bounds_check( name) - you'll need the pasting operator (forgot what it was in gcc, ## in some c dialects) then to create multiple names from one .. but it will look pretty clean that way.

[JovianPyx]

OH! THANK YOU. Totally forgot about macros. Yes, that is the solution. And it would then be easily removed after the code works properly.

[gdavis]

Microsoft is the queen of non-standard "features". They have to get there paws into everything, often making things worse. MS Explorer is a big part of why the web is such a mess.

And after lecturing me about your extensive C programming experience, you totally forgot about macros? No wonder your code is hard to read

But seriously, please don't take offense, I'm just giving some light hearted coder razzing.

I do feel that in general this type of checking is important to do though, even if it does make the code a little heavier and harder to read. Just as a side note, many security issues that exist in software are due to exactly this kind of problem, though obviously a synth isn't likely to pose a security problem.

If done right, it shouldn't be that bad. I understand the need to optimize critical code, but that needs to be considered carefully.

Tools like ASAN are interesting, but I can't help feeling it's more of a crutch for bad programming. Even with it, you can still have issues you don't catch during testing, and your program doesn't handle the issue gracefully.

BTW, would you mind sharing what you do for a living? Just curious, you seem pretty knowledgeable about DSP. Your website has been quite helpful to me.

[JovianPyx]

well, I agree with what you're saying and surely the visual crap is all fat and tard-like.

I learned C when there was no Windows, like DOS 3.something and it was pretty clean and nice and on a 80286 12 MHz system and was able to make a functional video game (space war). It was a hybrid program written mostly in C with a smattering of MASM where needed (the video "driver").

[JovianPyx]

As for a living, I am a retired programmer. DOS batch and C early on, automated software distribution, and later some business programming to automate clerical tasks in VB (eek).

I learned DSP because I really wanted to write my own synth stuff. It actually started with DSP on FPGAs which I learned the hard way (not school), rather reading sites like www.dspguide.com. I learned Verilog by doing a web tutorial and trying the examples on a Spartan-3E Starter Kit board. I then moved to DSP in assembly on a dsPIC and finally moved to a Raspberry Pi working with ALSA under Linux.

[JovianPyx]

This is a program I've written to demonstrate the principles that gdavis and blue hell suggested in the form of a macro:

[hrastprogrammer]

As the subject contains "Raspberry Pi synthesizer" I would like to mention my own approach here ...

I have a bunch of my own 32- and 64-bit synthesizers developed on Windows as VST 2.x plugins, Tranzistow being the most powerfull of all them:

http://www.hrastprogrammer.com/hrastwerk/tranzistow.htm
http://www.hrastprogrammer.com/hrastwerk/hrastbook.htm

They were initially developed in Delphi / Object Pascal (user interface and high-level functions/modules) in combination with x86/x64/SSE assembler (the complete audio engine and all low-level functions). I used built-in Delphi assembler because it works like a breeze.

Then I ported all of them to Free Pascal / Lazarus, first on Windows and later on Linux (both 32- and 64-bit versions) in form of standalone synthesizer applications and experimental VST 2.x plugins. Standalone synthesizers can work with ALSA, Jack and PortAudio / PortMIDI. Experimental plugins are still, well, experimental.

Then I thought about porting some of those standalone synthesizers to Raspberry Pi 3 and decided to start with Operator FM synthesizer because it doesn't contain a lot of assembler code and isn't nearly as demanding performace-wise compared to Tranzistow, for example. I ported ALSA layer only and, beside some x86=>ARM assembly conversions, no other code changes from Linux/PC version were needed.

And it really works like a charm with my IQAudio Pi-DAC+ sound card and E-MU Xboard49 USB MIDI keyboard ... CPU usage is less than 50% of a single core at full 16-voice polyphony, 44.1kHz sample rate and around 5ms latency

Unfortunately, I am sooooooooo busy on my regular job these days/weeks/months - game development with tons of (paid, fortunately) overtime hours - that I simply don't have time to make a free public version of this synthesizer ... I suppose I will have to wait for things to settle down to normal hours again

[JovianPyx]

Nice stuff HrastProgrammer - Seems to me you should start another thread describing how someone could approach the writing of VST programs. I know I would be interested. I would have no idea where to start, what tools I would need and some of the general principles involved.

[hrastprogrammer]

Yes, it could be interesting but the lack of free time is my biggest enemy at the moment

[JovianPyx]

Well, please keep it in mind for later when you find one of those round tuits because it looks to me that you know what you're doing...

I know I would like to try writing a VST some day.