Reverse engineering USB protocol, NM1 to G2 converter

qfingers

Hello,

I just bought a brand spanking new G2X from www.novamusik.com for $2100 with a case. I'm really happy with the box though some of threads around here are dis-heartening about the lack of updates for the G2. As the unit stands, it is a very solid machine. I like it alot. I also run a Roland Fantom-X7, and a Korg Triton Pro-X. I was tired of the lack of "analog" features of samplers. Plus most of the sound design for samplers came down to using the Sine,Square,Triangle,Sawtooth waves anyways. Why not get a analog or modular system? Hence I got the G2X.

That said, I just wanted to let everyone here know I'm working on reverse engineering the USB protocol and creating a tool for convertion of NM1 to G2 patches. I'm an embedded linux developer for my company. I'm really a linux geek. This is main development platform. I actually have the Modular G2 Demo running under wine and it works great under linux. The problem is wine doesn't support running drivers within it (yet?). I also do alot of python (a scripting language) programming for prototyping, testing, and developing the systems we build at work. So, I have a python module called pyusb that allows me to script communication to a USB device (i.e. the G2). I also downloaded and installed a USB protocol capture utiliy called SnoopyPro that allows me to view the USB message passing from application to device. I've got alot already captured and have verified the communication with my python script to make sure the messages work. I am able to switch variations, select knob pages, activate slots, and assign keyboard, and set the focus all through the USB connection using python and pyusb only.

The other project I'm working on is a NM1 to G2 converter. I'm also writing this in python. My goal is to create an application that can be modified to dynamically create the convertion of NM1 modules to multiple G2 modules when necesary. I want it expandable to cover areas where the need to build a group of modules to match the NM1 module, it can be adjusted accordingly. Right now I'm working on reading in G2 patch and performance files. I started with the exellent web page: http://www.dewb.org/g2/pch2format.html that helped me get started but had some flaws in the format. It was missing most of the module categories so there is work to be done there. Next, I need to create a NM1 patch parsing utility using the NM1 documentation provided by BlueHell.

Mind you, most of my development is under linux as I don't develop software for Windows machines. I do the USB protocol capture though. The python applications will be portable and will run under Windows and Mac OSX. I can create stand alone executables that will run under Windows. So whatever I come up with, will be usable on all platforms. I will attempt to get that to work.

Sorry for being so long winded but I wanted you to know that I'm gonna try my best to get this stuff done. It's going to take awhile, especially because I have several other projects that require my attention that will take precedence for the next couple of months. I will still be hacking away though from time to time. Anyone who wants to help with a knowledge of python, USB, libusb, pyusb, G2 internals, NM1 internals is more then welcome to help. I can give you my scripts and USB captures. I'm trying hard to document my tests and results.

Let me know and thanks for listening.

qfingers

Kassen · Posted: Thu Nov 30, 2006 6:05 pm Post subject:

Sounds great! Keep us posted!

glitched · Joined: Mar 25, 2006 Posts: 80 Location: phila., pa USA

Kudos to you!

This is great news considering that Clavia has all but abandoned the G2. (Why can't more Swedish electronic instrument companies be like Elektron? They are constantly developing and bug-fixing.)

I'll be watching this thread, as many others will, I'm sure.

-d

jksuperstar · Posted: Thu Nov 30, 2006 8:50 pm Post subject:

There's been some amount of work done already in translating the NM1 to the G2 modules set. Some links for this work are here:

http://www.dewb.org/g2/modules.html
http://electro-music.com/forum/viewtopic.php?highlight=nm1++module+translate&t=3159

and some work done on the patch format:
http://www.dewb.org/g2/pch2format.html

qfingers · Posted: Thu Nov 30, 2006 8:59 pm Post subject:

dasz · Posted: Thu Nov 30, 2006 9:13 pm Post subject:

qfingers, this is quite an undertaking and I wish you all the best in this project.

Once finished, your code could help in other areas where g2 has limitations - like keeping 2 g2's patches synchronized (so all the patches from one are sent to another) or being able to send sub patches to an existing patch (something along the lines of Init1 and Init2 but inserting modules into existing patches)

/Dasz

qfingers · Posted: Thu Nov 30, 2006 9:27 pm Post subject:

dasz · Posted: Thu Nov 30, 2006 9:36 pm Post subject:

qfingers · Posted: Thu Nov 30, 2006 9:58 pm Post subject:

dasz · Posted: Thu Nov 30, 2006 10:24 pm Post subject:

I've always preferred COM/dll's - you can do this such as callbacks, etc ... but I've always done windows devt. I hope your app will work on windows and a mac?

qfingers · Posted: Thu Nov 30, 2006 11:06 pm Post subject:

jksuperstar · Posted: Fri Dec 01, 2006 12:14 am Post subject:

G2 editor on Linux, Yahoo!! Single editor for NM1 and G2, Yahoo!!! Macros, here we come!! YahooO!!

Oh, one can dream, right? Tools like these certainly open ALOT of doors. GREAT effort, and I'd offer any help I can that's useful. I'm not a linux guru, but do expect to fully switch over in the next year, and I've put together my own live CD (hacked from others) and continue this effort.

However, I know no python, but this seems like a good reason to learn it. I know Python is powerful, but i've just not had a reason to use it yet.

blue hell · Posted: Fri Dec 01, 2006 4:22 am Post subject:

Don't have time now, but also see : http://www.iaf.nl/Users/BlueHell/html/nm-open/nm-open-g2info.htm

Christian Schneider who made the Nomad NM Classic thingy will be interested as well I guess. Please do a seach for that on this forum.

Later,

qfingers · Posted: Fri Dec 01, 2006 7:01 am Post subject:

blue hell · Posted: Fri Dec 01, 2006 2:38 pm Post subject:

Ok have time now, welcome

on the forom qfingers.

On actualy reading the thread I now see most things to be said already have been.

Yes an exact description of parameter values and their effects would be needed as well in the future. Guess I just didn't have such high ambitions in mind when I compilated the parameter stuff, but maybe later, this project would be a good excuse to put some more time in it Very Happy

Maybe a good warming up project for you would be http://electro-music.com/forum/topic-14727.html , but whatever you choose to do or not do, please keep us posted !

qfingers · Posted: Fri Dec 01, 2006 2:54 pm Post subject:

blue hell · Posted: Fri Dec 01, 2006 2:59 pm Post subject:

Afro88 · Posted: Fri Dec 01, 2006 6:44 pm Post subject:

Keep up the good work qfingers!

I like the idea of creating a midi driver from the usb interface, eliminating the need for 2 midi cables and a midi interface. Also, I would imagine it would be more stable during periods of high cc activity too.

A pch to pch2 converter... imo that's a bit of a pipe dream, but if you get it to work it would be a fantastic and extremely valuable acheivement - thousands of patches will be opened up for use instantly!

And the knob expander app would be killer too!

qfingers · Posted: Fri Dec 01, 2006 9:29 pm Post subject:

cappy2112 · Posted: Sun Dec 03, 2006 11:10 pm Post subject:

3phase · Posted: Mon Dec 04, 2006 12:17 am Post subject:

dasz · Posted: Mon Dec 04, 2006 9:26 am Post subject:

qfingers · Posted: Tue Dec 05, 2006 7:12 pm Post subject: Update

Just a quick update. I've almost got the whole .pch2 file format (minus a few unknown variables) decoded. All the sections but one are deciphered. The section with the 0x65 character header is the beginning of the Morph Parameters section. The sections with the 0x5b character header are the Morph Labels and Parameter Labels where you can change the labels of the 8 Morph dials on the editor. The .prf2 file is just a Performance section with 4 .pch2 files appended at the end of it. The Performance section is also almost decoded. I'm going to run my parser application against all the patches available from this forum to correct anymistakes and hopefully decode the missing variables and the last section (0x69 character).

I wrote a .pch (NM1 patch) parser that I've run against 27600 patches from the nm1 patch archive I got from somewhere with about 140 patches that it cannot parse. Most of them seem to have missing pieces at the end of the file. Plus it doesn't handle patch formats with versions < 3 which accounts for 92 of the 139. I think that's good enough. I may be able to parse the 2.10 but I'm not sure it's worth it for 92 files.

So I'm attaching the python script that does the parsing. Check it out. I coded this in several hours and spend a little time getting the thing to parse mis-formatted files. This will be used as the parser for the NM1 to G2 convertion. I also have a G2 parser that's almost ready. I just want to finish the issues described above.

You do have to download and install python. This was coded under python version 2.4. I'm not sure if it will work with earlier versions. It probably will work with 2.3 though. This is just a parser. The comments at the end of the file are the layout of the objects after the NM1File object is created from the file. I had to change the extension to .py.txt because the forum doesn't allow .py extensions (why????).

I'm going to parse BlueHell's patch303.txt and s303.txt that describe the module parameters and I'll have everything I need to get started on the convertion. I ready have parser for his module_table.txt and param_table.txt files which are the G2 module and parameter tables. If you want I can post that code as well as the parser for G2 files. Just reply to this message or send me a private message.

back to coding,
q

parsenm1.py.txt

Description:

Download (listen)

Filename:

parsenm1.py.txt

Filesize:

9.26 KB

Downloaded:

323 Time(s)

blue hell · Posted: Tue Dec 05, 2006 7:22 pm Post subject:

...

cappy2112 · Posted: Tue Dec 05, 2006 8:06 pm Post subject: Re: Update