Reverse engineering USB protocol, NM1 to G2 converter

Kassen · Posted: Tue Dec 05, 2006 8:36 pm Post subject:

He he he... but no.

The boards only allows files from a speciffic set of extentions and it seems that .py isn't on this.

cappy2112 · Posted: Tue Dec 05, 2006 10:33 pm Post subject:

[quote="dasz"]

tombola · Posted: Wed Dec 06, 2006 7:09 am Post subject:

Thanks qfingers. I'm not a programmer, so I've only understood a tiny bit of this thread, but if there's a possibility of 27,000 new patches and harmony within the NM spheres, then it's a great thing!

mosc · Posted: Wed Dec 06, 2006 10:08 am Post subject:

qfingers · Posted: Wed Dec 06, 2006 12:33 pm Post subject:

mosc: Thanks for updating the web site.

For you programmers/kind-of-know-programmers, the python web site http://www.python.org has a documentation section with a tutorial that takes a couple of hours to complete. The language is simple to learn (if you have any programming knowledge). The author actually has funding for research in using it to teach average people to program to better use there computer. I'm a seasoned programmer and like the language because of it's simplicity. If was the first language I used where I could understand many of the extension module it is shipped with. Most of the applications I write are written in python or as a prototype in python.

That script I posted takes about 15 minutes to run across all 27000 patches. It doesn't do much except set up the structure of the patch to allow further manipulation with other python code.

Here is a list of things done:
- .pch file parsing
- G2 Module database (module types, input/output ports, parameters, modes)
- .pch2 file parsing (about 95%)

Here is a list of things needed to be done for the NM1 to G2 converter:
- finish .pch2 file parsing (needed to make sure the writing .pch2 files and validating the output)
- Module database for NM1
- .pch2 write support (this is the hard one to get right).
- the main converter program with hooks to handle each NM1 module and cabling

The first version will be pretty simple, just convert known modules and connect to known input/output ports, set known parameters and modes. I will try and keep the vertical spacing between modules the same but not necessarily the actual vertical location. I may colorize stuff as well but I'm not sure yet. The module names will be kept in tact as well.

The next version will take into account the differing frequencies and times of various NM1 modules (mostly Osc frequencies, Env times, Lfo frequencies) into account and set the G2 modules parameters accordingly. I'm not sure about the modulation values, but those will have to be processed as well (of course).

The next version will include building block of modules to emulate the NM1's modules. Hopefully by this time the program will have a simple setup to allow non-programmers to fine tune the output of the converter. Things like creating macro blocks of G2 modules and defining what connections to use from the NM1 patch.

Versions after that will be more complex as in searching the cables and optimizing the G2's patch. This may not actually happen or be much useful. It would depend on how successful the previous 2 versions I described above work.

I think it will probably be about a month before the first version is done (if I can put time into it). We will see.

q

blue hell · Posted: Wed Dec 06, 2006 1:42 pm Post subject:

qfingers · Posted: Wed Dec 06, 2006 4:32 pm Post subject:

3phase · Posted: Wed Dec 06, 2006 11:21 pm Post subject: Re: Update

blue hell · Posted: Thu Dec 07, 2006 12:46 pm Post subject:

I split off the object oriented programming / python discussion to : http://electro-music.com/forum/topic-14839.html - as was suggested by some.

qfingers · Posted: Thu Dec 07, 2006 2:26 pm Post subject:

qfingers · Posted: Mon Dec 11, 2006 9:05 pm Post subject: Update

Hey, I just wanted everyone to know, I'm almost finished with my g2 .pch2 file parser/writer. This is the hardest piece of my NM1 to G2 converter because it's the biggest unknown. I've spent the entire weekend on disecting the file format. The problem is there are so many different versions of .pch2 files. Most parse ok, but when I try to load them into the editor, it displays an error message (mostly worthless in my estimation) or crashes the editor. I basically took the two G2 patch collections on the clavia site and read/wrote each file and tried to load it into the editor. Most of them work, I'm just fixing the code to handle the ones that done. Apparently, they have changed some of the modules inputs/outputs/parameters between versions. Once I have that done, I'm going to start the converter.

Actually, I think this is the easy part (at least the initial version). I'm going to create a set of classes with two methods: one that handles module types/parameters, and the other that handles the cable connections. This way I can build the missing NM1 modules into G2 modules and then setup the proper connections. If we have correct G2 models for the missing NM1 modules, it should be a piece of cake. Right now I won't create .pch2 files from NM1 files with missing modules. The last thing I'm going to do is a simple optimization removing unused modules or muted ones with no knobs assigned to them. One thing I'm not going to do yet is the envelope times conversion or the lfo frequency conversion, it will just map 0-127 to 0-127 (one to one). I'll correct that afterwards. Right now I just want to get a patch file that is loadable without error.

I post when I'm further along.

q

P.S. Is it ok to post .zip files? There will be a group of python modules necessary to run the converter.

cappy2112 · Posted: Tue Dec 12, 2006 12:37 am Post subject: Re: Update

Afro88 · Posted: Tue Dec 12, 2006 1:15 am Post subject:

Yeah! Nice work qfingers!

gomidas · Posted: Tue Dec 12, 2006 2:11 am Post subject:

Very good initiative, great work Very Happy

blue hell · Posted: Tue Dec 12, 2006 5:40 am Post subject: Re: Update

Good work !

ian-s · Posted: Tue Dec 12, 2006 11:10 am Post subject:

Yes great work. Although I personally would have no use for a G1>G2 converter. DX7>G2 is another matter.

cappy2112 · Posted: Tue Dec 12, 2006 11:29 am Post subject:

qfingers · Posted: Tue Dec 12, 2006 11:36 am Post subject:

cappy2112 · Posted: Tue Dec 12, 2006 11:43 am Post subject:

qfingers · Posted: Tue Dec 12, 2006 11:48 am Post subject:

cappy2112 · Posted: Tue Dec 12, 2006 12:11 pm Post subject:

[quote="qfingers"]

ian-s · Posted: Tue Dec 12, 2006 2:19 pm Post subject:

My fault, I meant that when I get the time, I can write my own DX7>G2 converter, thanks to all the hardwork done by yourself (and others) on the patch file format.

Wish you the best of luck with your G1>G2 project and think it will help Clavia a lot if you are even partially sucessfull.

qfingers · Posted: Tue Dec 12, 2006 2:49 pm Post subject:

What I would really like is for Clavia to port there editor to linux (or have me do it if they send me the code). I'm not a Windows guy. My hope is if they see this maybe they could do it or ask. Otherwise I have to revert to more difficult measures as in hacking wine ("Wine Is Not an Emulator" http://www.winehq.org) with enough USB driver support to run the editor if that is even possible.

Plus, I would like to check out 27000 (or some subset) of patches from the NM1. I think that would be pretty cool. I would keep me occupied for at least a month.

Not to mention having my name slapped on every converted patch.

q

3phase · Posted: Wed Dec 13, 2006 2:15 pm Post subject:

qfingers · Posted: Wed Dec 13, 2006 4:29 pm Post subject: