Foot Stompin' Music

[Inventor]

For various reasons I've been diagnosed with CHF, which is Congestive Heart Failure. It's serious but treatable and can be minimized with very mild exercise. One of the side effects of CHF is that your feet swell up like balloons because your heart isn't pumping properly and the blood pools at the lowest point. Time for a music hobby solution.

I just happen to have not one but two USB accelerometers that are experimental prototypes from Andy at Ultimarc in England. These little babies sense acceleration in X, Y, and Z directions and send it into ChucK as joystick information.

What I'd like to do is attach them to the top of a pair of shoes and make live music by wiggling my feet around. This would solve the exercise problem for the feet at least, and give me something to do with the accelerometers. Trouble is I don't really know what to control or what to do with them.

Ideas and suggestions, please!

[Inventor]

Here is a photo of the two accelerometers. The one on the bottom is the first prototype so it has two boards connected by a wiring harness wrapped in electrical tape. The other one is the factory prototype which is basically the same thing that customers will get. I guess I could just wrap them in static bags and mount them to the sneaker laces, I dunno.

I was thinking of doing a one-man-band thing, controlling the instruments with feet and kbd/mouse, but perhaps there is another way. My most recent and dare I say greatest success with ChucK so far is the KNoW music generator. Perhaps I can mix things up with that somehow?

---

Accelerometers.jpg
Description:	Photo of two accelerometers
Filesize:	268.82 KB
Viewed:	172 Time(s)
This image has been reduced to fit the page. Click on it to enlarge.

---

[Kassen]

Let me first say how happy I am; When I first read about your diagnosis and it having to do with the heart I got scared... The heart is pretty essential. It's good to hear this particular heart disease can be treated, that it can be minimised with exercise and that you have this outlook on the exercise. Maybe it's weird to say "I'm happy" (maybe "relieved is a better word) but I feared things could be far worse when you originally mentioned your ailment. I think, BTW, that with any sort of ailment your attitude towards the situation matters a lot and yours here sounds great to me.

So; as a way of saying "get well soon", let's get down to business;

You may want to look at Perry Cook's work with sensors, in the same period that he worked with coffee cups with tilt sensors he worked with pressure sensors in shoes, there is footstep synthesis as well. "Billy's gait" I believe it's called.

Aside from sensing, foot movements could be used to generate electricity, probably enough to power small boards. That might be interesting, especially with a board meant to generate music/sound that's designed to "degrade" in interesting ways at getting too little juice.

Aside from generating sound directly acoustically and being able to triger synthetic sounds feet might be used as a "tap tempo", a signal that determines the tempo of the music while the music plays.

Different sensors may be used and compared. You could for example wear a sensor on your chest/head as well. When the sensors indicate your feet are moving up and down yet your chest is mostly in place on the vertical axis that would indicate a different way of stepping from the situation where your whole body moves up and down. Another interesting thing to compare would be whether your right foot starts moving up slightly before or slightly after the left touches the ground, that would be the difference between jogging and walking. Such differences and relationships could allow you to extract more information from just a few sensors; more expressive, more economical and more fun to code as well.

I wonder if a cheap heart-rate sensor could be build, this might yield interesting data to sonify and could also be used to determine when you would be crossing the "mild exercise" line.

Those are just some ideas that pop up for me right now. Do consult a doctor about any exercise ideas that you have the slightest doubt about. Typically calm, steady walks are one of the most healthy ways of exercise in that there is almost 0 chance of injury, with our whole body set up for them (humanity used to live in nomadic tribes, that's where our bodies got their current form) but I'm no doctor; I know a thing or two about experimental interfaces but little about health. That said, this does sound like a amazing idea.

[Inventor]

Kassen, thank you for your concern and your comments. You are right on the mark healthwise, but a tad off the mark as for the musical interface I am interested in creating. It all started when I imagined a foot mouse long ago, and more recently imagined a form of it where the left foot movies the mouse left and right and the right foot moves the mouse up and down. In time one could adjust to that and learn to control the mouse that way.

The thing is, I don't have pedals, I have accelerometers. Also I don't have the expertise to write drivers for them to act like a mouse and the Mac doesn't have an option to substitute a joystick for a mouse. So this leaves me with a music application in ChucK.

For motion the easiest thing to sense is the angle with respect to gravity, so that's what I'll do, and we have two of them. So it's like having two potentiometer inputs. Now the question is what to control with them? I dunno and that's why I'm posting to get specific suggestions of what to do here.

I'll have a look at Spencer's work also.

Les

[Kassen]

I do seem to remember the specs for the larger Ultimark interface (without the tilt sensors) saying interfaces could be configured to work as mouse axis.

Whether it would be a good idea to control a mouse with accelerometers using your feet would be another question (to put it bluntly; I don't think it would be) but I thought that was possible.

As for the suggestions; it wasn't clear what type of movement you were aiming at so I was going with the title of the topic and assumed you wanted to stamp/step/walk/etc. The angle relative to gravity is a interesting way to use them but that's only a significant factor during slower motions, it would really depend on what kind of exercise you are looking for.

Can we start with that? Did your doctor recommend any specific movements? In this case it might be interesting to start from the movement, then see where that leads us.

[Inventor]

You're right, Kassen I wasn't specific at all. Basically the doc said that walking or pedaling or moving of any kind will help. The idea is that the foot muscles when contracting pump the blood up from the foot.

By the way, the swelling is way down today so I am happy about that. What I was thinking of was some kind of music control based on the two elevation values of the feet rocking on their heels. Then later gestures could be added by a quick twist to the left or right, but basically the two angles are the main control function.

For example, maybe I could play the guitar by selecting the frequency with one foot and the plucking with the other. But it has to be interesting, like a game or something... I could do it in MAUI for example. How about resurrecting Pong and controlling pong with foot action? Or space invaders or something? I dunno. It's got to be fun anyways.

[rogan]

Hi Inventor!

Like Kassen, I'm relieved to hear that your diagnosis is treatable. The only thing that's popping in my head right now would be

Chuck Chuck Revolution!!!

Although I'm not exactly sure the best way to implement it -- lights that turn on or off on MAUI, maybe? Or maybe more like Guitar Hero, but with your feet.... Well, it's a start

[Inventor]

Thanks rogan, what a great idea. But now I feel lazy. Let the old brain chew on this one for a while, i guess. Thanks!

[Kassen]

[Inventor]

OK, ChucK-ChucK Revolution it is. I'd like to do it kind of original though, using ChucK's MAUI abilities. For example, moving an LED around is sketchy at best because it leaves ghost images. I did write one game (Eliminator) in ChucK that worked with an array of LEDs, and that was ok. Then there are sliders which can be moved, and buttons which can be greyed.

Definitely the exercise aspect of DDR appeals to me, so that's the main benefit. But I gotta make it jam... let's see, how about if it responded to my iTunes music? Yeah! That way I could jam to my favorite tunes which would make it fun.

I could make the arrows appear as a function of the beat of the music somehow. It doesn't have to be strictly DDR with the arrows. Maybe it's a game of stomp on the FFT? As the song plays, the FFT grows according to the song, and You have to "stomp it out", with each foot tap in the right location chopping down the FFT there.

Well, it's a working concept. Opinions? Ideas?

[Inventor]

Now I think I have a good idea: it's a game of stomp on the spectrum. You play your iTunes music into it and then the spectrum grows and you have to stomp it out. Could be great fun! I'll have to give it some more thought though.

Edit: oh wait, I said that. Never mind. Strange mind at work!

[Kassen]

I agree, original is better. There are ideas here, the genre is quite diverse;
http://en.wikipedia.org/wiki/Music_video_game

FFT-based "whack a mole" sounds great, especially with a good mapping and simple music (I wouldn't use Chopin for that...). Maybe use a delay line so you could have visual indications of upcoming events before they are there.

At least at first I'd keep things at a lower tempo with clear signal for in- and output, you can always go into more detail once you know the simple version works and/or when your shape improves (ideally these would go hand in hand, but the real world isn't ideal and both getting in shape and audio analysis can be hard, still; we can dream).

Personally I'd also include some sort of feedback on successful moves, especially ones that are perfectly timed. This will let you see how you improve which is one of the main appeals to games like that. That's good as it will make you stick to exercising. Most interesting to me is probably musical feedback, for example FX. I could imagine a exercise game that would be completely about remixing a track by stepping/moving in the right way at the right moment.

[Inventor]

OK, I'm still leaning towards FFT whack-a-mole. It has the advantage that both feet do the same thing, rather than one foot aims and the other shoots. We still have the keys and/or mouse left to use though. And Kassen spoke of shaping the music somehow, I wonder how?

Perhaps this. As gameplay progresses, The FFT envelope is the FFT amplitude function. So if you've stomped out all your treble, you hear a lot of bass. This is your clue to get on over there and do some bass stomping. And so on. If you let the whole FFT grow, which will happen over time, the music gets louder and louder. OK I can work with that.

Any other ideas before I begin coding?

[rogan]

This sounds great. I can imagine using this sort of technology in live situations.

Here's a couple things that could be fun.

1. Make the game stereo, have one foot control each output's spectrum.
2. Add in gestures that can send the sound back and forth in the stereo range -- like a damped harmonic oscillator that you excite by kicking it (this isn't really compatible with 1).

[Inventor]

Rogan, now you've got me thinking about a BOXING game. The original idea was to move the feet, but i believe that any exercise would get the ol' blood pumping and help. I've thought about having a boxing game for years!

I'm thinking like a headbanger thing where the rock is playing loud and you pound your fist to the beat of the music for some reason. Left hand on left channel and right hand on right channel perhaps. Or for simplicity's sake just do one hand, most people will only have one sensor anyway.

I'm not programming yet cause i'm feeling really lazy - no coffee or sugar in the apartment, sigh. So there's plenty of time to think this out. What shall it be?...

[Kassen]

Boxing actually involves quite a lot of footwork, I think. Headbanging on the other hand is probably quite light on it....

[Inventor]

Well, lol, headbanging it turned out to be. Now I've secured the accelerometer to the top of my headphones and when I tilt my head left and right the music fades left and right, haha.

Further, I made a grid of LEDs, 30x20, and I light one up green based on the head tilting left/right for horizontal and forward/backward for vertical.

This is the basis for a game but I don't know where to go with it. I know I've gotten away from the main purpose which was foot exercise, but whatever, I'm having fun with it. More later.

[Inventor]

Well, I ended up making a Berzerk style game, as shown below. The red dots are enemies that cruiese around in random directions and the blue dot is a bullet. Not shown is the green player dot because it is a game over situation.

The music pans left and right and a band pass filter emphasizes bass when the player is at the bottom and treble when at the top.

It's kind of fun to play. Something to do, I guess. No sense posting source code since nobody else has an accelerometer.

---

BerzerkImage.jpg
Description:	Image of the game
Filesize:	162.56 KB
Viewed:	163 Time(s)
This image has been reduced to fit the page. Click on it to enlarge.

---

[Kassen]

[Inventor]

I sure wish that cross-platform MAUI would happen. That or GlucK.

[Kassen]

I think both have their purpose. GlucK would be nicer to play with but with the rise of netbooks and Ge's current interest in mobile computing I think there are still a lot of applications for simple 2d interfaces.

I have to say though; these "led array" displays are interesting as well in that it's so extremely minimal and low-res. that leads to interesting questions. I'm one of those annoying people that believes the very old computer games had more emphasis on gameplay (which is where I think the lasting fun is) because there was so little else that one could do.

[Antimon]

[Kassen]

A text adventure? Like this, you mean?
http://netjam.org/quoth/

:¬)

[Antimon]