Setup for improvising live

[Dstream]

I have made performances together with a painter where we have improvised over existing music of mine. (Really nice way of making art by the way). That means I had some loops, arpeggios, sounds and stuff preprogrammed and arranged them live and improvised som new stuff on top and I know a few things that will appear on her painting but she improvises as well. (Tell me if you don't understand and I'll elaborate.) Now we have talked about making perfomances where we don't have anything desided when we get on stage. Therefor I now started to create a setup where I live can do almost whatever I wish in the heat of the moment, starting from nothing. I would love to get some ideas, thoughts and feedback on this matter.
I use ableton live as the center of it all with the APC40 as controller. I've started out by having two channels with drums and percussion of varous kinds and on these channels I've inserted a Max-plugin that turns the apc to a step-sequencer when tracks are selected. I have two midichannels taking input from midikeyboards, one m-audio keystation pro88 and one radium. These are routed out to a yamaha motif rack and a korg tr-rack. The reason I use the midicontrollers through the computer is that I intend to assign knobs and faders to stuff I want to control inside ableton, and of course I can record a midibit and let ableton loop it for me. I also intend to have some more fun synths such as Roland SH101 and a yamaha SY77 to play around with. In the small mixer I've put the input on my computer on a group and in ableton I've got some empty audiotracks so if I wish to record an audiopart for looping from any hardware synth I just hit the groupassign and record in a slot of my choice.
One thing I've been thinking about is how do I have as many choices of drums as possible with this kind of setup? I don't want to have to many tracks in ableton, since that makes it easy to get lost and pushing the wrong button, and I rather not start to drag and drop samples while playing since it sometimes affect the audio-output.

This is my first steps in creating this and any ideas and thoughts and how you would do it are most welcome!