electro-music.com   Dedicated to experimental electro-acoustic
and electronic music
 
    Front Page  |  Articles  |  Radio
 |  Media  |  Forum  |  Wiki  |  Links  |  Store
Forum with support of Syndicator RSS
 FAQFAQ   CalendarCalendar   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   LinksLinks
 RegisterRegister   ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in  Chat RoomChat Room 
 Forum index » DIY Hardware and Software » ChucK programming language
Generating Music Videos with ChucK
Post new topic   Reply to topic Moderators: Kassen
Page 1 of 2 [26 Posts]
View unread posts
View new posts in the last week
Mark the topic unread :: View previous topic :: View next topic
Goto page: 1, 2 Next
Author Message
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Sat Oct 20, 2007 11:22 am    Post subject: Generating Music Videos with ChucK
Subject description: Lame example of promising technique
Reply with quote  Mark this post and the followings unread

Attached are a two-second long *.MPG file that was generated by ChucK and POV-Ray, and also the ChucK source code for doing it. ChucK listens to the microphone and does an FFT on it. This FFT is then represented by a chain of grey cylinders, each representing a vector in the complex plane. By starting with fft.cvals(0), making a vector, then getting fft.cvals(1) and adding that vector to the first vector, and repeating until we have gone all the way through the FFT output, we get a graphical linkage-looking thing that represents the entire FFT. This is the image that I chose to animate for the video.

I then have Chuck send out formatted text that contains these numbers arranged along with POV-Ray commands in such a way as to form an input source code for POV-Ray. POV-Ray then draws each frame of the animation and compiles them together into a *.mov file which I then convert into a *.MPG file with a file converter so that everyone can see it (not everyone can view *.mov files).

The result is a very lame music video, but it is promising because it is an animation sequence that was generated by ChucK itself, just post-processed by animation software. Since POV-Ray is a photo-realistic renderer, we can make stuff like bouncing balls, waving grass, even animated human forms that move in a way that comes directly from the music.

For example, we could take blob-man, which is a POV-Ray human with movable joints, and make him dance in response to the music (although the author of blob-man expressly forbids monetary profit from blob-man). There is a seemingly unending list of possibilities in creating music videos this way.

By the way, if you watch the video, ignore the black patches and concentrate on the gray FFT linkages. The patches are an accidental artifact of the way I am making the linkages appear and disappear. I also noticed that ChucK sends output from <<< "example" >>>; to stderr instead of stdio for some reason, so you have to do "chuck chuck2POVRay2.ck >& temp.pov" to capture the text. Does anyone know how to get the text output of ChucK to go to stdio? No big deal, it works this way.

One major problem I had is that running the program at full speed creates nan's, or not-a-number's in the FFT output. Not sure why that is, I just put in a variable that skips over more time than one FFT input sample to temporarily patch this, but it needs attention.

If you are interested in POV-Ray, you can find it at:

http://www.povray.org

POV-Ray is freeware and it runs on Mac, PC, and various Unix's.

I'm open to suggestions as to how you would choose to represent portions of the music stream as video information. If such a suggestion lends itself well to POV-Ray, I might code it. Oh, also I'm not too keen on timing the music to the video, or what tool to use to add them together, I guess for me it would be iMovie since I'm on a Mac.

Your ideas and responses are encouraged.


temp.MPG
 Description:
Ultra-lame but promising music video generated by ChucK and POV-Ray, 2 seconds long.

Download
 Filename:  temp.MPG
 Filesize:  63.55 KB
 Downloaded:  369 Time(s)


chuck2POVRay1.ck
 Description:
The chuck source code that generated the POV-Ray source code that made the video.

Download
 Filename:  chuck2POVRay1.ck
 Filesize:  2.69 KB
 Downloaded:  202 Time(s)

Back to top
View user's profile Send private message Send e-mail
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Sun Oct 21, 2007 4:01 am    Post subject: Reply with quote  Mark this post and the followings unread

OK, this video is a little bit better. It is a fly-over of the time-varying FFT of my Boolean_Trio.ck song. Actually it is only the first 1/4 of the FFT spectrum. The fly-over is 5 frames per second and it lasts for five seconds. It could use better lighting and a more interesting spectrum, etc, but what I am going for here is general concepts, not details.

The chuck code produced all of the POV-Ray source code, then I just deleted a few spurious text lines and rendered the animation. There is no sound in the video as I am still just playing around with ideas.

I solved the problem of getting not-a-number's in the FFT output by taking the entire time series of FFTs and stuffing them into a 2D array, then doing the text output after all of the data was collected. Works great!


chuck2POVRay2.MPG
 Description:
FFT fly-over, kind of dull but better than my first attempt. 5 seconds long at 5 frames per second.

Download
 Filename:  chuck2POVRay2.MPG
 Filesize:  467.55 KB
 Downloaded:  329 Time(s)


chuck2POVRay2.ck
 Description:
The ChucK source code for generating the FFT fly-over.

Download
 Filename:  chuck2POVRay2.ck
 Filesize:  3.06 KB
 Downloaded:  184 Time(s)

Back to top
View user's profile Send private message Send e-mail
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Sun Oct 21, 2007 5:49 am    Post subject: Reply with quote  Mark this post and the followings unread

I made another FFT fly-over and this one is much better. I'll refrain from posting it here, though because of file size limitations. You can view it on my website at http://www.freedomodds.com/music/. It's not much, but its a good beginning.
Back to top
View user's profile Send private message Send e-mail
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Mon Oct 22, 2007 11:05 am    Post subject: Reply with quote  Mark this post and the followings unread

This one is getting closer to a decent animation. It features animated blobman and blobwoman who move in response to the frequencies of the instruments played. Also there are three spheres that change size in proportion to the frequency of the three instruments, plus an FFT overhead.

I'm pleased to now be able to do animations properly in POV-Ray, and happy with the blobpeople, though it will take some time to learn how to position them. Oh, as before, all of the video is generated from ChucK.


blobman2.mov
 Description:
Blobman and blobwoman move in response to music. Also has spheres and fft. short duration.

Download
 Filename:  blobman2.mov
 Filesize:  327.02 KB
 Downloaded:  446 Time(s)

Back to top
View user's profile Send private message Send e-mail
Kassen
Janitor
Janitor


Joined: Jul 06, 2004
Posts: 7678
Location: The Hague, NL
G2 patch files: 3

PostPosted: Tue Oct 23, 2007 3:38 am    Post subject: Reply with quote  Mark this post and the followings unread

Excelent stuff!!!!

I've been planning to try a similar sort of thing for a while now, this is very inpiring, do keep us up to date about your progress!

_________________
Kassen
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Tue Oct 23, 2007 6:39 am    Post subject: Reply with quote  Mark this post and the followings unread

Kassen wrote:
Excelent stuff!!!!

I've been planning to try a similar sort of thing for a while now, this is very inpiring, do keep us up to date about your progress!


Thanks a bunch, I live for compliments! I am up to blobman6 now. This one has no spheres. There is a scrolling time-series FFT that forms a planar surface on which the two blobpeople now stand. I have them squatting for low frequencies, standing for high freqeuencies, and moving in-between for midrange frequencies. Blobman is keyed to the drums and blobwoman is keyed to the clarinet. Also the camera rotates and I colored the FFT with red for low frequencies, green for high frequencies, and shaded red/green in the midrange. Also the drums are on the left channel while the clarinet is on the right channel.

The only problem is that iMovie insists on converting the video from 8 frames per second to 12 frames per second, so the characters are not precisely timed with the notes. It still works ok though, there is plenty of eye candy - you can watch either figure or the scrolling FFT but you can't really take it all in at once easily.

The video is kind of big and I don't want to use up all my forum disk quota, so I put it on my music web page and you can view it there at:

http://www.freedomodds.com/music/

I also put the source code there if you are interested in that. The source generates the music and outputs all of the text to create the animation file. So I have a program running on a program that writes a program that runs on another program, haha. Enjoy!
Back to top
View user's profile Send private message Send e-mail
Kassen
Janitor
Janitor


Joined: Jul 06, 2004
Posts: 7678
Location: The Hague, NL
G2 patch files: 3

PostPosted: Wed Oct 24, 2007 5:40 am    Post subject: Reply with quote  Mark this post and the followings unread

I watched 6 and the new 7. I like how the plot becomes like their grass. keep it up, it's great to see your progress chronicled like this! Do you have a end goal in mind? A ice-skating couple to a aklgorithmic waltz would be good! oh, and breakdancing to electro, of course.

I think Ge had some plans to make the Audicle "dude" dance as well, that would be good.

Last night I made some modest progress in developing my own visualisation plan, mostly by talking to people who actually know things about visuals, which I don't Laughing

_________________
Kassen
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Kassen
Janitor
Janitor


Joined: Jul 06, 2004
Posts: 7678
Location: The Hague, NL
G2 patch files: 3

PostPosted: Wed Oct 24, 2007 5:45 am    Post subject: Reply with quote  Mark this post and the followings unread

Oh, and let me say how happy I am to see you use a low FPS timed to the BPM! I was planning on doing that as well to get aesthetically pleasing effects at more modest cpu cost but as I mentioned I know nothing about graphics so I had no idea whether that was at all sensible.
_________________
Kassen
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Wed Oct 24, 2007 12:01 pm    Post subject: Reply with quote  Mark this post and the followings unread

Kassen, I appreciate that you took the time to view my videos, it would be no fun if people didn't check out my stuff. As to an end-goal, I purposely have no plans. I lived all my life studying and laboring as an engineer and there are always plans! Managers make up unrealistic schedules with project "planning" software, then purposely try to evoke fear in their employees to motivate them to accomplish the goals of the plan. It took its toll on me over the years, so now I am intentionally pursuing this music hobby with no plan whatsoever! They say living well is the best revenge, well this is my revenge, haha.

I'd say if there is any real goal, it is to provide others with yet another way to express their creativity. By using free software that runs on almost any computer, I am helping to make this form of music video production available to anyone with the time and the motivation to do it. I also intend to have a really good time doing stuff that other people enjoy.

As to animating the characters, I chose the low FPS timed to the BPM simply because it was the easiest thing to do. The program has 16 nested for loops with the inner loop having a one-note time increment of 1::second/beats_per_second. If I wanted to do 30 FPS I would have to interpolate the position of the blobpeople which now that you mention it would not be that difficult - it just didn't occur to me. Also the animation time comes in to play. It's only 4 seconds per frame on this computer, but that quickly adds up and these short videos require about 10 minutes of animation time.

I like your ideas of the ice skater and the break dancer. The graceful nature of a waltz might have smoothly varying parameters that could be encoded into a skater's motion, and perhaps some free-form break dancer could be done from the right music. My first experiment of randomly assigning frequencies to body parts went poorly, however, which is why I simplified and did the whole body motion with only one parameter: the frequency of one instrument. However, I believe that studying the music carefully with the body animation in mind would lead us in the right direction.

Now you've got me curious as to what your animation plans are. Would you mind sharing some ideas, or are you keeping it quiet for now? If it would be helpful, I could possibly create some POV-Ray animations with ChucK for you as a crude initial sketch of some of your ideas. I like to collaborate on projects sometimes, if I have the time and the motivation.
Back to top
View user's profile Send private message Send e-mail
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Thu Oct 25, 2007 10:04 pm    Post subject: Reply with quote  Mark this post and the followings unread

I just made some more videos, most notably Torus 1. It has a bass guitar and lead guitar from kijjaz, Boolean sequencing, and a rotating glass torus with sparkling colored lights. I got the idea from one of CyberDude's videos in which he has a slowly rotating torus with texture on it. This one is 60 seconds long and I finally figured out how to make iMovie NOT compress the video size. Have a look if you like and let me know what you think.

http://www.freedomodds.com/music/
Back to top
View user's profile Send private message Send e-mail
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Sat Oct 27, 2007 5:44 am    Post subject: Reply with quote  Mark this post and the followings unread

I have now created an FFT plot that changes with the music, but this one has a nice feature: it works like rec-auto.ck. If you want the FFT animation, you modify the source code with your desired framerate and duration in seconds, then just add the FFTbars.ck filename and text redirection to your chuck command line, like this: "chuck your_music_file.ck FFTbars.ck >& temp.pov". FFTbars will listen to the dac and make its plots, then you can render it on POV-Ray. Here is an image of the animation:

Posted Image, might have been reduced in size. Click Image to view fullscreen.

You can view a 60 second music video that I made with FFTbars.ck here:

http://www.freedomodds.com/music


FFTbars.ck
 Description:
Makes an FFT music video to go with any chuck song. Render it in POV-Ray.

Download
 Filename:  FFTbars.ck
 Filesize:  2.71 KB
 Downloaded:  190 Time(s)

Back to top
View user's profile Send private message Send e-mail
Kassen
Janitor
Janitor


Joined: Jul 06, 2004
Posts: 7678
Location: The Hague, NL
G2 patch files: 3

PostPosted: Sun Oct 28, 2007 8:09 am    Post subject: Reply with quote  Mark this post and the followings unread

I love the new stuff!

To -belatedly- reply to your question; what I want to do is still very vague, I'm not even sure I will use a 3d library since I've been thinking that maybe 2d would suit me better, at any rate I'm most interested in working from a 2d perspective, putting more focus on the eventual image on the screen then on creating a virtual world.

What I do know is that I'd like to approach this from the data in the sequencer (the patterns) instead of a analysis of the eventual sound. I'm hoping this will allow for more knowledge in the system about the overall musical context being preserved, I think it will also allow for a more detailed way of dealing with latency.

But aside from that nearly everything is still open... I'm in the unfortunate stage where I have enough ideas and hints of ideas to keep me awake at night, but not enough clarity to start coding yet :¬)

_________________
Kassen
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Mon Oct 29, 2007 4:57 am    Post subject: Reply with quote  Mark this post and the followings unread

Kassen wrote:
But aside from that nearly everything is still open... I'm in the unfortunate stage where I have enough ideas and hints of ideas to keep me awake at night, but not enough clarity to start coding yet :¬)


Ah, well, I can't be much help then, I'll have to wait and see what you choose to do! I'm glad you like the new stuff, its kind of cool to have ChucK generating some video, even if it is just a simple FFT graph. Now what next? Hmmm...
Back to top
View user's profile Send private message Send e-mail
kijjaz



Joined: Sep 20, 2004
Posts: 765
Location: bangkok, thailand
Audio files: 4

PostPosted: Tue Oct 30, 2007 8:38 am    Post subject: Reply with quote  Mark this post and the followings unread

I wanna see light sources and objects flying around according to FFT complex output hahahahaha.. -_-"
That'd be cool coz the complex is 2D already.
And we'll get some cool circular movements in the animation, I hope.

This is just one idea from me, I'm starting to want to learn POV-ray
Back to top
View user's profile Send private message Send e-mail Visit poster's website Yahoo Messenger MSN Messenger
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Tue Oct 30, 2007 10:18 am    Post subject: Reply with quote  Mark this post and the followings unread

kijjaz wrote:
I wanna see light sources and objects flying around according to FFT complex output hahahahaha.. -_-"
That'd be cool coz the complex is 2D already.
And we'll get some cool circular movements in the animation, I hope.

This is just one idea from me, I'm starting to want to learn POV-ray


Funny you should mention that, kijjaz, because I recently tried to make some graphics with the FFT complex output. Here is the lightning that I made:

Posted Image, might have been reduced in size. Click Image to view fullscreen.

It flashes around every 1/8 of a second on the video that I made which is called "FFT Bars 5" on my site:

http://www.freedomodds.com/music/

It is currently the second entry, but that will change as I add more stuff. I would like to do something else with the complex FFT output but I don't know what yet. Lights flying around, hmmm... let me give it some more thought!
Back to top
View user's profile Send private message Send e-mail
Blue Hell
Site Admin


Joined: Apr 03, 2004
Posts: 20618
Location: The Netherlands, Enschede
Audio files: 148
G2 patch files: 318

PostPosted: Tue Oct 30, 2007 4:10 pm    Post subject: Reply with quote  Mark this post and the followings unread

Inventor wrote:
Lights flying around, hmmm... let me give it some more thought!


maybe flying balls, like the ones that I've toying with recently again. Just a play of force, attraction, repulsion, size, temperature, lots of possible parameters to play with !


_________________
Jan
Back to top
View user's profile Send private message Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Wed Oct 31, 2007 11:43 pm    Post subject: Reply with quote  Mark this post and the followings unread

I enjoyed the swarm of colored balls, how did you do that? Though I couldn't hear any music on this one. I did try something with larger glass spheres and colored lights that turned out nicely, maybe I could work with something like that.
Back to top
View user's profile Send private message Send e-mail
Blue Hell
Site Admin


Joined: Apr 03, 2004
Posts: 20618
Location: The Netherlands, Enschede
Audio files: 148
G2 patch files: 318

PostPosted: Thu Nov 01, 2007 2:00 pm    Post subject: Reply with quote  Mark this post and the followings unread

Inventor wrote:
I enjoyed the swarm of colored balls, how did you do that? Though I couldn't hear any music on this one. I did try something with larger glass spheres and colored lights that turned out nicely, maybe I could work with something like that.


There is no music in the video, and it's not done with ChucK either. Just some Pascal code I had laying around from some old screen saver I made.

It's a model where balls have have a sort of "mass" depending on size (r^3), they live in a 3d box and depending on color they will attract or repel other balls, and I can control the force constant, the amount of random movement and damping. When they hit a wall they'll bounce, but they can freely move through each other. Sort of a fantasy science Very Happy

When you want I could post the code here.

_________________
Jan
Back to top
View user's profile Send private message Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Thu Nov 01, 2007 2:52 pm    Post subject: Reply with quote  Mark this post and the followings unread

Blue Hell wrote:
When you want I could post the code here.


Yes, I'd like to see the code. I'm wondering what to do for my next video. I want to use clear objects and colored lights, possibly flying above a crystal FFT ocean, but I still don't know how I will animate them. Some physics-based thing like that might be it... kind of like to make it respond to the music though... maybe using kijjaz's integrator, I'm not sure.
Back to top
View user's profile Send private message Send e-mail
Blue Hell
Site Admin


Joined: Apr 03, 2004
Posts: 20618
Location: The Netherlands, Enschede
Audio files: 148
G2 patch files: 318

PostPosted: Thu Nov 01, 2007 4:17 pm    Post subject: Reply with quote  Mark this post and the followings unread

Ok, here is the uncleaned version for now, working on a cleanup.

Damn, I must turn on the pas extension .... ok.


FrmMain.pas
 Description:

Download
 Filename:  FrmMain.pas
 Filesize:  20.88 KB
 Downloaded:  199 Time(s)


_________________
Jan
Back to top
View user's profile Send private message Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Thu Nov 01, 2007 7:50 pm    Post subject: Reply with quote  Mark this post and the followings unread

Yes, I remember doing something like this once before for a graphical version of "ls". Each ball has a mass, with some negative and some positive and magnitude according to color. You give them initial conditions and apply gravity to them, and they move about in response to each other. How did they relax to the same end-position as start position, and how does the temperature work? Is it that they start out at low temperature, you apply high temperature, and then you reduce to low temperature at the end?
Back to top
View user's profile Send private message Send e-mail
Blue Hell
Site Admin


Joined: Apr 03, 2004
Posts: 20618
Location: The Netherlands, Enschede
Audio files: 148
G2 patch files: 318

PostPosted: Fri Nov 02, 2007 3:42 am    Post subject: Reply with quote  Mark this post and the followings unread

For the movie I controlled temperature, damping and the constant of gravity with sliders. Temperature just adds a random component to the movement (either speed or acceleration) independent of force, but dependent on mass. So these are the parameters to control the model with, and I thought it would be nice to derive those from music in some way - something I'd like to try some day Very Happy
_________________
Jan
Back to top
View user's profile Send private message Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Sat Nov 03, 2007 12:34 am    Post subject: Reply with quote  Mark this post and the followings unread

Here is another music video I just made called "Crown of Thorns". It features a rotating glass crown of thorns with colored light and a flute/mandolin duet.

http://www.freedomodds.com/music/movies/CrownOfThorns.mov

I made the crown algorithmically by drawing cylinders connected by sphere joints, then added random thorns. There are control variables for various parameters such as thorn length, stick thickness, and number of turns. I like the way it turned out, though it took 8 hours to render!
Back to top
View user's profile Send private message Send e-mail
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 5978
Location: San Antonio, Tx, USA
Audio files: 258

PostPosted: Mon Nov 05, 2007 10:31 am    Post subject: Reply with quote  Mark this post and the followings unread

I just finished yet another 1 minute video short. This one is called "Crosses" and it features three backlit glass crosses, one for bass one for midrange, and one for treble. The crosses change size and jump out at you in response to the music. I made this file like rec-auto.ck, it listens to the dac and does its FFT magic all on its own so you can ChucK it with any other ChucK files you have. Here is the movie:

http://www.freedomodds.com/music/movies/Crosses.mov
Back to top
View user's profile Send private message Send e-mail
skwrKing



Joined: Jul 24, 2007
Posts: 16
Location: MI

PostPosted: Tue Nov 06, 2007 5:57 am    Post subject: Reply with quote  Mark this post and the followings unread

The videos look good. I imagine it would be difficult to get it set up to render these visuals in realtime... a shame. I've not heard of POV-Ray before. It's always good to come across powerful free software. But anyway, it sounds like an interesting technique you've come up with. Keep it coming.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic Moderators: Kassen
Page 1 of 2 [26 Posts]
View unread posts
View new posts in the last week
Goto page: 1, 2 Next
Mark the topic unread :: View previous topic :: View next topic
 Forum index » DIY Hardware and Software » ChucK programming language
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum
e-m mkii

Please support our site. If you click through and buy from
our affiliate partners, we earn a small commission.


Forum with support of Syndicator RSS
Powered by phpBB © 2001, 2005 phpBB Group
Copyright © 2003 through 2009 by electro-music.com - Conditions Of Use