electro-music.com   Dedicated to experimental electro-acoustic
and electronic music
 
    Front Page  |  Radio
 |  Media  |  Forum  |  Wiki  |  Links
Forum with support of Syndicator RSS
 FAQFAQ   CalendarCalendar   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   LinksLinks
 RegisterRegister   ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in  Chat RoomChat Room 
 Forum index » DIY Hardware and Software » ChucK programming language
Memory Exhausted
Post new topic   Reply to topic Moderators: Kassen
Page 1 of 2 [26 Posts]
View unread posts
View new posts in the last week
Mark the topic unread :: View previous topic :: View next topic
Goto page: 1, 2 Next
Author Message
ultrasine



Joined: Aug 31, 2006
Posts: 16
Location: Europe

PostPosted: Thu Jun 05, 2008 1:08 pm    Post subject:  Memory Exhausted
Subject description: Input file with large number of lines or using big arrays causes crash
Reply with quote  Mark this post and the followings unread

Hey everyone, hope you all are chilling.

I just started messing about with chuck again after a long break but straight away I started crashing it with the following example error output:

[test.ck]:line(3).char(19991): memory exhausted

Having large number of lines (>5000?) or big arrays causes this.

I shan't post the input files, apart from top secret chuck norris kick butt code, I don't want to be held responsible for other peoples computers melting.

Anyone know what the limits are and how I can break these? I'm running chuck.oss 1.2.0.7 on a 256MB debian laptop.

Aye.
Back to top
View user's profile Send private message Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 6221
Location: near Austin, Tx, USA
Audio files: 267

PostPosted: Tue Jun 10, 2008 2:32 pm    Post subject: Reply with quote  Mark this post and the followings unread

Hi, sorry I didn't see your post earlier. The error message indicates that you have almost 20,000 characters on a single line of your code. I can only guess, but perhaps you are running into a line length limit of some sort. could you break up that line with carriage returns? Just a guess. Good luck!
_________________
"Let's make noise for peace." - Kijjaz
Back to top
View user's profile Send private message Send e-mail
Kassen
Janitor
Janitor


Joined: Jul 06, 2004
Posts: 7678
Location: The Hague, NL
G2 patch files: 3

PostPosted: Tue Jun 10, 2008 5:52 pm    Post subject: Reply with quote  Mark this post and the followings unread

Erm, 20K characters on a single line? How do you edit or read that at all? Something is going wrong there. 5000 lines shouldn't be a issue but 20 thousand characters on a line is definitely unusual. I don't think that should inherently be cause for a crash either... but it is kinda extreme, I never heard of such a thing. Could you perhaps explain a little about *why* you need or like such lines?

I don't mind a molten computer or two, BTW, could you condense the code to the smallest possible construct that still crashes?

Oh, and 1.2.0.7 is quite old. Regardless of whether it fixes the issue I'd upgrade to the latest greatest (1.2.1.1b )

_________________
Kassen
Back to top
View user's profile Send private message Send e-mail Visit poster's website
ultrasine



Joined: Aug 31, 2006
Posts: 16
Location: Europe

PostPosted: Wed Jun 11, 2008 2:42 am    Post subject: Reply with quote  Mark this post and the followings unread

Aha! Thanks for the responses.

The error I posted was for a very long line. Before that I got a similar error for having to many lines of code - thats why I tested with a single line. So its not a matter of just breaking up the line. It is probably to do with how big the script is.

I downloaded chuck 1.2.1.1 source and compiled it but this gave the same results so I decided to investigate further.

In the file src/chuck.tab.c we have a "YYMAXDEPTH 10000" with comments:

/* YYMAXDEPTH -- maximum size the stacks can grow to (effective only
if the built-in stack extension method is used).

Do not make this value too large; the results are undefined if
YYSTACK_ALLOC_MAXIMUM < YYSTACK_BYTES (YYMAXDEPTH)
evaluated with infinite-precision integer arithmetic. */

I tried changing this constant but managed to pump it upto 15000. Any higher caused problems.

So, is this causing me problems? If so, how is this value decided and does it depend on the users hardware? I tried using unlimited stack size on my linux machine (256M RAM) but I still see the error.

I just created a test file and I post it here for you to check it as well. On my machine I get:
[large.ck]:line(4998).char(9): memory exhausted

As to why I need or like such things - dude, I have no idea. But when I get limited I usually don't have a need or liking for the tool. I just move on....


large.ck.zip
 Description:
At least 5000 lines of code to test how much chuck can grok.

Download
 Filename:  large.ck.zip
 Filesize:  12.77 KB
 Downloaded:  293 Time(s)

Back to top
View user's profile Send private message Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 6221
Location: near Austin, Tx, USA
Audio files: 267

PostPosted: Wed Jun 11, 2008 3:54 am    Post subject: Reply with quote  Mark this post and the followings unread

Well, before you move on let's give folks here on the forum and also on the DEV team a chance to look into the problem. ChucK is purposely coded with low performance in some ways so that it can be so free and effective in other ways. We usually run into CPU limitations by spinning around on samp => now or somesuch thing. This is a new limitation that you've found as far as I know. I'm not the best one to help, but perhaps we can post to the ChucK users group and see what they say. Even ChucK Norris has his limits, Grasshopper! (Grin)
_________________
"Let's make noise for peace." - Kijjaz
Back to top
View user's profile Send private message Send e-mail
ultrasine



Joined: Aug 31, 2006
Posts: 16
Location: Europe

PostPosted: Wed Jun 11, 2008 4:06 am    Post subject: Reply with quote  Mark this post and the followings unread

Inventor wrote:
Well, before you move on let's give folks here on the forum and also on the DEV team a chance to look into the problem. ChucK is purposely coded with low performance in some ways so that it can be so free and effective in other ways. We usually run into CPU limitations by spinning around on samp => now or somesuch thing. This is a new limitation that you've found as far as I know. I'm not the best one to help, but perhaps we can post to the ChucK users group and see what they say. Even ChucK Norris has his limits, Grasshopper! (Grin)


Sorry, ignore my last statement. I'm not with it today. If you can post to the users group on my behalf it would be appreciated.

Don't worry I'll be sticking with chuck for some time yet. Even tho I know his limits he is always ready and willing for some serious kick butt action.
Back to top
View user's profile Send private message Visit poster's website
Kassen
Janitor
Janitor


Joined: Jul 06, 2004
Posts: 7678
Location: The Hague, NL
G2 patch files: 3

PostPosted: Wed Jun 11, 2008 4:56 am    Post subject: Reply with quote  Mark this post and the followings unread

Weird.
Apparently 5000 lines *is* a issue. My desktop gives the exact same error you get, both in the MiniAudicle and in command-line ChucK.

Weird.

Why do you think this stack parameter is related to it?

_________________
Kassen
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 6221
Location: near Austin, Tx, USA
Audio files: 267

PostPosted: Wed Jun 11, 2008 5:23 am    Post subject: Reply with quote  Mark this post and the followings unread

OK ultrasine, I sent a message to the ChucK users group list for you, should have an answer in a day or two, maybe sooner. In the mean time, don't smash any bricks in frustration! Rolling Eyes
_________________
"Let's make noise for peace." - Kijjaz
Back to top
View user's profile Send private message Send e-mail
Kassen
Janitor
Janitor


Joined: Jul 06, 2004
Posts: 7678
Location: The Hague, NL
G2 patch files: 3

PostPosted: Wed Jun 11, 2008 5:49 am    Post subject: Reply with quote  Mark this post and the followings unread

Maybe I'm missing several important aspects to this.... again.... but erm, why wouldn't Ultrasine sign up to the list himself and post about the bug?
_________________
Kassen
Back to top
View user's profile Send private message Send e-mail Visit poster's website
blue hell
Site Admin


Joined: Apr 03, 2004
Posts: 24079
Location: The Netherlands, Enschede
Audio files: 278
G2 patch files: 320

PostPosted: Wed Jun 11, 2008 6:24 am    Post subject: Reply with quote  Mark this post and the followings unread

ultrasine wrote:
In the file src/chuck.tab.c we have a "YYMAXDEPTH 10000" with comments:

/* YYMAXDEPTH -- maximum size the stacks can grow to (effective only
if the built-in stack extension method is used).

Do not make this value too large; the results are undefined if
YYSTACK_ALLOC_MAXIMUM < YYSTACK_BYTES (YYMAXDEPTH)
evaluated with infinite-precision integer arithmetic. */


YYMAXDEPTH is likely the maximum depth of the parser stack, as this is the standard name for that used by Yacc / Bison (The YY prefixe comes from Yacc originally, but nowadays mostly Bison is used). I don't know if Lex / Flex and Yacc / Bison are invoked from the standard make (I think not) but when they are the value used in the C file will not be used at all, you'd have to change it in the .y or .l file (or possibly an include file) where it's defined.

The parser stack depth limits the complexity (nesting level) of the language constructs that can be parsed, it does not have a direct influence on the acceptable source code size or the size of data structures that can be used.

_________________
Jan
also .. could someone please turn down the thermostat a bit.
Posted Image, might have been reduced in size. Click Image to view fullscreen.
Back to top
View user's profile Send private message Visit poster's website
ultrasine



Joined: Aug 31, 2006
Posts: 16
Location: Europe

PostPosted: Wed Jun 11, 2008 6:27 am    Post subject: Reply with quote  Mark this post and the followings unread

Kassen wrote:
Maybe I'm missing several important aspects to this.... again.... but erm, why wouldn't Ultrasine sign up to the list himself and post about the bug?


New url, username and password to remember. Also for some strange reason I cannot reach lists.cs.princeton.edu or the chuck site. OK , I'm lazy.
Back to top
View user's profile Send private message Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 6221
Location: near Austin, Tx, USA
Audio files: 267

PostPosted: Wed Jun 11, 2008 6:44 am    Post subject: Reply with quote  Mark this post and the followings unread

ultrasine wrote:
Kassen wrote:
Maybe I'm missing several important aspects to this.... again.... but erm, why wouldn't Ultrasine sign up to the list himself and post about the bug?


New url, username and password to remember. Also for some strange reason I cannot reach lists.cs.princeton.edu or the chuck site. OK , I'm lazy.


I just figured I'd save him the trouble for just one question. I know I get frustrated with all the passwords and such that I keep up with these days. He seemed to be iffy about commitment to being a ChucKist just yet, so why not do the guy a favor, eh?

_________________
"Let's make noise for peace." - Kijjaz
Back to top
View user's profile Send private message Send e-mail
Kassen
Janitor
Janitor


Joined: Jul 06, 2004
Posts: 7678
Location: The Hague, NL
G2 patch files: 3

PostPosted: Wed Jun 11, 2008 6:59 am    Post subject: Reply with quote  Mark this post and the followings unread

Check.

I don't think I even know my own password for the mailing list anymore, I think there was a trick to getting the list server to send you your info on request.

_________________
Kassen
Back to top
View user's profile Send private message Send e-mail Visit poster's website
chuckles



Joined: Apr 02, 2007
Posts: 72
Location: San Diego, California

PostPosted: Wed Jun 11, 2008 9:21 am    Post subject: I also can't get to http://chuck.cs.princeton.edu Reply with quote  Mark this post and the followings unread

ultrasine wrote:
Kassen wrote:
Maybe I'm missing several important aspects to this.... again.... but erm, why wouldn't Ultrasine sign up to the list himself and post about the bug?


New url, username and password to remember. Also for some strange reason I cannot reach lists.cs.princeton.edu or the chuck site. OK , I'm lazy.


I've been having the same problem the last couple days: attempting to access chuck.cs.princeton.edu fails. Presumable the sysadmins at Princeton will fix this some day...
Back to top
View user's profile Send private message
ultrasine



Joined: Aug 31, 2006
Posts: 16
Location: Europe

PostPosted: Wed Jun 11, 2008 10:32 am    Post subject: Re: I also can't get to http://chuck.cs.princeton.edu Reply with quote  Mark this post and the followings unread

chuckles wrote:
ultrasine wrote:
Kassen wrote:
Maybe I'm missing several important aspects to this.... again.... but erm, why wouldn't Ultrasine sign up to the list himself and post about the bug?


New url, username and password to remember. Also for some strange reason I cannot reach lists.cs.princeton.edu or the chuck site. OK , I'm lazy.


I've been having the same problem the last couple days: attempting to access chuck.cs.princeton.edu fails. Presumable the sysadmins at Princeton will fix this some day...


Hard to believe but cs.princeton.edu is down.
Back to top
View user's profile Send private message Visit poster's website
ultrasine



Joined: Aug 31, 2006
Posts: 16
Location: Europe

PostPosted: Wed Jun 11, 2008 10:52 am    Post subject: Reply with quote  Mark this post and the followings unread

OK ladies, I think I've found the hack to increase this limit we have encountered.
You just need to increase the YYMAXDEPTH in src/chuck.tab.c and recompile.

I have tested with increasing YYMAXDEPTH and ever bigger inputs and can verify there is a direct relation with the 'memory exhausted' error.

I went all the upto 0.5 million lines of input (YYMAXDEPTH=1000001) before chuck started to take over the machine (start using hard drive as memory?) and I had to control-c after 4 hours without audio output. At least chuck did a clean exit.

My system limits are the following (thought stack size would be unlimited but no... strange?):

ultrasine@xenon:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


Looks like I have the biggest bad ass Chuck than all of ya!
Back to top
View user's profile Send private message Visit poster's website
chuckles



Joined: Apr 02, 2007
Posts: 72
Location: San Diego, California

PostPosted: Wed Jun 11, 2008 12:30 pm    Post subject: Re: I also can't get to http://chuck.cs.princeton.edu Reply with quote  Mark this post and the followings unread

ultrasine wrote:
Hard to believe but cs.princeton.edu is down.


That's got to be the very defintion of "ironic".
Back to top
View user's profile Send private message
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 6221
Location: near Austin, Tx, USA
Audio files: 267

PostPosted: Wed Jun 11, 2008 4:36 pm    Post subject: Reply with quote  Mark this post and the followings unread

ultrasine wrote:
Looks like I have the biggest bad ass Chuck than all of ya!


hahaha just what we need, a new ChucKist with a chip on his shoulder! Rock on dude!

_________________
"Let's make noise for peace." - Kijjaz
Back to top
View user's profile Send private message Send e-mail
ultrasine



Joined: Aug 31, 2006
Posts: 16
Location: Europe

PostPosted: Thu Jun 12, 2008 12:56 am    Post subject:   Reply with quote  Mark this post and the followings unread

Alrighty, I think I'll finish this thread by saying that it would be cool if this magic number YYMAXDEPTH could be set via an optional command line parameter.

Cheers everyone!
Back to top
View user's profile Send private message Visit poster's website
blue hell
Site Admin


Joined: Apr 03, 2004
Posts: 24079
Location: The Netherlands, Enschede
Audio files: 278
G2 patch files: 320

PostPosted: Thu Jun 12, 2008 4:27 am    Post subject: Reply with quote  Mark this post and the followings unread

I looked into this. The real problem is not YYMAXDEPTH but rather grammar rules like :

Code:

statement_list
        : statement                         { $$ = new_stmt_list( $1, EM_lineNum ); }
        | statement statement_list          { $$ = prepend_stmt_list( $1, $2, EM_lineNum ); }
        ;


in chuck.y.

This is right recursion which lets the stack grow without bounds in a linear fashion with the length of the statement_list.

Formulating this with left recursion, as in :

Code:

statement_list
        : statement                         { $$ = new_stmt_list( $1, EM_lineNum ); }
        | statement_list statement          { $$ = prepend_stmt_list( $2, $1, EM_lineNum ); }
        ;


would work with a bounded stack.

Also see the Bison manual for this, ( recursive rules : http://www.cs.utah.edu/dept/old/texinfo/bison/bison.html#SEC42 ).

_________________
Jan
also .. could someone please turn down the thermostat a bit.
Posted Image, might have been reduced in size. Click Image to view fullscreen.
Back to top
View user's profile Send private message Visit poster's website
Inventor
Stream Operator


Joined: Oct 13, 2007
Posts: 6221
Location: near Austin, Tx, USA
Audio files: 267

PostPosted: Thu Jun 12, 2008 6:31 am    Post subject: Reply with quote  Mark this post and the followings unread

Wow, Jan, that's a heck of a brainy thing you did there! Very Happy
_________________
"Let's make noise for peace." - Kijjaz
Back to top
View user's profile Send private message Send e-mail
ultrasine



Joined: Aug 31, 2006
Posts: 16
Location: Europe

PostPosted: Thu Jun 12, 2008 7:05 am    Post subject: Reply with quote  Mark this post and the followings unread

Hi Jan,

I changed the code in the file src/chuck.y to the second block of code you posted because that was the only thing to do. When I tried compiling I got:

chuck.y: In function ‘yyparse’:
chuck.y:299: warning: passing argument 1 of ‘prepend_stmt_list’ from incompatible pointer type
chuck.y:299: warning: passing argument 2 of ‘prepend_stmt_list’ from incompatible pointer type

I have never worked with grammer parsers so am clueless. Heck, I also avoid compiled languages so you can see this is not much fun for me.

All I want is chuck not to barf on large inputs. I know how to do that by changing YYMAXDEPTH and compiling on linux.

If you can supply the file chuck.y which makes chuck more flexible in handling large inputs then please do so.
Back to top
View user's profile Send private message Visit poster's website
blue hell
Site Admin


Joined: Apr 03, 2004
Posts: 24079
Location: The Netherlands, Enschede
Audio files: 278
G2 patch files: 320

PostPosted: Thu Jun 12, 2008 7:14 am    Post subject: Reply with quote  Mark this post and the followings unread

This is just one of the "standard" problems with Yacc / Bison. The default value for YYMAXDEPTH in Yacc used to always be 150 which with a correct grammar nearly never gets exhausted.

For Bison there (apparently) was some dynamics added in that the parser stack starts at a depth of 200 and it can grow dynamically to 10,000 (by default) - so running out of that is just absurd. Apart from that the re-allocations will cost time, especially when you start to run into virtual memory ... don't know though what the size of the stack items is, well it's not larger than a C double type but there are associated objects as well (those are probably on the heap).

There may BTW be other means to solve the problem by adding tactical %left precedence operators. Yacc is a bit of a black art, and it can cost quite some time to get things right for a complex grammar like ChucK has. Anyway, usually left and right recursion can usually just be changed into each other without introducing other problems - but please note that the $x arguments in the associated code (between the curly brackets "{" "}") need to be changed as well.

_________________
Jan
also .. could someone please turn down the thermostat a bit.
Posted Image, might have been reduced in size. Click Image to view fullscreen.
Back to top
View user's profile Send private message Visit poster's website
blue hell
Site Admin


Joined: Apr 03, 2004
Posts: 24079
Location: The Netherlands, Enschede
Audio files: 278
G2 patch files: 320

PostPosted: Thu Jun 12, 2008 7:22 am    Post subject: Reply with quote  Mark this post and the followings unread

ultrasine wrote:
Hi Jan,

I changed the code in the file src/chuck.y to the second block of code you posted because that was the only thing to do. When I tried compiling I got:

chuck.y: In function ‘yyparse’:
chuck.y:299: warning: passing argument 1 of ‘prepend_stmt_list’ from incompatible pointer type
chuck.y:299: warning: passing argument 2 of ‘prepend_stmt_list’ from incompatible pointer type


Please note that the $1 and $2 arguments in the associated code (between the curly braces) should be exchanged as well, as they positionally map onto the rules on the left side. Besides, I gave just one example of a rule to be changed .. there are more that have this problem although the code example you posted is likely to run into this specific one (being a statement_list).

I do not have a ChucK development environment setup at the moment, so I can't really experiment with this myself.

It would be easier BTW to just recode your example into a for() loop Wink

_________________
Jan
also .. could someone please turn down the thermostat a bit.
Posted Image, might have been reduced in size. Click Image to view fullscreen.
Back to top
View user's profile Send private message Visit poster's website
ultrasine



Joined: Aug 31, 2006
Posts: 16
Location: Europe

PostPosted: Thu Jun 12, 2008 11:21 am    Post subject: Reply with quote  Mark this post and the followings unread

Blue Hell wrote:
ultrasine wrote:
Hi Jan,

I changed the code in the file src/chuck.y to the second block of code you posted because that was the only thing to do. When I tried compiling I got:

chuck.y: In function ‘yyparse’:
chuck.y:299: warning: passing argument 1 of ‘prepend_stmt_list’ from incompatible pointer type
chuck.y:299: warning: passing argument 2 of ‘prepend_stmt_list’ from incompatible pointer type


Please note that the $1 and $2 arguments in the associated code (between the curly braces) should be exchanged as well, as they positionally map onto the rules on the left side. Besides, I gave just one example of a rule to be changed .. there are more that have this problem although the code example you posted is likely to run into this specific one (being a statement_list).

I do not have a ChucK development environment setup at the moment, so I can't really experiment with this myself.

It would be easier BTW to just recode your example into a for() loop Wink


Ha Ha Ha Razz for() loop ... To end the day on a smile is always nice. Sure, the example can be recoded as a for loop. But it was an example to show the error, so the for loop would have been pointless. Unless I did 1 million for loops! Dude, examples are examples, the real world cannot survive just on for loops.

Anyway, as for the grammer file, I have no idea or the time to investigate further.
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic Moderators: Kassen
Page 1 of 2 [26 Posts]
View unread posts
View new posts in the last week
Goto page: 1, 2 Next
Mark the topic unread :: View previous topic :: View next topic
 Forum index » DIY Hardware and Software » ChucK programming language
Jump to:  

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Forum with support of Syndicator RSS
Powered by phpBB © 2001, 2005 phpBB Group
Copyright © 2003 through 2009 by electro-music.com - Conditions Of Use