Author |
Message |
ultrasine
Joined: Aug 31, 2006 Posts: 16 Location: Europe
|
Posted: Thu Jun 05, 2008 1:08 pm Post subject:
Memory Exhausted Subject description: Input file with large number of lines or using big arrays causes crash |
|
|
Hey everyone, hope you all are chilling.
I just started messing about with chuck again after a long break but straight away I started crashing it with the following example error output:
[test.ck]:line(3).char(19991): memory exhausted
Having large number of lines (>5000?) or big arrays causes this.
I shan't post the input files, apart from top secret chuck norris kick butt code, I don't want to be held responsible for other peoples computers melting.
Anyone know what the limits are and how I can break these? I'm running chuck.oss 1.2.0.7 on a 256MB debian laptop.
Aye. |
|
Back to top
|
|
|
Inventor
Stream Operator
Joined: Oct 13, 2007 Posts: 6221 Location: near Austin, Tx, USA
Audio files: 267
|
Posted: Tue Jun 10, 2008 2:32 pm Post subject:
|
|
|
Hi, sorry I didn't see your post earlier. The error message indicates that you have almost 20,000 characters on a single line of your code. I can only guess, but perhaps you are running into a line length limit of some sort. could you break up that line with carriage returns? Just a guess. Good luck! _________________ "Let's make noise for peace." - Kijjaz |
|
Back to top
|
|
|
Kassen
Janitor
Joined: Jul 06, 2004 Posts: 7678 Location: The Hague, NL
G2 patch files: 3
|
Posted: Tue Jun 10, 2008 5:52 pm Post subject:
|
|
|
Erm, 20K characters on a single line? How do you edit or read that at all? Something is going wrong there. 5000 lines shouldn't be a issue but 20 thousand characters on a line is definitely unusual. I don't think that should inherently be cause for a crash either... but it is kinda extreme, I never heard of such a thing. Could you perhaps explain a little about *why* you need or like such lines?
I don't mind a molten computer or two, BTW, could you condense the code to the smallest possible construct that still crashes?
Oh, and 1.2.0.7 is quite old. Regardless of whether it fixes the issue I'd upgrade to the latest greatest (1.2.1.1b ) _________________ Kassen |
|
Back to top
|
|
|
ultrasine
Joined: Aug 31, 2006 Posts: 16 Location: Europe
|
|
Back to top
|
|
|
Inventor
Stream Operator
Joined: Oct 13, 2007 Posts: 6221 Location: near Austin, Tx, USA
Audio files: 267
|
Posted: Wed Jun 11, 2008 3:54 am Post subject:
|
|
|
Well, before you move on let's give folks here on the forum and also on the DEV team a chance to look into the problem. ChucK is purposely coded with low performance in some ways so that it can be so free and effective in other ways. We usually run into CPU limitations by spinning around on samp => now or somesuch thing. This is a new limitation that you've found as far as I know. I'm not the best one to help, but perhaps we can post to the ChucK users group and see what they say. Even ChucK Norris has his limits, Grasshopper! (Grin) _________________ "Let's make noise for peace." - Kijjaz |
|
Back to top
|
|
|
ultrasine
Joined: Aug 31, 2006 Posts: 16 Location: Europe
|
Posted: Wed Jun 11, 2008 4:06 am Post subject:
|
|
|
Inventor wrote: | Well, before you move on let's give folks here on the forum and also on the DEV team a chance to look into the problem. ChucK is purposely coded with low performance in some ways so that it can be so free and effective in other ways. We usually run into CPU limitations by spinning around on samp => now or somesuch thing. This is a new limitation that you've found as far as I know. I'm not the best one to help, but perhaps we can post to the ChucK users group and see what they say. Even ChucK Norris has his limits, Grasshopper! (Grin) |
Sorry, ignore my last statement. I'm not with it today. If you can post to the users group on my behalf it would be appreciated.
Don't worry I'll be sticking with chuck for some time yet. Even tho I know his limits he is always ready and willing for some serious kick butt action. |
|
Back to top
|
|
|
Kassen
Janitor
Joined: Jul 06, 2004 Posts: 7678 Location: The Hague, NL
G2 patch files: 3
|
Posted: Wed Jun 11, 2008 4:56 am Post subject:
|
|
|
Weird.
Apparently 5000 lines *is* a issue. My desktop gives the exact same error you get, both in the MiniAudicle and in command-line ChucK.
Weird.
Why do you think this stack parameter is related to it? _________________ Kassen |
|
Back to top
|
|
|
Inventor
Stream Operator
Joined: Oct 13, 2007 Posts: 6221 Location: near Austin, Tx, USA
Audio files: 267
|
Posted: Wed Jun 11, 2008 5:23 am Post subject:
|
|
|
OK ultrasine, I sent a message to the ChucK users group list for you, should have an answer in a day or two, maybe sooner. In the mean time, don't smash any bricks in frustration! _________________ "Let's make noise for peace." - Kijjaz |
|
Back to top
|
|
|
Kassen
Janitor
Joined: Jul 06, 2004 Posts: 7678 Location: The Hague, NL
G2 patch files: 3
|
Posted: Wed Jun 11, 2008 5:49 am Post subject:
|
|
|
Maybe I'm missing several important aspects to this.... again.... but erm, why wouldn't Ultrasine sign up to the list himself and post about the bug? _________________ Kassen |
|
Back to top
|
|
|
blue hell
Site Admin
Joined: Apr 03, 2004 Posts: 24079 Location: The Netherlands, Enschede
Audio files: 278
G2 patch files: 320
|
Posted: Wed Jun 11, 2008 6:24 am Post subject:
|
|
|
ultrasine wrote: | In the file src/chuck.tab.c we have a "YYMAXDEPTH 10000" with comments:
/* YYMAXDEPTH -- maximum size the stacks can grow to (effective only
if the built-in stack extension method is used).
Do not make this value too large; the results are undefined if
YYSTACK_ALLOC_MAXIMUM < YYSTACK_BYTES (YYMAXDEPTH)
evaluated with infinite-precision integer arithmetic. */
|
YYMAXDEPTH is likely the maximum depth of the parser stack, as this is the standard name for that used by Yacc / Bison (The YY prefixe comes from Yacc originally, but nowadays mostly Bison is used). I don't know if Lex / Flex and Yacc / Bison are invoked from the standard make (I think not) but when they are the value used in the C file will not be used at all, you'd have to change it in the .y or .l file (or possibly an include file) where it's defined.
The parser stack depth limits the complexity (nesting level) of the language constructs that can be parsed, it does not have a direct influence on the acceptable source code size or the size of data structures that can be used. _________________ Jan
also .. could someone please turn down the thermostat a bit.
|
|
Back to top
|
|
|
ultrasine
Joined: Aug 31, 2006 Posts: 16 Location: Europe
|
Posted: Wed Jun 11, 2008 6:27 am Post subject:
|
|
|
Kassen wrote: | Maybe I'm missing several important aspects to this.... again.... but erm, why wouldn't Ultrasine sign up to the list himself and post about the bug? |
New url, username and password to remember. Also for some strange reason I cannot reach lists.cs.princeton.edu or the chuck site. OK , I'm lazy. |
|
Back to top
|
|
|
Inventor
Stream Operator
Joined: Oct 13, 2007 Posts: 6221 Location: near Austin, Tx, USA
Audio files: 267
|
Posted: Wed Jun 11, 2008 6:44 am Post subject:
|
|
|
ultrasine wrote: | Kassen wrote: | Maybe I'm missing several important aspects to this.... again.... but erm, why wouldn't Ultrasine sign up to the list himself and post about the bug? |
New url, username and password to remember. Also for some strange reason I cannot reach lists.cs.princeton.edu or the chuck site. OK , I'm lazy. |
I just figured I'd save him the trouble for just one question. I know I get frustrated with all the passwords and such that I keep up with these days. He seemed to be iffy about commitment to being a ChucKist just yet, so why not do the guy a favor, eh? _________________ "Let's make noise for peace." - Kijjaz |
|
Back to top
|
|
|
Kassen
Janitor
Joined: Jul 06, 2004 Posts: 7678 Location: The Hague, NL
G2 patch files: 3
|
Posted: Wed Jun 11, 2008 6:59 am Post subject:
|
|
|
Check.
I don't think I even know my own password for the mailing list anymore, I think there was a trick to getting the list server to send you your info on request. _________________ Kassen |
|
Back to top
|
|
|
chuckles
Joined: Apr 02, 2007 Posts: 72 Location: San Diego, California
|
Posted: Wed Jun 11, 2008 9:21 am Post subject:
I also can't get to http://chuck.cs.princeton.edu |
|
|
ultrasine wrote: | Kassen wrote: | Maybe I'm missing several important aspects to this.... again.... but erm, why wouldn't Ultrasine sign up to the list himself and post about the bug? |
New url, username and password to remember. Also for some strange reason I cannot reach lists.cs.princeton.edu or the chuck site. OK , I'm lazy. |
I've been having the same problem the last couple days: attempting to access chuck.cs.princeton.edu fails. Presumable the sysadmins at Princeton will fix this some day... |
|
Back to top
|
|
|
ultrasine
Joined: Aug 31, 2006 Posts: 16 Location: Europe
|
Posted: Wed Jun 11, 2008 10:32 am Post subject:
Re: I also can't get to http://chuck.cs.princeton.edu |
|
|
chuckles wrote: | ultrasine wrote: | Kassen wrote: | Maybe I'm missing several important aspects to this.... again.... but erm, why wouldn't Ultrasine sign up to the list himself and post about the bug? |
New url, username and password to remember. Also for some strange reason I cannot reach lists.cs.princeton.edu or the chuck site. OK , I'm lazy. |
I've been having the same problem the last couple days: attempting to access chuck.cs.princeton.edu fails. Presumable the sysadmins at Princeton will fix this some day... |
Hard to believe but cs.princeton.edu is down. |
|
Back to top
|
|
|
ultrasine
Joined: Aug 31, 2006 Posts: 16 Location: Europe
|
Posted: Wed Jun 11, 2008 10:52 am Post subject:
|
|
|
OK ladies, I think I've found the hack to increase this limit we have encountered.
You just need to increase the YYMAXDEPTH in src/chuck.tab.c and recompile.
I have tested with increasing YYMAXDEPTH and ever bigger inputs and can verify there is a direct relation with the 'memory exhausted' error.
I went all the upto 0.5 million lines of input (YYMAXDEPTH=1000001) before chuck started to take over the machine (start using hard drive as memory?) and I had to control-c after 4 hours without audio output. At least chuck did a clean exit.
My system limits are the following (thought stack size would be unlimited but no... strange?):
ultrasine@xenon:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Looks like I have the biggest bad ass Chuck than all of ya! |
|
Back to top
|
|
|
chuckles
Joined: Apr 02, 2007 Posts: 72 Location: San Diego, California
|
Posted: Wed Jun 11, 2008 12:30 pm Post subject:
Re: I also can't get to http://chuck.cs.princeton.edu |
|
|
ultrasine wrote: | Hard to believe but cs.princeton.edu is down. |
That's got to be the very defintion of "ironic". |
|
Back to top
|
|
|
Inventor
Stream Operator
Joined: Oct 13, 2007 Posts: 6221 Location: near Austin, Tx, USA
Audio files: 267
|
Posted: Wed Jun 11, 2008 4:36 pm Post subject:
|
|
|
ultrasine wrote: | Looks like I have the biggest bad ass Chuck than all of ya! |
hahaha just what we need, a new ChucKist with a chip on his shoulder! Rock on dude! _________________ "Let's make noise for peace." - Kijjaz |
|
Back to top
|
|
|
ultrasine
Joined: Aug 31, 2006 Posts: 16 Location: Europe
|
Posted: Thu Jun 12, 2008 12:56 am Post subject:
|
|
|
Alrighty, I think I'll finish this thread by saying that it would be cool if this magic number YYMAXDEPTH could be set via an optional command line parameter.
Cheers everyone! |
|
Back to top
|
|
|
blue hell
Site Admin
Joined: Apr 03, 2004 Posts: 24079 Location: The Netherlands, Enschede
Audio files: 278
G2 patch files: 320
|
Posted: Thu Jun 12, 2008 4:27 am Post subject:
|
|
|
I looked into this. The real problem is not YYMAXDEPTH but rather grammar rules like :
Code: |
statement_list
: statement { $$ = new_stmt_list( $1, EM_lineNum ); }
| statement statement_list { $$ = prepend_stmt_list( $1, $2, EM_lineNum ); }
;
|
in chuck.y.
This is right recursion which lets the stack grow without bounds in a linear fashion with the length of the statement_list.
Formulating this with left recursion, as in :
Code: |
statement_list
: statement { $$ = new_stmt_list( $1, EM_lineNum ); }
| statement_list statement { $$ = prepend_stmt_list( $2, $1, EM_lineNum ); }
;
|
would work with a bounded stack.
Also see the Bison manual for this, ( recursive rules : http://www.cs.utah.edu/dept/old/texinfo/bison/bison.html#SEC42 ). _________________ Jan
also .. could someone please turn down the thermostat a bit.
|
|
Back to top
|
|
|
Inventor
Stream Operator
Joined: Oct 13, 2007 Posts: 6221 Location: near Austin, Tx, USA
Audio files: 267
|
Posted: Thu Jun 12, 2008 6:31 am Post subject:
|
|
|
Wow, Jan, that's a heck of a brainy thing you did there! _________________ "Let's make noise for peace." - Kijjaz |
|
Back to top
|
|
|
ultrasine
Joined: Aug 31, 2006 Posts: 16 Location: Europe
|
Posted: Thu Jun 12, 2008 7:05 am Post subject:
|
|
|
Hi Jan,
I changed the code in the file src/chuck.y to the second block of code you posted because that was the only thing to do. When I tried compiling I got:
chuck.y: In function ‘yyparse’:
chuck.y:299: warning: passing argument 1 of ‘prepend_stmt_list’ from incompatible pointer type
chuck.y:299: warning: passing argument 2 of ‘prepend_stmt_list’ from incompatible pointer type
I have never worked with grammer parsers so am clueless. Heck, I also avoid compiled languages so you can see this is not much fun for me.
All I want is chuck not to barf on large inputs. I know how to do that by changing YYMAXDEPTH and compiling on linux.
If you can supply the file chuck.y which makes chuck more flexible in handling large inputs then please do so. |
|
Back to top
|
|
|
blue hell
Site Admin
Joined: Apr 03, 2004 Posts: 24079 Location: The Netherlands, Enschede
Audio files: 278
G2 patch files: 320
|
Posted: Thu Jun 12, 2008 7:14 am Post subject:
|
|
|
This is just one of the "standard" problems with Yacc / Bison. The default value for YYMAXDEPTH in Yacc used to always be 150 which with a correct grammar nearly never gets exhausted.
For Bison there (apparently) was some dynamics added in that the parser stack starts at a depth of 200 and it can grow dynamically to 10,000 (by default) - so running out of that is just absurd. Apart from that the re-allocations will cost time, especially when you start to run into virtual memory ... don't know though what the size of the stack items is, well it's not larger than a C double type but there are associated objects as well (those are probably on the heap).
There may BTW be other means to solve the problem by adding tactical %left precedence operators. Yacc is a bit of a black art, and it can cost quite some time to get things right for a complex grammar like ChucK has. Anyway, usually left and right recursion can usually just be changed into each other without introducing other problems - but please note that the $x arguments in the associated code (between the curly brackets "{" "}") need to be changed as well. _________________ Jan
also .. could someone please turn down the thermostat a bit.
|
|
Back to top
|
|
|
blue hell
Site Admin
Joined: Apr 03, 2004 Posts: 24079 Location: The Netherlands, Enschede
Audio files: 278
G2 patch files: 320
|
Posted: Thu Jun 12, 2008 7:22 am Post subject:
|
|
|
ultrasine wrote: | Hi Jan,
I changed the code in the file src/chuck.y to the second block of code you posted because that was the only thing to do. When I tried compiling I got:
chuck.y: In function ‘yyparse’:
chuck.y:299: warning: passing argument 1 of ‘prepend_stmt_list’ from incompatible pointer type
chuck.y:299: warning: passing argument 2 of ‘prepend_stmt_list’ from incompatible pointer type |
Please note that the $1 and $2 arguments in the associated code (between the curly braces) should be exchanged as well, as they positionally map onto the rules on the left side. Besides, I gave just one example of a rule to be changed .. there are more that have this problem although the code example you posted is likely to run into this specific one (being a statement_list).
I do not have a ChucK development environment setup at the moment, so I can't really experiment with this myself.
It would be easier BTW to just recode your example into a for() loop _________________ Jan
also .. could someone please turn down the thermostat a bit.
|
|
Back to top
|
|
|
ultrasine
Joined: Aug 31, 2006 Posts: 16 Location: Europe
|
Posted: Thu Jun 12, 2008 11:21 am Post subject:
|
|
|
Blue Hell wrote: | ultrasine wrote: | Hi Jan,
I changed the code in the file src/chuck.y to the second block of code you posted because that was the only thing to do. When I tried compiling I got:
chuck.y: In function ‘yyparse’:
chuck.y:299: warning: passing argument 1 of ‘prepend_stmt_list’ from incompatible pointer type
chuck.y:299: warning: passing argument 2 of ‘prepend_stmt_list’ from incompatible pointer type |
Please note that the $1 and $2 arguments in the associated code (between the curly braces) should be exchanged as well, as they positionally map onto the rules on the left side. Besides, I gave just one example of a rule to be changed .. there are more that have this problem although the code example you posted is likely to run into this specific one (being a statement_list).
I do not have a ChucK development environment setup at the moment, so I can't really experiment with this myself.
It would be easier BTW to just recode your example into a for() loop |
Ha Ha Ha for() loop ... To end the day on a smile is always nice. Sure, the example can be recoded as a for loop. But it was an example to show the error, so the for loop would have been pointless. Unless I did 1 million for loops! Dude, examples are examples, the real world cannot survive just on for loops.
Anyway, as for the grammer file, I have no idea or the time to investigate further. |
|
Back to top
|
|
|
|