memory - When should I avoid using `seq` in Clojure? -



memory - When should I avoid using `seq` in Clojure? -

in this thread, learned keeping reference seq on big collection prevent entire collection beingness garbage-collected.

first, thread 2009. still true in "modern" clojure (v1.4.0 or v1.5.0)?

second, issue apply lazy sequences? example, (def s (drop 999 (seq (range 1000)))) allow garbage collector retire first 999 elements of sequence?

lastly, there way around issue big collections? in other words, if had vector of, say, 10 1000000 elements, consume vector in such way consumed parts garbage collected? if had hashmap 10 1000000 elements?

the reason inquire i'm operating on big info sets, , having more careful not retain references objects, objects don't need can garbage collected. is, i'm encountering java.lang.outofmemoryerror: gc overhead limit exceeded error in cases.

it case if "hold onto head" of sequence clojure forced maintain in memory. doesn't have choice: still keeping reference it.

however "gc overhead limit reached" isn't same out of memory error - it's more sign running fictitious workload creating , discarding objects fast tricking gc thinking overloaded.

see:

gc overhead limit exceeded

if set actual workload on items beingness processed, suspect see error won't happen more. can process lazy sequences larger available memory in case.

concrete collections vectors , hashmaps different matter however: these not lazy, must held in memory. if have datasets larger memory options include:

use lazy sequences , don't hold onto head use specialised collections back upwards lazy loading (datomic uses structures believe) treat info event stream (using storm) write custom code partition info chunks , process them 1 @ time.

memory memory-management clojure garbage-collection

Comments

Post a Comment

Popular posts from this blog

web services - java.lang.NoClassDefFoundError: Could not initialize class net.sf.cglib.proxy.Enhancer -

web services - java.lang.NoClassDefFoundError: Could not initialize class net.sf.cglib.proxy.Enhancer - when trying request rest service @ time of response getting exception.. feb 21, 2013 2:34:49 pm com.sun.jersey.spi.container.containerresponse mapmappablecontainerexception severe: exception contained within mappablecontainerexception not mapped response, re-throwing http container java.lang.noclassdeffounderror: not initialize class net.sf.cglib.proxy.enhancer @ org.hibernate.proxy.pojo.cglib.cgliblazyinitializer.getproxyfactory(cgliblazyinitializer.java:117) @ org.hibernate.proxy.pojo.cglib.cglibproxyfactory.postinstantiate(cglibproxyfactory.java:43) @ org.hibernate.tuple.entity.pojoentitytuplizer.buildproxyfactory(pojoentitytuplizer.java:162) @ org.hibernate.tuple.entity.abstractentitytuplizer.<init>(abstractentitytuplizer.java:135) @ org.hibernate.tuple.entity.pojoentitytuplizer.<init>(pojoentitytuplizer.java:55) @ org.hibernate.tuple.e...
Read more

Accessing MATLAB's unicode strings from C -

Accessing MATLAB's unicode strings from C - how can access underlying unicode info of matlab strings through matlab engine or mex c interfaces? here's example. let's set unicode characters in utf-8 encoded file test.txt, read as fid=fopen('test.txt','r','l','utf-8'); s=fscanf(fid, '%s') in matlab. now if first feature('defaultcharacterset', 'utf-8') , c engevalstring(ep, "s") , output text file utf-8. proves matlab stores unicode internally. if mxarraytostring(enggetvariable(ep, "s")) , unicode2native(s, 'latin-1') give me in matlab: non-latin-1 characters replaced character code 26. need getting access underlying unicode info c string in unicode format (utf-8, utf-16, etc.), , preserving non-latin-1 characters. is possible? my platform os x, matlab r2012b. addendum: documentation explicitly states "[mxarraytostring()] supports multibyte encoded characters...
Read more

javascript - mongodb won't find my schema method in nested container -

javascript - mongodb won't find my schema method in nested container - i trying access method of schema stored within mixed container. here situation : have cases model can many different things, have schema each of these things stored in "casecontent" mixed property. var caseschema = mongoose.schema({ casecontent : {}, object : {type:string, default : "null"}, collision : {type : boolean, default : false} }); the casecontent property filled model of 1 of schemas, 1 exemple : var treeschema = new mongoose.schema({ applecount : {type : number, default : 3} }); treeschema.methods.dostuff = function (data) { console.log('hey, listen'); homecoming true; }; then, want utilize method of schema original container : caseschema.methods.dostuff = function (data) { if (this.casecontent.dostuff !== undefined) { this.casecontent.dostuff(); console.log('it worked'); } else { ...
Read more