I am trying to run a deletion framework that I created which has a very huge amount of data. While trying to run Corb2, I get the below warning and sometimes an error because of insufficient memory as well.
WARNING: Slow receive! Consider increasing max heap size and using -XX:+UseConcMarkSweepGC
Error:
Sep 23, 2021 1:06:23 PM com.marklogic.developer.corb.QueryUrisLoader closeRequestAndSession
INFO: closing uris session
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at com.marklogic.io.IOHelper.literalStringFromReader(IOHelper.java:50)
at com.marklogic.io.IOHelper.literalStringFromStream(IOHelper.java:66)
at com.marklogic.http.MultipartBuffer.getBodyAsString(MultipartBuffer.java:75)
at com.marklogic.xcc.impl.AbstractResultSequence.instantiateResultItem(AbstractResultSequence.java:138)
at com.marklogic.xcc.impl.StreamingResultSequence.next(StreamingResultSequence.java:147)
at com.marklogic.xcc.impl.StreamingResultSequence.next(StreamingResultSequence.java:166)
at com.marklogic.developer.corb.QueryUrisLoader.populateQueue(QueryUrisLoader.java:212)
at com.marklogic.developer.corb.QueryUrisLoader.createAndPopulateQueue(QueryUrisLoader.java:201)
at com.marklogic.developer.corb.QueryUrisLoader.open(QueryUrisLoader.java:122)
at com.marklogic.developer.corb.Manager.runUrisLoader(Manager.java:809)
at com.marklogic.developer.corb.Manager.populateQueue(Manager.java:830)
at com.marklogic.developer.corb.Manager.run(Manager.java:578)
at com.marklogic.developer.corb.Manager.main(Manager.java:138)
Has anybody used it (especially with JavaScript) to be able to work on larger volumes of data?
Rather than trying to find the right size of memory to allocate to hold all of the URIs in-memory, enable the DISK-QUEUE option to allow for the URIs queue to spill to disk in a temporary file that is consumed, instead of all being held in-memory.
You can optionally configure the path for the temp files to be written with DISK-QUEUE-TEMP-DIR. If not specified, it will use the TEMP-DIR, or java.io.tmpdir.