[Vtigercrm-developers] batch update task eats up memory

Sukhdev Mohan s.mohan at myti.it
Fri Oct 23 06:16:38 GMT 2020


I had a similar issue. The problem is the vtiger API call, they add 8mb
circa at each call.

The solution for me was process before in batch of 5k (this is the max it
could process without existing in my case, try fine tune it), then dump the
batch to a JSON. After processing I created a stupid script that reads the
JSON and executes. I've serialised it, the main script launched multiple
instances of executor script in parallel.


Another approach we took was pentaho data integration with which we called
the crm's API (custom) while all the data fetch and processing was done on
pentaho.

Il ven 23 ott 2020, 05:43 Rubén A. Estrada Orozco <rulotec1 at gmail.com> ha
scritto:

> Hi everyone,
>
> So I have a script that does something like this:
>
> [image: image.png]
> It gets data from an sql server database and updates accounts' fields in
> vtiger.
> It's a greatly simplified version, but the essence is that of the above
> code.
>
> I execute if from the command line. But after some time, I get an "Allowed
> memory size of 92274688 bytes exhausted"
> In this example I use only 5000 records instead of the actual ~32k I need
> to update. And I lowered PHP's memory_limit to 88MB from 1GB to be able to
> get to the error quickly.
>
> What you can see in the following screenshots is that memory usage keeps
> increasing until all available memory is exhausted. Why would that happen?
> I thought it should remain  constant. It's as if PearDatabase was eating up
> the memory.
>
> [image: image.png]
>
> [image: image.png]
>
> This is supposed to run a few times a day and it needs to be efficient.
>
> Any thoughts?
>
> Saludos
>
> Rubén
> _______________________________________________
> http://www.vtiger.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.vtigercrm.com/pipermail/vtigercrm-developers/attachments/20201023/9c16d2e8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 24753 bytes
Desc: not available
URL: <http://lists.vtigercrm.com/pipermail/vtigercrm-developers/attachments/20201023/9c16d2e8/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 118482 bytes
Desc: not available
URL: <http://lists.vtigercrm.com/pipermail/vtigercrm-developers/attachments/20201023/9c16d2e8/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 85238 bytes
Desc: not available
URL: <http://lists.vtigercrm.com/pipermail/vtigercrm-developers/attachments/20201023/9c16d2e8/attachment-0005.png>


More information about the vtigercrm-developers mailing list