[Vtigercrm-developers] Importing records

[lajeesh k]

I have fixed this issue
added
if($this->getModule() == 'Import' && $this->isRunning()) return $runnable;
in vtlib/Vtiger/Cron.php line around 158

function now look like this

    function isRunnable() {
        $runnable = false;

        if (!$this->isDisabled()) {
            if($this->getModule() == 'Import' && $this->isRunning()) return
$runnable;
            // Take care of last time (end - on success, start - if
timedout)
            // Take care to start the cron im
            $lastTime = ($this->getLastStart() > 0) ? $this->getLastStart()
: $this->getLastEnd();
            $elapsedTime = time() - $lastTime;
            $runnable = ($elapsedTime >= ($this->getFrequency()-60));
        }
        return $runnable;
    }

Now it will run only one import at a time so no duplicates

my current issue is the import is too slow, but it not due to this change
its imported only 50k records in 8 hours
I have added indexing to firstname and lastname, that may be the reason







Regards,
Lajeesh

On Sat, Sep 26, 2015 at 3:23 PM, Simone Travaglini <
simonetravaglini at gmail.com> wrote:

> Hi lajeesh,
> we resolved removing cronjob. Then we used vtigercron.sh from bash (and we
> have set scheduler to 45 min) and we were able to import also 200k record
> in one shot with no duplicate. With our VPS for 200k record we need 1h.
> Consider that in old shared osting for 100k record we need 10h...
>
> 2015-09-26 10:24 GMT+02:00 lajeesh k <lajeeshk at gmail.com>:
>
>> Hi,
>>
>> I too facing same duplicate issue. I was importing 5 lacks or records and
>> it takes more than 10 hours to finish. It creating lots of duplicate
>> records.
>>
>> some one mentioned that cron frequncy is the issue. But if we run this
>> vtigercon.sh while this import is running it is giving the warning that the
>> import is already running.
>>
>> @Alan can you share your server api script with us, so that i can save
>> time for writing the script
>>
>>
>>
>> Regards,
>> Lajeesh
>>
>> On Sat, Sep 12, 2015 at 12:16 PM, Simone Travaglini <
>> simonetravaglini at gmail.com> wrote:
>>
>>> Hi new test on this issue: at this moment our empirical test say that it
>>> depends on scheduler... we have set scheduler with longer time than 15 min
>>> (to 45 min) and imported succesfully 100k lead for 2 time. I update you on
>>> next tests, becouse we need to import 2m of lead! ;)
>>>
>>> 2015-08-10 11:29 GMT+02:00 Błażej Pabiszczak <b.pabiszczak at yetiforce.com
>>> >:
>>>
>>>> This happened to us several times when the clients used proxy. We
>>>> solved that problem by allowing the import to take place directly from the
>>>> server console instead from the browser.
>>>> ---
>>>> Z poważaniem / Regards
>>>>
>>>> *Błażej Pabiszczak*
>>>> *Chief Executive Officer*
>>>> M: +48.884999123
>>>> E: b.pabiszczak at yetiforce.com
>>>> ------------------------------
>>>>
>>>>
>>>>
>>>> W dniu 2015-07-28 07:58, Alan Lord (News) napisał(a):
>>>>
>>>> On 28/⁠07/⁠15 00:23, Hamono, Chris (DPC) wrote:
>>>>
>>>>
>>>> Have you experienced the duplication issue people are complaining about?
>>>>
>>>>
>>>> No -⁠ it's not something I have ever noticed.
>>>>
>>>> Alan
>>>>
>>>>
>>>> _______________________________________________
>>>> http://www.vtiger.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Simone Travaglini
>>> 328 5499846
>>> Linkedin: Simone Travaglini
>>>
>>>
>>> Rispetta l'ambiente: non stampare questa mail se non ti è veramente
>>> necessario!
>>>
>>> _______________________________________________
>>> http://www.vtiger.com/
>>>
>>
>>
>> _______________________________________________
>> http://www.vtiger.com/
>>
>
>
>
> --
> Simone Travaglini
> 328 5499846
> Linkedin: Simone Travaglini
>
>
> Rispetta l'ambiente: non stampare questa mail se non ti è veramente
> necessario!
>
> _______________________________________________
> http://www.vtiger.com/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.vtigercrm.com/pipermail/vtigercrm-developers/attachments/20150927/a9d9df56/attachment-0001.html>