[Vtigercrm-developers] How to import and overwrite data without changing record IDs

Wed Sep 16 11:34:37 GMT 2020

This is what I think would be ideal:

1. In any case the duplicate records in the CRM should not be merged
2. If the operation is merge then *update* all the matching records fields,
with the field values in the CSV file if the values in the CRM record is
empty or not equal to default value for that field
3. If the operation is overwrite then *update* all the matching records
fields with the field values in CSV blindly

Since we are not merging duplicate records in CRM into one single record
anymore it would also be possible to Map the auto-generated fields.

Our team has started work on this and will share the progress soon. Please
feel free to share your thoughts.

On Tue, Sep 15, 2020 at 2:57 PM Uma S <uma.s at vtiger.com> wrote:

> Hi Martin,
>
> That's a better idea, We need to improvise on the existing options(Merge &
> overwrite) to import field values in the field mapping phase for blank
> field values and add support to introduce unique field in field mapping to
> resolve this issue.
>
> Thanks! for the insight Martin :) Updated your observation on the issue
> <https://code.vtiger.com/vtiger/vtigercrm/issues/1483>.
>
>
> On Tue, Sep 15, 2020 at 12:54 PM Martin Allen <martin.allen at clystnet.com>
> wrote:
>
>> I don't think you need to make the User choose between 'new' or 'update'
>> as then they'd potentially have to do two imports instead of being able to
>> do both in 1 step - just need to ensure that the 'update' or 'merge' method
>> is very explicit in how it works (and does work correctly).
>> Adding the ability to select autoincrement fields as the duplicate check
>> (being the only true Unique ID field) is key to making this work properly
>> though. Being the only non-editable field, it means you can then update ANY
>> existing field even if it's a field you would previously have used as a
>> duplicate check field! (Which you can currently do!)
>>
>> When the record already exists maybe, allowing the User to specify per
>> field (in the Field Mapping section) whether to always use the supplied
>> value (overwrite), or whichever one is not blank (merge), and if both not
>> blank then whether to use the supplied value or the existing value?
>>
>> Martin Allen
>>
>>
>> *01392 248692 - Main Office01392 690659 - Direct Line*
>>
>> *[image: signature2]*
>>
>> Have you visited our website recently? http://www.clystnet.com
>>
>>
>> The information in this email is confidential If you are not the intended
>> recipient, you must not read or use that information. This email and any
>> attachments are believed to be virus free however no responsibility is
>> accepted by Clystnet for any loss or damage arising in any way from receipt
>> or use thereof. Clystnet Ltd (company reg number 7164503) is based at
>> Silverdown Park, Fair Oak Close, Clyst Honiton, EX5 2UX
>>
>>
>> On Tue, 15 Sep 2020 at 07:37, Uma S <uma.s at vtiger.com> wrote:
>>
>>> Thanks! for the Idea Ruben. Updated the details on the issue
>>> <https://code.vtiger.com/vtiger/vtigercrm/issues/1483>.
>>>
>>> On Mon, Sep 14, 2020 at 9:38 PM Rubén A. Estrada Orozco <
>>> rulotec1 at gmail.com> wrote:
>>>
>>>> I agree, there should be an "update" option which only updates fields
>>>> in existing records but that doesn't create new records. And the merge
>>>> option that currently exists should create records if they do not exist and
>>>> update them if they do (here we would just need to fix the existing bugs).
>>>> Saludos
>>>>
>>>> Rubén
>>>>
>>>>
>>>> On Sat, Sep 12, 2020 at 12:12 PM Uma S <uma.s at vtiger.com> wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> I think we should enhance the import functionality, So that systems
>>>>> should first analyze the user's need for import on either to update the
>>>>> existing records or to replace the record (i.e., delete existing and
>>>>> creating new one). Based on selected choice consequent steps should be
>>>>> taken on records.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 11, 2020 at 10:42 AM nilay khatri <nilay.spartan at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> And thanks for the input everyone.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 11, 2020 at 12:59 AM Rubén A. Estrada Orozco <
>>>>>> rulotec1 at gmail.com> wrote:
>>>>>>
>>>>>>> Naily, I've seen similar issues. When using the merge feature for
>>>>>>> example, we noticed that if you have a phone column in your csv and its
>>>>>>> empty and if the record in vtgier has already a phone, it gets deleted.
>>>>>>> What you would expect from a merge is that it would keep the non-empty
>>>>>>> field.
>>>>>>>
>>>>>>> I've opened these issues:
>>>>>>> * merge function deletes fields:
>>>>>>> https://code.vtiger.com/vtiger/vtigercrm/issues/1483
>>>>>>> * add record number field for updating purposes:
>>>>>>> https://code.vtiger.com/vtiger/vtigercrm/issues/1484
>>>>>>>
>>>>>>> Saludos
>>>>>>>
>>>>>>> Rubén
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 10, 2020 at 9:36 AM Martin Allen <
>>>>>>> martin.allen at clystnet.com> wrote:
>>>>>>>
>>>>>>>> We've actually written our own import scripts that will read a CSV
>>>>>>>> file and will update / create records via Webservice, allowing you to do a
>>>>>>>> check for an existing record using whatever fields you want first (and can
>>>>>>>> identify and handle if it returns multiple records) and then update only
>>>>>>>> the fields you want to.
>>>>>>>>
>>>>>>>> Martin Allen
>>>>>>>>
>>>>>>>>
>>>>>>>> *01392 248692 - Main Office01392 690659 - Direct Line*
>>>>>>>>
>>>>>>>> *[image: signature2]*
>>>>>>>>
>>>>>>>> Have you visited our website recently? http://www.clystnet.com
>>>>>>>>
>>>>>>>>
>>>>>>>> The information in this email is confidential If you are not the
>>>>>>>> intended recipient, you must not read or use that information. This email
>>>>>>>> and any attachments are believed to be virus free however no responsibility
>>>>>>>> is accepted by Clystnet for any loss or damage arising in any way from
>>>>>>>> receipt or use thereof. Clystnet Ltd (company reg number 7164503) is based
>>>>>>>> at Silverdown Park, Fair Oak Close, Clyst Honiton, EX5 2UX
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 10 Sep 2020 at 15:28, Martin Allen <
>>>>>>>> martin.allen at clystnet.com> wrote:
>>>>>>>>
>>>>>>>>> I believe that in vTiger hosted version it is possible to have the
>>>>>>>>> Lead Number (or any Auto Increment field in any other module - UI Type 4)
>>>>>>>>> field included in an Import which can then be used for checking for
>>>>>>>>> duplicates (obviously this field value is not imported, only used for
>>>>>>>>> duplicate checking).
>>>>>>>>>
>>>>>>>>> Would be good if this was available in the open source version, as
>>>>>>>>> it would make importing data (to update records rather than creating new
>>>>>>>>> ones) much simpler and safer, as any other field cannot be guaranteed to be
>>>>>>>>> unique, and sometimes the field you want to update are the fields that you
>>>>>>>>> would typically use to identify duplicates (i.e. First Name, Last Name,
>>>>>>>>> Email Address for Contacts)
>>>>>>>>>
>>>>>>>>> Martin Allen
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *01392 248692 - Main Office01392 690659 - Direct Line*
>>>>>>>>>
>>>>>>>>> *[image: signature2]*
>>>>>>>>>
>>>>>>>>> Have you visited our website recently? http://www.clystnet.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The information in this email is confidential If you are not the
>>>>>>>>> intended recipient, you must not read or use that information. This email
>>>>>>>>> and any attachments are believed to be virus free however no responsibility
>>>>>>>>> is accepted by Clystnet for any loss or damage arising in any way from
>>>>>>>>> receipt or use thereof. Clystnet Ltd (company reg number 7164503) is based
>>>>>>>>> at Silverdown Park, Fair Oak Close, Clyst Honiton, EX5 2UX
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, 10 Sep 2020 at 13:22, nilay khatri <
>>>>>>>>> nilay.spartan at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> This is a very common challenge, users want to update certain
>>>>>>>>>> data of records in CRM by importing a CSV.
>>>>>>>>>>
>>>>>>>>>> Let us say Leads, we want to update Lead Status, so the CSV sheet
>>>>>>>>>> contains 2 columns Primary Email and Status.
>>>>>>>>>>
>>>>>>>>>> If we import the CSV and set the duplicate check criteria in step
>>>>>>>>>> 2 on Primary Email and import following things would happen:
>>>>>>>>>>
>>>>>>>>>> - quite possible all the other data in fields will be wiped out
>>>>>>>>>> - if the system contains more than 1 Lead with same email id, it
>>>>>>>>>> would delete them, keeping only 1
>>>>>>>>>> - the record IDs will change
>>>>>>>>>>
>>>>>>>>>> If I talk about contact module that is full of surprises, contact
>>>>>>>>>> Id is available to select in the duplicate selection criteria, but not
>>>>>>>>>> available in the field mapping!
>>>>>>>>>>
>>>>>>>>>> --------------------------------------
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> In the end we want to achieve a very simple thing, import a List
>>>>>>>>>> of Leads and update respective status without changing record Ids or
>>>>>>>>>> deleting any other matching record in system.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> http://www.vtiger.com/
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>> http://www.vtiger.com/
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> http://www.vtiger.com/
>>>>>>
>>>>>> _______________________________________________
>>>>>> http://www.vtiger.com/
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> With
>>>>> Best Regards
>>>>> Uma.S
>>>>> Vtiger Team
>>>>> _______________________________________________
>>>>> http://www.vtiger.com/
>>>>
>>>> _______________________________________________
>>>> http://www.vtiger.com/
>>>
>>>
>>>
>>> --
>>> With
>>> Best Regards
>>> Uma.S
>>> Vtiger Team
>>> _______________________________________________
>>> http://www.vtiger.com/
>>
>> _______________________________________________
>> http://www.vtiger.com/
>
>
>
> --
> With
> Best Regards
> Uma.S
> Vtiger Team
> _______________________________________________
> http://www.vtiger.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.vtigercrm.com/pipermail/vtigercrm-developers/attachments/20200916/1b4d04a9/attachment-0001.html>