[Vtigercrm-developers] How to import and overwrite data without changing record IDs

Uma S uma.s at vtiger.com
Sat Sep 19 17:20:35 GMT 2020


Hi Nilay,

In the Duplicate handling step if a user opted for merge records, Then
newly created record is retained to overwrite the data.

On Fri, Sep 18, 2020 at 7:18 AM nilay khatri <nilay.spartan at gmail.com>
wrote:

> Hi Uma,
>
> coming back on this, does on the cloud version, the record IDs change if a
> duplicate match is found?
>
> On Thu, Sep 17, 2020 at 6:16 PM nilay khatri <nilay.spartan at gmail.com>
> wrote:
>
>> Okay, so if that is the case I would ask my team to stop the work on this
>> and would wait to hear further from the Vtiger team.
>>
>> On Thu, Sep 17, 2020 at 5:04 PM Uma S <uma.s at vtiger.com> wrote:
>>
>>> Hi Alan,
>>>
>>> Yes! We took an approach to review our cloud version and see if we can
>>> port the import functionality, which resolves most of the issues we
>>> discussed over here.
>>>
>>> Right now, cloud supports 4 different options in Duplicate Handling as
>>> noted below.
>>>
>>>    - Merge & Overwrite, skip blank values from CSV - If a mapped field
>>>    has blank value in the imported file, then existing value in CRM is retained
>>>    - Merge & Overwrite, apply any blank values from CSV - If a mapped
>>>    field has blank value in the imported file, then existing value in CRM will
>>>    be removed.
>>>    - Skip Duplicates - Duplicates entries in the file will not be
>>>    imported if a matching record is already in the CRM.
>>>    - Don't check Duplicates - All record from import file are imported
>>>
>>> In addition to this unique field is supported during the field mapping
>>> step. So that import is carried out successfully.
>>>
>>>
>>> On Wed, Sep 16, 2020 at 5:32 PM Alan Lord <alanslists at gmail.com> wrote:
>>>
>>>> FWIW, I think it would be helpful if vtiger discussed how this
>>>> currently
>>>> works in vtiger Cloud. In my experience it seems to work  quite well
>>>> there.
>>>>
>>>> Al
>>>>
>>>>
>>>> On 16/09/2020 12:34, nilay khatri wrote:
>>>> > This is what I think would be ideal:
>>>> >
>>>> > 1. In any case the duplicate records in the CRM should not be merged
>>>> > 2. If the operation is merge then *update* all the matching records
>>>> > fields, with the field values in the CSV file if the values in the
>>>> CRM
>>>> > record is empty or not equal to default value for that field
>>>> > 3. If the operation is overwrite then *update* all the matching
>>>> records
>>>> > fields with the field values in CSV blindly
>>>> >
>>>> > Since we are not merging duplicate records in CRM into one single
>>>> record
>>>> > anymore it would also be possible to Map the auto-generated fields.
>>>> >
>>>> > Our team has started work on this and will share the progress soon.
>>>> > Please feel free to share your thoughts.
>>>> >
>>>> > On Tue, Sep 15, 2020 at 2:57 PM Uma S <uma.s at vtiger.com
>>>> > <mailto:uma.s at vtiger.com>> wrote:
>>>> >
>>>> >     Hi Martin,
>>>> >
>>>> >     That's a better idea, We need to improvise on the existing
>>>> >     options(Merge & overwrite) to import field values in the field
>>>> >     mapping phase for blank field values and add support to introduce
>>>> >     unique field in field mapping to resolve this issue.
>>>> >
>>>> >     Thanks! for the insight Martin :) Updated your observation on the
>>>> >     issue <https://code.vtiger.com/vtiger/vtigercrm/issues/1483>.
>>>> >
>>>> >
>>>> >     On Tue, Sep 15, 2020 at 12:54 PM Martin Allen
>>>> >     <martin.allen at clystnet.com <mailto:martin.allen at clystnet.com>>
>>>> wrote:
>>>> >
>>>> >         I don't think you need to make the User choose between 'new'
>>>> or
>>>> >         'update' as then they'd potentially have to do two imports
>>>> >         instead of being able to do both in 1 step - just need to
>>>> ensure
>>>> >         that the 'update' or 'merge' method is very explicit in how it
>>>> >         works (and does work correctly).
>>>> >         Adding the ability to select autoincrement fields as the
>>>> >         duplicate check (being the only true Unique ID field) is key
>>>> to
>>>> >         making this work properly though. Being the only non-editable
>>>> >         field, it means you can then update ANY existing field even if
>>>> >         it's a field you would previously have used as a duplicate
>>>> check
>>>> >         field! (Which you can currently do!)
>>>> >
>>>> >         When the record already exists maybe, allowing the User to
>>>> >         specify per field (in the Field Mapping section) whether to
>>>> >         always use the supplied value (overwrite), or whichever one is
>>>> >         not blank (merge), and if both not blank then whether to use
>>>> the
>>>> >         supplied value or the existing value?
>>>> >
>>>> >         Martin Allen
>>>> >
>>>> >         *01392 248692 - Main Office
>>>> >         01392 690659 - Direct Line*
>>>> >
>>>> >         *signature2*
>>>> >
>>>> >         Have you visited our website recently?
>>>> http://www.clystnet.com
>>>> >         <http://www.clystnet.com/>
>>>> >
>>>> >
>>>> >         The information in this email is confidential If you are not
>>>> the
>>>> >         intended recipient, you must not read or use that information.
>>>> >         This email and any attachments are believed to be virus free
>>>> >         however no responsibility is accepted by Clystnet for any loss
>>>> >         or damage arising in any way from receipt or use thereof.
>>>> >         Clystnet Ltd (company reg number 7164503) is based at
>>>> Silverdown
>>>> >         Park, Fair Oak Close, Clyst Honiton, EX5 2UX
>>>> >
>>>> >
>>>> >
>>>> >         On Tue, 15 Sep 2020 at 07:37, Uma S <uma.s at vtiger.com
>>>> >         <mailto:uma.s at vtiger.com>> wrote:
>>>> >
>>>> >             Thanks! for the Idea Ruben. Updated the details on the
>>>> issue
>>>> >             <https://code.vtiger.com/vtiger/vtigercrm/issues/1483>.
>>>> >
>>>> >             On Mon, Sep 14, 2020 at 9:38 PM Rubén A. Estrada Orozco
>>>> >             <rulotec1 at gmail.com <mailto:rulotec1 at gmail.com>> wrote:
>>>> >
>>>> >                 I agree, there should be an "update" option which only
>>>> >                 updates fields in existing records but that doesn't
>>>> >                 create new records. And the merge option that
>>>> currently
>>>> >                 exists should create records if they do not exist and
>>>> >                 update them if they do (here we would just need to fix
>>>> >                 the existing bugs).
>>>> >                 Saludos
>>>> >
>>>> >                 Rubén
>>>> >
>>>> >
>>>> >                 On Sat, Sep 12, 2020 at 12:12 PM Uma S <
>>>> uma.s at vtiger.com
>>>> >                 <mailto:uma.s at vtiger.com>> wrote:
>>>> >
>>>> >                     Hi Team,
>>>> >
>>>> >                     I think we should enhance the import
>>>> functionality,
>>>> >                     So that systems should first analyze the user's
>>>> need
>>>> >                     for import on either to update the existing
>>>> records
>>>> >                     or to replace the record (i.e., delete existing
>>>> and
>>>> >                     creating new one). Based on selected choice
>>>> >                     consequent steps should be taken on records.
>>>> >
>>>> >
>>>> >
>>>> >                     On Fri, Sep 11, 2020 at 10:42 AM nilay khatri
>>>> >                     <nilay.spartan at gmail.com
>>>> >                     <mailto:nilay.spartan at gmail.com>> wrote:
>>>> >
>>>> >                         And thanks for the input everyone.
>>>> >
>>>> >
>>>> >
>>>> >                         On Fri, Sep 11, 2020 at 12:59 AM Rubén A.
>>>> >                         Estrada Orozco <rulotec1 at gmail.com
>>>> >                         <mailto:rulotec1 at gmail.com>> wrote:
>>>> >
>>>> >                             Naily, I've seen similar issues. When
>>>> using
>>>> >                             the merge feature for example, we noticed
>>>> >                             that if you have a phone column in your
>>>> csv
>>>> >                             and its empty and if the record in
>>>> >                             vtgier has already a phone, it gets
>>>> deleted.
>>>> >                             What you would expect from a merge is that
>>>> >                             it would keep the non-empty field.
>>>> >
>>>> >                             I've opened these issues:
>>>> >                             * merge function deletes fields:
>>>> >
>>>> https://code.vtiger.com/vtiger/vtigercrm/issues/1483
>>>> >                             * add record number field for updating
>>>> >                             purposes:
>>>> >
>>>> https://code.vtiger.com/vtiger/vtigercrm/issues/1484
>>>> >
>>>> >                             Saludos
>>>> >
>>>> >                             Rubén
>>>> >
>>>> >
>>>> >                             On Thu, Sep 10, 2020 at 9:36 AM Martin
>>>> Allen
>>>> >                             <martin.allen at clystnet.com
>>>> >                             <mailto:martin.allen at clystnet.com>>
>>>> wrote:
>>>> >
>>>> >                                 We've actually written our own import
>>>> >                                 scripts that will read a CSV file and
>>>> >                                 will update / create records via
>>>> >                                 Webservice, allowing you to do a check
>>>> >                                 for an existing record using whatever
>>>> >                                 fields you want first (and can
>>>> identify
>>>> >                                 and handle if it returns multiple
>>>> >                                 records) and then update only the
>>>> fields
>>>> >                                 you want to.
>>>> >
>>>> >                                 Martin Allen
>>>> >
>>>> >                                 *01392 248692 - Main Office
>>>> >                                 01392 690659 - Direct Line*
>>>> >
>>>> >                                 *signature2*
>>>> >
>>>> >                                 Have you visited our website recently?
>>>> >                                 http://www.clystnet.com
>>>> >                                 <http://www.clystnet.com/>
>>>> >
>>>> >
>>>> >                                 The information in this email is
>>>> >                                 confidential If you are not the
>>>> intended
>>>> >                                 recipient, you must not read or use
>>>> that
>>>> >                                 information. This email and any
>>>> >                                 attachments are believed to be virus
>>>> >                                 free however no responsibility is
>>>> >                                 accepted by Clystnet for any loss or
>>>> >                                 damage arising in any way from receipt
>>>> >                                 or use thereof. Clystnet Ltd (company
>>>> >                                 reg number 7164503) is based at
>>>> >                                 Silverdown Park, Fair Oak Close, Clyst
>>>> >                                 Honiton, EX5 2UX
>>>> >
>>>> >
>>>> >
>>>> >                                 On Thu, 10 Sep 2020 at 15:28, Martin
>>>> >                                 Allen <martin.allen at clystnet.com
>>>> >                                 <mailto:martin.allen at clystnet.com>>
>>>> wrote:
>>>> >
>>>> >                                     I believe that in vTiger hosted
>>>> >                                     version it is possible to have the
>>>> >                                     Lead Number (or any Auto Increment
>>>> >                                     field in any other module - UI
>>>> Type
>>>> >                                     4) field included in an Import
>>>> which
>>>> >                                     can then be used for checking for
>>>> >                                     duplicates (obviously this field
>>>> >                                     value is not imported, only used
>>>> for
>>>> >                                     duplicate checking).
>>>> >
>>>> >                                     Would be good if this was
>>>> available
>>>> >                                     in the open source version, as it
>>>> >                                     would make importing data (to
>>>> update
>>>> >                                     records rather than creating new
>>>> >                                     ones) much simpler and safer, as
>>>> any
>>>> >                                     other field cannot be guaranteed
>>>> to
>>>> >                                     be unique, and sometimes the field
>>>> >                                     you want to update are the fields
>>>> >                                     that you would typically use to
>>>> >                                     identify duplicates (i.e. First
>>>> >                                     Name, Last Name, Email Address for
>>>> >                                     Contacts)
>>>> >
>>>> >                                     Martin Allen
>>>> >
>>>> >                                     *01392 248692 - Main Office
>>>> >                                     01392 690659 - Direct Line*
>>>> >
>>>> >                                     *signature2*
>>>> >
>>>> >                                     Have you visited our website
>>>> >                                     recently? http://www.clystnet.com
>>>> >                                     <http://www.clystnet.com/>
>>>> >
>>>> >
>>>> >                                     The information in this email is
>>>> >                                     confidential If you are not the
>>>> >                                     intended recipient, you must not
>>>> >                                     read or use that information. This
>>>> >                                     email and any attachments are
>>>> >                                     believed to be virus free however
>>>> no
>>>> >                                     responsibility is accepted by
>>>> >                                     Clystnet for any loss or damage
>>>> >                                     arising in any way from receipt or
>>>> >                                     use thereof. Clystnet Ltd (company
>>>> >                                     reg number 7164503) is based at
>>>> >                                     Silverdown Park, Fair Oak Close,
>>>> >                                     Clyst Honiton, EX5 2UX
>>>> >
>>>> >
>>>> >
>>>> >                                     On Thu, 10 Sep 2020 at 13:22,
>>>> nilay
>>>> >                                     khatri <nilay.spartan at gmail.com
>>>> >                                     <mailto:nilay.spartan at gmail.com>>
>>>> wrote:
>>>> >
>>>> >                                         This is a very common
>>>> challenge,
>>>> >                                         users want to update certain
>>>> >                                         data of records in CRM by
>>>> >                                         importing a CSV.
>>>> >
>>>> >                                         Let us say Leads, we want to
>>>> >                                         update Lead Status, so the CSV
>>>> >                                         sheet contains 2 columns
>>>> Primary
>>>> >                                         Email and Status.
>>>> >
>>>> >                                         If we import the CSV and set
>>>> the
>>>> >                                         duplicate check criteria in
>>>> step
>>>> >                                         2 on Primary Email and import
>>>> >                                         following things would happen:
>>>> >
>>>> >                                         - quite possible all the other
>>>> >                                         data in fields will be wiped
>>>> out
>>>> >                                         - if the system contains more
>>>> >                                         than 1 Lead with same email
>>>> id,
>>>> >                                         it would delete them, keeping
>>>> >                                         only 1
>>>> >                                         - the record IDs will change
>>>> >
>>>> >                                         If I talk about contact module
>>>> >                                         that is full of surprises,
>>>> >                                         contact Id is available to
>>>> >                                         select in the duplicate
>>>> >                                         selection criteria, but not
>>>> >                                         available in the field
>>>> mapping!
>>>> >
>>>> >
>>>>  --------------------------------------
>>>> >
>>>> >
>>>> >                                         In the end we want to achieve
>>>> a
>>>> >                                         very simple thing, import a
>>>> List
>>>> >                                         of Leads and update respective
>>>> >                                         status without changing record
>>>> >                                         Ids or deleting any other
>>>> >                                         matching record in system.
>>>> >
>>>> >
>>>> >
>>>>  _______________________________________________
>>>> >                                         http://www.vtiger.com/
>>>> >
>>>> >
>>>>  _______________________________________________
>>>> >                                 http://www.vtiger.com/
>>>> >
>>>> >
>>>>  _______________________________________________
>>>> >                             http://www.vtiger.com/
>>>> >
>>>> >
>>>>  _______________________________________________
>>>> >                         http://www.vtiger.com/
>>>> >
>>>> >
>>>> >
>>>> >                     --
>>>> >                     With
>>>> >                     Best Regards
>>>> >                     Uma.S
>>>> >                     Vtiger Team
>>>> >                     _______________________________________________
>>>> >                     http://www.vtiger.com/
>>>> >
>>>> >                 _______________________________________________
>>>> >                 http://www.vtiger.com/
>>>> >
>>>> >
>>>> >
>>>> >             --
>>>> >             With
>>>> >             Best Regards
>>>> >             Uma.S
>>>> >             Vtiger Team
>>>> >             _______________________________________________
>>>> >             http://www.vtiger.com/
>>>> >
>>>> >         _______________________________________________
>>>> >         http://www.vtiger.com/
>>>> >
>>>> >
>>>> >
>>>> >     --
>>>> >     With
>>>> >     Best Regards
>>>> >     Uma.S
>>>> >     Vtiger Team
>>>> >     _______________________________________________
>>>> >     http://www.vtiger.com/
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > http://www.vtiger.com/
>>>> >
>>>> _______________________________________________
>>>> http://www.vtiger.com/
>>>
>>>
>>>
>>> --
>>> With
>>> Best Regards
>>> Uma.S
>>> Vtiger Team
>>> _______________________________________________
>>> http://www.vtiger.com/
>>
>> _______________________________________________
> http://www.vtiger.com/



-- 
With
Best Regards
Uma.S
Vtiger Team
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.vtigercrm.com/pipermail/vtigercrm-developers/attachments/20200919/70db7ed5/attachment-0001.html>


More information about the vtigercrm-developers mailing list