[Vtigercrm-developers] Email marketing scenario
Alan Bell
alan.bell at libertus.co.uk
Thu Aug 18 09:33:49 GMT 2016
On 17/08/16 19:34, Błażej Pabiszczak wrote:
>
> @Alan
>
> *
>
> but does it work on the demo version you sent for testing? Because
> I don't think it did, so I'm just asking... in theory you can do
> anything, what's important is what you get in standard.
>
> *
>
> Do you really add results until you reach 100?... Because it's not
> really that efficient, for a large database it'll always be slower
> than regular searching by index. In searching you only search for
> matching results, and you have to search for the data but you
> additionally first have download and process them... The larger
> the base the slower it will work in comparison to the standard
> solution which are regular indexes on the base.
>
actually I didn't describe that correctly at all, (it was an earlier
approach, not what we settled on) what we get back from elastic search
is a big list of entity IDs (up to the maxresults setting in the
configuration, which is a picklist with options 50,100,250,500) we throw
that list of IDs at vtiger in chunks of up to 100 at a time (the vtws
limit) with a vtws_query along the lines of
select id from Accounts where id in ('x_123','x_124', . . .)
and that gives us back the IDs that the current user can see. This bit
may take a few requests if there are more than 100 results (we know the
upper limit is 500, so a max of 5 quick requests). We then display to
the user results relating to the IDs they can see. This ends up still
being quick as the queries we do in the vtiger engine are all highly
indexed and optimised and the query we do in the elastic search is fast
for full text queries.
> *
>
>
>
> We're about to finish optimizing YetiForce for 40M records, then we'll
> launch Vtiger with 40M records for you and then we can compare the
> efficiency of the global search that you offer as paid solution. If
> you don't feel like it, or you don't have the time then we can just
> forget about it.
>
brilliant, would love to know the results.
>
> @IT-Solutions4You
>
> Remember that we're only talking about global search here, which
> should only search by globally available fields. And from the business
> point of view in my opinion it's okay. Visibility of fields/columns is
> controlled already on the records list and in preview/edition and that
> seems enough. You can change it but then you'd have to store separate
> labels for each field. The question above probably results from the
> fact that some modules search by all fields [which imo is redundant],
> and in our system you configure the fields to search by at the panel
> level.
>
OK, so with that extra configuration step you can index on those columns
and it will be fast for searching the start of those fields - so if a
field contains "Alan Bell" you could quickly search for "like 'Alan%'
and get an instant result, however searching for Like '%Bell%' is
probably not going to be able to use the index - unless you have another
trick going on?
>
> When it comes to large databases searching on the fly is out of
> question... everything has to be cached, even permissions for records
> [right now thy are not cached in vtiger, we added it to our system for
> each record separately] and among other reasons, this is why Alan's
> solution won't work for large databases.
>
> ---
> Z poważaniem / Regards
> *Błażej Pabiszczak*
> /Chief Executive Officer/
> M: +48.884999123
> E: b.pabiszczak at yetiforce.com <mailto:b.pabiszczak at yetiforce.com>
>
> W dniu 2016-08-15 09:57, IT-Solutions4You napisał(a):
>
>> Hi Alan
>>
>>> it runs on the various events, create/save/delete and updates the
>>> corresponding document in the full text server. The full text server
>>> basically gets given the column_values array as key value pairs.
>>
>> I have question:
>> Do you check fields permission during search? And what about
>> deactivate/activate custom fields using layout editor. I think custom
>> fields can be also deleted.
>>
>> IMHO search has to be done on the fly not from stored data. What do
>> you think ?
>>
>> Matus
>>
>> Dňa 12. 8. 2016 o 16:13 Alan Bell napísal(a):
>>> Hi Błażej,
>>>
>>>
>>> On 27/07/16 08:56, Błażej Pabiszczak wrote:
>>>>
>>>> @Alan
>>>>
>>>> for me this is just a discussion, if you prefer switching to emails
>>>> then I have no problem with that [though this is vtiger engine problem
>>>> so the list is the perfect place to discuss it]. I don't intend to
>>>> anyhow reduce the value of your module, it's a module like every other
>>>> module, I only want to talk about the logic, and point out some
>>>> problems...
>>>>
>>>> 1.
>>>>
>>>> We have a module to generate random data [as far as I remember it
>>>> should support generating data for vtiger by default as well], if
>>>> you have any problems generating the data, I'll gladly send you
>>>> the module.
>>>>
>>> sounds interesting, yes that would be helpful, thanks.
>>>>
>>>> 1.
>>>>
>>>>
>>>> 2.
>>>>
>>>> I don't know how your solution works, but if it's based on
>>>> Full-Text Search, then you won't get these data, for us it's a
>>>> significant limitation for a search tool.
>>>>
>>> it runs on the various events, create/save/delete and updates the
>>> corresponding document in the full text server. The full text server
>>> basically gets given the column_values array as key value pairs.
>>>>
>>>> 1.
>>>>
>>>>
>>>> 2.
>>>>
>>>> So your search is not very useful when it comes to large
>>>> databases? When you send queries to the database, how many records
>>>> do you get [50, 100, 1000?], if you for example get 100 records,
>>>> and the system takes away the permissions on all records [because
>>>> the record that the user can access is on the 2000^th place], then
>>>> will you show the user that no data was found? The History of
>>>> Changes widget on the homepage works like that right now – the
>>>> bigger the base and the less data the user sees, the more often he
>>>> sees nothing at all in the history of changes. If the mechanism
>>>> works in the way I described [and I hope I'm wrong] then it has no
>>>> business value because it falsifies the result.
>>>>
>>> we keep getting more data until we run out of results or fill the page
>>> we want to send to the user, so if they asked for 100 records we might
>>> look through 500 until we manage to get the top 100 that they can see.
>>>>
>>>> 1.
>>>>
>>>>
>>>> As far as large number of users and test data are concerned, I'm
>>>> trying to show you that it doesn't matter if your search engine works
>>>> fast on large databases or not, because Vtiger as a whole doesn't work
>>>> well with large databases. I'll give you an example: in what way does
>>>> vtiger transfer data to select type fields? It transfers data by
>>>> sending html, instead of sending queries at AJAX database level and
>>>> sending the users list directly there. Additionally, it uses Chosen
>>>> instead of Select2; Select 2 is five times faster when it comes to
>>>> large amounts of data [eg 10,000]. When you load 20,000 users in
>>>> vtiger then one field will take SEVERAL SECONDS to load. There are
>>>> hundreds of examples like this, and they result from the fact that
>>>> nobody has deployed this system for a large company [I don't mean 100
>>>> users, I mean 2,000 – 20,000 users].
>>>>
>>> yeah, we haven't tried scaling vtiger to thousands of users, but we have
>>> done things with pretty chunky amounts of data. Scaling the users means
>>> you hit different problems. The full text index is aimed at large
>>> amounts of records, it could search gigabytes or terrabytes of data but
>>> it doesn't solve any of the problems of a large number of users.
>>>>
>>>> For the system to work properly it's not enough that you change the
>>>> global search engine, you also have to change the logic and the
>>>> core... because the changes are related to hundreds of files. So my
>>>> question is – do you want to develop the system on at the highest
>>>> level as developers, or do you want to make toy modules for micro
>>>> companies? The core needs changes, and they must be introduced to
>>>> code.vtiger.com, instead of trying to find a solution by implementing
>>>> modules, because it can't work like that!
>>>>
>>> yeah, there are plenty of core changes that need doing, I want to get
>>> the javascript libraries in order, perhaps using bower as a package
>>> manager for it. Mobile device support is something that can be improved,
>>> there is lots that can be done! The way I see modules working well are
>>> things for integrating vtiger with another tool that most people won't
>>> want, or vertical market solutions - perhaps something for estate agents
>>> selling houses for example. If it is something that everyone would want
>>> then it might as well go in the core.
>>>
>>>> ---
>>>> Z poważaniem / Regards
>>>>
>>>> *Błażej Pabiszczak*
>>>> /Chief Executive Officer/
>>>> M: +48.884999123
>>>> E: b.pabiszczak at yetiforce.com <mailto:b.pabiszczak at yetiforce.com>
>>>> <mailto:b.pabiszczak at yetiforce.com <mailto:b.pabiszczak at yetiforce.com>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> YetiForce 3.0 LTS has arrived! Test
>>>> <https://gitdeveloper.yetiforce.com/> the latest, most innovative open
>>>> source system in the world, and join
>>>> <https://github.com/YetiForceCompany/YetiForceCRM>our community.
>>>>
>>>>
>>>>
>>>> W dniu 2016-07-26 16:42, Alan Bell napisał(a):
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 26/07/16 15:10, Błażej Pabiszczak wrote:
>>>>>>
>>>>>> 1. Can you fill some test system with data, eg. Add 10.000.000
>>>>>> records into the Accounts module?
>>>>>>
>>>>> yeah, I might have a go at that. I guess numbered data and some
>>>>> random stuff should be fairly easy to generate. I am not expecting it
>>>>> to affect the performance much.
>>>>>> 2. Does your search engine allow to search parts of words, eg. I
>>>>>> want to search for "EFG" in the word "ABCDEFGHI"?
>>>>> it should do, yes but there may be a performance hit for putting a
>>>>> wildcard at the start of the query. Looking at it now, I am not
>>>>> entirely sure we are handling wildcards correctly, but the underlying
>>>>> engine should work with them.
>>>>>>
>>>>>> 3. Can you add 10.000 users to the tested system?
>>>>>>
>>>>> won't make any different to the search performance as such - however
>>>>> when we get the search results back we do a vtiger API query to see
>>>>> if each crmid should be seen by the current user, that way you can't
>>>>> find things that are private and assigned to other people. I am not
>>>>> sure if that scales to 10,000 users however if it doesn't then list
>>>>> views in vtiger don't work either!
>>>>>
>>>>>> ---
>>>>>> Z poważaniem / Regards
>>>>>>
>>>>>> *Błażej Pabiszczak*
>>>>>> /Chief Executive Officer/
>>>>>> M: +48.884999123
>>>>>> E: b.pabiszczak at yetiforce.com <mailto:b.pabiszczak at yetiforce.com>
>>>>>> <mailto:b.pabiszczak at yetiforce.com
>>>>>> <mailto:b.pabiszczak at yetiforce.com>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> YetiForce 3.0 LTS has arrived! Test
>>>>>> <https://gitdeveloper.yetiforce.com/> the latest, most innovative
>>>>>> open source system in the world, and join
>>>>>> <https://github.com/YetiForceCompany/YetiForceCRM>our community.
>>>>>>
>>>>>>
>>>>>>
>>>>>> W dniu 2016-07-20 14:32, Alan Lord napisał(a):
>>>>>>
>>>>>> On 20/07/16 11:50, Alan Bell wrote:
>>>>>>
>>>>>>
>>>>>> 3. Does the search engine need any special search attributes?
>>>>>>
>>>>>> not really, we do get a bit of a query language for free by using
>>>>>> elastic search, but these are users who just want to type in words or
>>>>>> phone numbers or dates and have the system find stuff.
>>>>>>
>>>>>>
>>>>>> For anyone interested, you can try it out on our demo server:
>>>>>>
>>>>>> http://geotools.libertus.co.uk/index.php
>>>>>>
>>>>>> Just start typing anything in the normal search box at the top of any page.
>>>>>>
>>>>>> After three characters it will start sending stuff to the
>>>>>> Elasticsearch server and displaying results in a popup.
>>>>>>
>>>>>> If you want to you can go to the Vlastic Search view, which
>>>>>> shows multiple ListViews of results on the same page with
>>>>>> pagination too.
>>>>>>
>>>>>> e.g.:
>>>>>> http://geotools.libertus.co.uk/index.php?module=LSVlasticSearch&view=Search&value=smithf*
>>>>>>
>>>>>> This demo system has about 120,000 records in it but it could
>>>>>> have lots more...
>>>>>>
>>>>>> The manual explains some of the search language capabilities,
>>>>>> such as AND, OR, parenthesis, wildcards etc...
>>>>>>
>>>>>> But if you type in "smith" it will find all records that has the
>>>>>> word smith in them somewhere.
>>>>>>
>>>>>> If you wanted to find something like smithfields (a meat market
>>>>>> in London) you could type smith* or smithf* for example.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Al
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> http://www.vtiger.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> http://www.vtiger.com/
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> http://www.vtiger.com/
>>>>
>>>>
>>>> _______________________________________________
>>>> http://www.vtiger.com/
>>>
>>>
>>>
>>> _______________________________________________
>>> http://www.vtiger.com/
>>>
>>
>>
>> _______________________________________________
>> http://www.vtiger.com/
>
>
> _______________________________________________
> http://www.vtiger.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.vtigercrm.com/pipermail/vtigercrm-developers/attachments/20160818/952a11ba/attachment-0001.html>
More information about the vtigercrm-developers
mailing list