<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 17/08/16 19:34, Błażej Pabiszczak
wrote:<br>
</div>
<blockquote
cite="mid:d2e1fbac4865dc819cc5bcd854bb8dd6@yetiforce.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p lang="en-US"><span>@Alan</span></p>
<ul>
<li>
<p lang="en-US"><span><span><span>but does it work on the demo
version you sent for testing? Because I don't think it
did, so I'm just asking... in theory you can do
anything, what's important is what you get in
standard.</span></span></span></p>
</li>
<li>
<p lang="en-US"><span><span><span>Do you really add results
until you reach 100?... Because it's not really that
efficient, for a large database it'll always be slower
than regular searching by index. In searching you only
search for matching results, and you have to search
for the data but you additionally first have download
and process them... The larger the base the slower it
will work in comparison to the standard solution which
are regular indexes on the base.</span></span></span></p>
</li>
</ul>
</blockquote>
actually I didn't describe that correctly at all, (it was an earlier
approach, not what we settled on) what we get back from elastic
search is a big list of entity IDs (up to the maxresults setting in
the configuration, which is a picklist with options 50,100,250,500)
we throw that list of IDs at vtiger in chunks of up to 100 at a time
(the vtws limit) with a vtws_query along the lines of <br>
<br>
select id from Accounts where id in ('x_123','x_124', . . .)<br>
<br>
and that gives us back the IDs that the current user can see. This
bit may take a few requests if there are more than 100 results (we
know the upper limit is 500, so a max of 5 quick requests). We then
display to the user results relating to the IDs they can see. This
ends up still being quick as the queries we do in the vtiger engine
are all highly indexed and optimised and the query we do in the
elastic search is fast for full text queries.<br>
<br>
<br>
<blockquote
cite="mid:d2e1fbac4865dc819cc5bcd854bb8dd6@yetiforce.com"
type="cite">
<ul>
<li>
<br>
</li>
</ul>
<p lang="en-US"><span><span><span>We're about to finish optimizing
YetiForce for 40M records, then we'll launch Vtiger with
40M records for you and then we can compare the efficiency
of the global search that you offer as paid solution. If
you don't feel like it, or you don't have the time then we
can just forget about it.</span></span></span></p>
</blockquote>
brilliant, would love to know the results.<br>
<blockquote
cite="mid:d2e1fbac4865dc819cc5bcd854bb8dd6@yetiforce.com"
type="cite">
<p lang="en-US"><span><span><span>@IT-Solutions4You</span></span></span></p>
<p lang="en-US"><span><span><span>Remember that we're only talking
about global search here, which should only search by
globally available fields. And from the business point of
view in my opinion it's okay. Visibility of fields/columns
is controlled already on the records list and in
preview/edition and that seems enough. You can change it
but then you'd have to store separate labels for each
field. The question above probably results from the fact
that some modules search by all fields [which imo is
redundant], and in our system you configure the fields to
search by at the panel level.</span></span></span></p>
</blockquote>
OK, so with that extra configuration step you can index on those
columns and it will be fast for searching the start of those fields
- so if a field contains "Alan Bell" you could quickly search for
"like 'Alan%' and get an instant result, however searching for Like
'%Bell%' is probably not going to be able to use the index - unless
you have another trick going on?<br>
<blockquote
cite="mid:d2e1fbac4865dc819cc5bcd854bb8dd6@yetiforce.com"
type="cite">
<p lang="en-US"><span><span><span>When it comes to large databases
searching on the fly is out of question... everything has
to be cached, even permissions for records [right now thy
are not cached in vtiger, we added it to our system for
each record separately] and among other reasons, this is
why Alan's solution won't work for large databases.</span></span></span></p>
<div>---<br>
<div>Z poważaniem / Regards</div>
<div> </div>
<div><strong>Błażej Pabiszczak</strong></div>
<div><em>Chief Executive Officer</em></div>
<div>M: +48.884999123<br>
E: <a moz-do-not-send="true" title="Mail do Błażej Pabiszczak"
href="mailto:b.pabiszczak@yetiforce.com">b.pabiszczak@yetiforce.com</a></div>
</div>
<p> </p>
<p>W dniu 2016-08-15 09:57, IT-Solutions4You napisał(a):</p>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<div class="pre" style="margin: 0; padding: 0; font-family:
monospace"><span style="white-space: nowrap;">Hi Alan</span><br>
<br>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0"><span style="white-space:
nowrap;">it runs on the various events, create/save/delete and updates the</span><br>
<span style="white-space: nowrap;">corresponding document in the full text server. The full text server</span><br>
<span style="white-space: nowrap;">basically gets given the column_values array as key value pairs.</span></blockquote>
<br>
<span style="white-space: nowrap;">I have question:</span><br>
Do you check fields permission during search? And what about
deactivate/activate custom fields using layout editor. I think
custom fields can be also deleted.<br>
<br>
IMHO search has to be done on the fly not from stored data.
What do you think ?<br>
<br>
Matus<br>
<br>
<span style="white-space: nowrap;">Dňa 12. 8. 2016 o 16:13 Alan Bell napísal(a):</span>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0"><span style="white-space:
nowrap;">Hi Błażej,</span><br>
<br>
<br>
<span style="white-space: nowrap;">On 27/07/16 08:56, Błażej Pabiszczak wrote:</span>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><br>
<span style="white-space: nowrap;">@Alan</span><br>
<br>
<span style="white-space: nowrap;">for me this is just a discussion, if you prefer switching to emails</span><br>
<span style="white-space: nowrap;">then I have no problem with that [though this is vtiger engine problem</span><br>
<span style="white-space: nowrap;">so the list is the perfect place to discuss it]. I don't intend to</span><br>
<span style="white-space: nowrap;">anyhow reduce the value of your module, it's a module like every other</span><br>
<span style="white-space: nowrap;">module, I only want to talk about the logic, and point out some</span><br>
<span style="white-space: nowrap;">problems...</span><br>
<br>
<span style="white-space: nowrap;">1.</span><br>
<br>
<span style="white-space: nowrap;"> We have a module to generate random data [as far as I remember it</span><br>
<span style="white-space: nowrap;"> should support generating data for vtiger by default as well], if</span><br>
<span style="white-space: nowrap;"> you have any problems generating the data, I'll gladly send you</span><br>
<span style="white-space: nowrap;"> the module.</span><br>
<br>
</blockquote>
<span style="white-space: nowrap;">sounds interesting, yes that would be helpful, thanks.</span>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><br>
<span style="white-space: nowrap;">1.</span><br>
<br>
<br>
<span style="white-space: nowrap;">2.</span><br>
<br>
<span style="white-space: nowrap;"> I don't know how your solution works, but if it's based on</span><br>
<span style="white-space: nowrap;"> Full-Text Search, then you won't get these data, for us it's a</span><br>
<span style="white-space: nowrap;"> significant limitation for a search tool.</span><br>
<br>
</blockquote>
<span style="white-space: nowrap;">it runs on the various events, create/save/delete and updates the</span><br>
<span style="white-space: nowrap;">corresponding document in the full text server. The full text server</span><br>
<span style="white-space: nowrap;">basically gets given the column_values array as key value pairs.</span>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><br>
<span style="white-space: nowrap;">1.</span><br>
<br>
<br>
<span style="white-space: nowrap;">2.</span><br>
<br>
<span style="white-space: nowrap;"> So your search is not very useful when it comes to large</span><br>
<span style="white-space: nowrap;"> databases? When you send queries to the database, how many records</span><br>
<span style="white-space: nowrap;"> do you get [50, 100, 1000?], if you for example get 100 records,</span><br>
<span style="white-space: nowrap;"> and the system takes away the permissions on all records [because</span><br>
<span style="white-space: nowrap;"> the record that the user can access is on the 2000^th place], then</span><br>
<span style="white-space: nowrap;"> will you show the user that no data was found? The History of</span><br>
<span style="white-space: nowrap;"> Changes widget on the homepage works like that right now – the</span><br>
<span style="white-space: nowrap;"> bigger the base and the less data the user sees, the more often he</span><br>
<span style="white-space: nowrap;"> sees nothing at all in the history of changes. If the mechanism</span><br>
<span style="white-space: nowrap;"> works in the way I described [and I hope I'm wrong] then it has no</span><br>
<span style="white-space: nowrap;"> business value because it falsifies the result.</span><br>
<br>
</blockquote>
<span style="white-space: nowrap;">we keep getting more data until we run out of results or fill the page</span><br>
<span style="white-space: nowrap;">we want to send to the user, so if they asked for 100 records we might</span><br>
<span style="white-space: nowrap;">look through 500 until we manage to get the top 100 that they can see.</span>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><br>
<span style="white-space: nowrap;">1.</span><br>
<br>
<br>
<span style="white-space: nowrap;">As far as large number of users and test data are concerned, I'm</span><br>
<span style="white-space: nowrap;">trying to show you that it doesn't matter if your search engine works</span><br>
<span style="white-space: nowrap;">fast on large databases or not, because Vtiger as a whole doesn't work</span><br>
<span style="white-space: nowrap;">well with large databases. I'll give you an example: in what way does</span><br>
<span style="white-space: nowrap;">vtiger transfer data to select type fields? It transfers data by</span><br>
<span style="white-space: nowrap;">sending html, instead of sending queries at AJAX database level and</span><br>
<span style="white-space: nowrap;">sending the users list directly there. Additionally, it uses Chosen</span><br>
<span style="white-space: nowrap;">instead of Select2; Select 2 is five times faster when it comes to</span><br>
<span style="white-space: nowrap;">large amounts of data [eg 10,000]. When you load 20,000 users in</span><br>
<span style="white-space: nowrap;">vtiger then one field will take SEVERAL SECONDS to load. There are</span><br>
<span style="white-space: nowrap;">hundreds of examples like this, and they result from the fact that</span><br>
<span style="white-space: nowrap;">nobody has deployed this system for a large company [I don't mean 100</span><br>
<span style="white-space: nowrap;">users, I mean 2,000 – 20,000 users].</span><br>
<br>
</blockquote>
<span style="white-space: nowrap;">yeah, we haven't tried scaling vtiger to thousands of users, but we have</span><br>
<span style="white-space: nowrap;">done things with pretty chunky amounts of data. Scaling the users means</span><br>
<span style="white-space: nowrap;">you hit different problems. The full text index is aimed at large</span><br>
<span style="white-space: nowrap;">amounts of records, it could search gigabytes or terrabytes of data but</span><br>
<span style="white-space: nowrap;">it doesn't solve any of the problems of a large number of users.</span>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><br>
<span style="white-space: nowrap;">For the system to work properly it's not enough that you change the</span><br>
<span style="white-space: nowrap;">global search engine, you also have to change the logic and the</span><br>
<span style="white-space: nowrap;">core... because the changes are related to hundreds of files. So my</span><br>
<span style="white-space: nowrap;">question is – do you want to develop the system on at the highest</span><br>
<span style="white-space: nowrap;">level as developers, or do you want to make toy modules for micro</span><br>
<span style="white-space: nowrap;">companies? The core needs changes, and they must be introduced to</span><br>
<span style="white-space: nowrap;">code.vtiger.com, instead of trying to find a solution by implementing</span><br>
<span style="white-space: nowrap;">modules, because it can't work like that!</span><br>
<br>
</blockquote>
<span style="white-space: nowrap;">yeah, there are plenty of core changes that need doing, I want to get</span><br>
<span style="white-space: nowrap;">the javascript libraries in order, perhaps using bower as a package</span><br>
<span style="white-space: nowrap;">manager for it. Mobile device support is something that can be improved,</span><br>
<span style="white-space: nowrap;">there is lots that can be done! The way I see modules working well are</span><br>
<span style="white-space: nowrap;">things for integrating vtiger with another tool that most people won't</span><br>
<span style="white-space: nowrap;">want, or vertical market solutions - perhaps something for estate agents</span><br>
<span style="white-space: nowrap;">selling houses for example. If it is something that everyone would want</span><br>
<span style="white-space: nowrap;">then it might as well go in the core.</span><br>
<br>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><span
style="white-space: nowrap;">---</span><br>
<span style="white-space: nowrap;">Z poważaniem / Regards</span><br>
<br>
<span style="white-space: nowrap;">*Błażej Pabiszczak*</span><br>
<span style="white-space: nowrap;">/Chief Executive Officer/</span><br>
<span style="white-space: nowrap;">M: +48.884999123</span><br>
<span style="white-space: nowrap;">E: <a
moz-do-not-send="true"
href="mailto:b.pabiszczak@yetiforce.com">b.pabiszczak@yetiforce.com</a></span><br>
<span style="white-space: nowrap;"><mailto:<a
moz-do-not-send="true"
href="mailto:b.pabiszczak@yetiforce.com">b.pabiszczak@yetiforce.com</a>></span><br>
<span style="white-space: nowrap;">------------------------------------------------------------------------</span><br>
<br>
<span style="white-space: nowrap;">YetiForce 3.0 LTS has arrived! Test</span><br>
<span style="white-space: nowrap;"><<a
moz-do-not-send="true"
href="https://gitdeveloper.yetiforce.com/">https://gitdeveloper.yetiforce.com/</a>> the latest, most innovative open</span><br>
<span style="white-space: nowrap;">source system in the world, and join</span><br>
<span style="white-space: nowrap;"><<a
moz-do-not-send="true"
href="https://github.com/YetiForceCompany/YetiForceCRM">https://github.com/YetiForceCompany/YetiForceCRM</a>>our community.</span><br>
<br>
<br>
<br>
<span style="white-space: nowrap;">W dniu 2016-07-26 16:42, Alan Bell napisał(a):</span><br>
<br>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><br>
<br>
<br>
<span style="white-space: nowrap;">On 26/07/16 15:10, Błażej Pabiszczak wrote:</span>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><br>
<span style="white-space: nowrap;"> 1. Can you fill some test system with data, eg. Add 10.000.000</span><br>
<span style="white-space: nowrap;"> records into the Accounts module?</span><br>
<br>
</blockquote>
<span style="white-space: nowrap;">yeah, I might have a go at that. I guess numbered data and some</span><br>
<span style="white-space: nowrap;">random stuff should be fairly easy to generate. I am not expecting it</span><br>
<span style="white-space: nowrap;">to affect the performance much.</span>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><span
style="white-space: nowrap;">2. Does your search engine allow to search parts of words, eg. I</span><br>
<span style="white-space: nowrap;">want to search for "EFG" in the word "ABCDEFGHI"?</span></blockquote>
<span style="white-space: nowrap;">it should do, yes but there may be a performance hit for putting a</span><br>
<span style="white-space: nowrap;">wildcard at the start of the query. Looking at it now, I am not</span><br>
<span style="white-space: nowrap;">entirely sure we are handling wildcards correctly, but the underlying</span><br>
<span style="white-space: nowrap;">engine should work with them.</span>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><br>
<span style="white-space: nowrap;">3. Can you add 10.000 users to the tested system?</span><br>
<br>
</blockquote>
<span style="white-space: nowrap;">won't make any different to the search performance as such - however</span><br>
<span style="white-space: nowrap;">when we get the search results back we do a vtiger API query to see</span><br>
<span style="white-space: nowrap;">if each crmid should be seen by the current user, that way you can't</span><br>
<span style="white-space: nowrap;">find things that are private and assigned to other people. I am not</span><br>
<span style="white-space: nowrap;">sure if that scales to 10,000 users however if it doesn't then list</span><br>
<span style="white-space: nowrap;">views in vtiger don't work either!</span><br>
<br>
<blockquote type="cite" style="padding: 0 0.4em;
border-left: #1010ff 2px solid; margin: 0"><span
style="white-space: nowrap;">---</span><br>
<span style="white-space: nowrap;">Z poważaniem / Regards</span><br>
<br>
<span style="white-space: nowrap;">*Błażej Pabiszczak*</span><br>
<span style="white-space: nowrap;">/Chief Executive Officer/</span><br>
<span style="white-space: nowrap;">M: +48.884999123</span><br>
<span style="white-space: nowrap;">E: <a
moz-do-not-send="true"
href="mailto:b.pabiszczak@yetiforce.com">b.pabiszczak@yetiforce.com</a></span><br>
<span style="white-space: nowrap;"><mailto:<a
moz-do-not-send="true"
href="mailto:b.pabiszczak@yetiforce.com">b.pabiszczak@yetiforce.com</a>></span><br>
<span style="white-space: nowrap;">------------------------------------------------------------------------</span><br>
<br>
<span style="white-space: nowrap;">YetiForce 3.0 LTS has arrived! Test</span><br>
<span style="white-space: nowrap;"><<a
moz-do-not-send="true"
href="https://gitdeveloper.yetiforce.com/">https://gitdeveloper.yetiforce.com/</a>> the latest, most innovative</span><br>
<span style="white-space: nowrap;">open source system in the world, and join</span><br>
<span style="white-space: nowrap;"><<a
moz-do-not-send="true"
href="https://github.com/YetiForceCompany/YetiForceCRM">https://github.com/YetiForceCompany/YetiForceCRM</a>>our community.</span><br>
<br>
<br>
<br>
<span style="white-space: nowrap;">W dniu 2016-07-20 14:32, Alan Lord napisał(a):</span><br>
<br>
<span style="white-space: nowrap;"> On 20/07/16 11:50, Alan Bell wrote:</span><br>
<br>
<br>
<span style="white-space: nowrap;"> 3. Does the search engine need any special search attributes?</span><br>
<br>
<span style="white-space: nowrap;"> not really, we do get a bit of a query language for free by using</span><br>
<span style="white-space: nowrap;"> elastic search, but these are users who just want to type in words or</span><br>
<span style="white-space: nowrap;"> phone numbers or dates and have the system find stuff.</span><br>
<br>
<br>
<span style="white-space: nowrap;"> For anyone interested, you can try it out on our demo server:</span><br>
<br>
<span style="white-space: nowrap;"> <a
moz-do-not-send="true"
href="http://geotools.libertus.co.uk/index.php">http://geotools.libertus.co.uk/index.php</a></span><br>
<br>
<span style="white-space: nowrap;"> Just start typing anything in the normal search box at the top of any page.</span><br>
<br>
<span style="white-space: nowrap;"> After three characters it will start sending stuff to the</span><br>
<span style="white-space: nowrap;"> Elasticsearch server and displaying results in a popup.</span><br>
<br>
<span style="white-space: nowrap;"> If you want to you can go to the Vlastic Search view, which</span><br>
<span style="white-space: nowrap;"> shows multiple ListViews of results on the same page with</span><br>
<span style="white-space: nowrap;"> pagination too.</span><br>
<br>
<span style="white-space: nowrap;"> e.g.:</span><br>
<span style="white-space: nowrap;"> <a
moz-do-not-send="true"
href="http://geotools.libertus.co.uk/index.php?module=LSVlasticSearch&view=Search&value=smithf*">http://geotools.libertus.co.uk/index.php?module=LSVlasticSearch&view=Search&value=smithf*</a></span><br>
<br>
<span style="white-space: nowrap;"> This demo system has about 120,000 records in it but it could</span><br>
<span style="white-space: nowrap;"> have lots more...</span><br>
<br>
<span style="white-space: nowrap;"> The manual explains some of the search language capabilities,</span><br>
<span style="white-space: nowrap;"> such as AND, OR, parenthesis, wildcards etc...</span><br>
<br>
<span style="white-space: nowrap;"> But if you type in "smith" it will find all records that has the</span><br>
<span style="white-space: nowrap;"> word smith in them somewhere.</span><br>
<br>
<span style="white-space: nowrap;"> If you wanted to find something like smithfields (a meat market</span><br>
<span style="white-space: nowrap;"> in London) you could type smith* or smithf* for example.</span><br>
<br>
<span style="white-space: nowrap;"> Cheers</span><br>
<br>
<span style="white-space: nowrap;"> Al</span><br>
<br>
<br>
<span style="white-space: nowrap;"> _______________________________________________</span><br>
<span style="white-space: nowrap;"> <a
moz-do-not-send="true"
href="http://www.vtiger.com/">http://www.vtiger.com/</a></span><br>
<br>
<br>
<br>
_______________________________________________<br>
<span style="white-space: nowrap;"><a
moz-do-not-send="true"
href="http://www.vtiger.com/">http://www.vtiger.com/</a></span></blockquote>
<br>
<br>
_______________________________________________<br>
<span style="white-space: nowrap;"><a
moz-do-not-send="true" href="http://www.vtiger.com/">http://www.vtiger.com/</a></span></blockquote>
<br>
<br>
_______________________________________________<br>
<span style="white-space: nowrap;"><a
moz-do-not-send="true" href="http://www.vtiger.com/">http://www.vtiger.com/</a></span></blockquote>
<br>
<br>
<br>
_______________________________________________<br>
<span style="white-space: nowrap;"><a moz-do-not-send="true"
href="http://www.vtiger.com/">http://www.vtiger.com/</a></span><br>
<br>
</blockquote>
<br>
<br>
_______________________________________________<br>
<span style="white-space: nowrap;"><a moz-do-not-send="true"
href="http://www.vtiger.com/">http://www.vtiger.com/</a></span></div>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
<a class="moz-txt-link-freetext" href="http://www.vtiger.com/">http://www.vtiger.com/</a></pre>
</blockquote>
<br>
</body>
</html>