<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Błażej,</p>
<br>
<div class="moz-cite-prefix">On 27/07/16 08:56, Błażej Pabiszczak
wrote:<br>
</div>
<blockquote
cite="mid:b7cf57f685d3936205df8db5ed6504d3@yetiforce.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p lang="en-US"><span>@Alan</span></p>
<p lang="en-US"><span><span><span>for me this is just a
discussion, if you prefer switching to emails then I have
no problem with that [though this is vtiger engine problem
so the list is the perfect place to discuss it]. I don't
intend to anyhow reduce the value of your module, it's a
module like every other module, I only want to talk about
the logic, and point out some problems...</span></span></span></p>
<ol>
<li>
<p lang="en-US"><span><span><span>We have a module to generate
random data [as far as I remember it should support
generating data for vtiger by default as well], if you
have any problems generating the data, I'll gladly
send you the module.</span></span></span></p>
</li>
</ol>
</blockquote>
sounds interesting, yes that would be helpful, thanks.<br>
<blockquote
cite="mid:b7cf57f685d3936205df8db5ed6504d3@yetiforce.com"
type="cite">
<ol>
<li>
<br>
</li>
<li>
<p lang="en-US"><span><span><span>I don't know how your
solution works, but if it's based on Full-Text Search,
then you won't get these data, for us it's a
significant limitation for a search tool.</span></span></span></p>
</li>
</ol>
</blockquote>
it runs on the various events, create/save/delete and updates the
corresponding document in the full text server. The full text server
basically gets given the column_values array as key value pairs.<br>
<blockquote
cite="mid:b7cf57f685d3936205df8db5ed6504d3@yetiforce.com"
type="cite">
<ol>
<li>
<br>
</li>
<li>
<p lang="en-US"><span><span><span>So your search is not very
useful when it comes to large databases? When you send
queries to the database, how many records do you get
[50, 100, 1000?], if you for example get 100 records,
and the system takes away the permissions on all
records [because the record that the user can access
is on the 2000<sup>th</sup> place], then will you show
the user that no data was found? The History of
Changes widget on the homepage works like that right
now – the bigger the base and the less data the user
sees, the more often he sees nothing at all in the
history of changes. If the mechanism works in the way
I described [and I hope I'm wrong] then it has no
business value because it falsifies the result.</span></span></span></p>
</li>
</ol>
</blockquote>
we keep getting more data until we run out of results or fill the
page we want to send to the user, so if they asked for 100 records
we might look through 500 until we manage to get the top 100 that
they can see.<br>
<blockquote
cite="mid:b7cf57f685d3936205df8db5ed6504d3@yetiforce.com"
type="cite">
<ol>
<li>
<br>
</li>
</ol>
<p lang="en-US"><span><span><span>As far as large number of users
and test data are concerned, I'm trying to show you that
it doesn't matter if your search engine works fast on
large databases or not, because Vtiger as a whole doesn't
work well with large databases. I'll give you an example:
in what way does vtiger transfer data to select type
fields? It transfers data by sending html, instead of
sending queries at AJAX database level and sending the
users list directly there. Additionally, it uses Chosen
instead of Select2; Select 2 is five times faster when it
comes to large amounts of data [eg 10,000]. When you load
20,000 users in vtiger then one field will take SEVERAL
SECONDS to load. There are hundreds of examples like this,
and they result from the fact that nobody has deployed
this system for a large company [I don't mean 100 users, I
mean 2,000 – 20,000 users].</span></span></span></p>
</blockquote>
yeah, we haven't tried scaling vtiger to thousands of users, but we
have done things with pretty chunky amounts of data. Scaling the
users means you hit different problems. The full text index is aimed
at large amounts of records, it could search gigabytes or terrabytes
of data but it doesn't solve any of the problems of a large number
of users.<br>
<blockquote
cite="mid:b7cf57f685d3936205df8db5ed6504d3@yetiforce.com"
type="cite">
<p><span><span><span>For the system to work properly it's not
enough that you change the global search engine, you also
have to change the logic and the core... because the
changes are related to hundreds of files. So my question
is – do you want to develop the system on at the highest
level as developers, or do you want to make toy modules
for micro companies? The core needs changes, and they must
be introduced to code.vtiger.com, instead of trying to
find a solution by implementing modules, because it can't
work like that!</span></span></span></p>
</blockquote>
yeah, there are plenty of core changes that need doing, I want to
get the javascript libraries in order, perhaps using bower as a
package manager for it. Mobile device support is something that can
be improved, there is lots that can be done! The way I see modules
working well are things for integrating vtiger with another tool
that most people won't want, or vertical market solutions - perhaps
something for estate agents selling houses for example. If it is
something that everyone would want then it might as well go in the
core.<br>
<br>
<blockquote
cite="mid:b7cf57f685d3936205df8db5ed6504d3@yetiforce.com"
type="cite">
<div>---<br>
<div>Z poważaniem / Regards</div>
<div> </div>
<div><strong>Błażej Pabiszczak</strong></div>
<div><em>Chief Executive Officer</em></div>
<div>M: +48.884999123<br>
E: <a moz-do-not-send="true" title="Mail do Błażej Pabiszczak"
href="mailto:b.pabiszczak@yetiforce.com">b.pabiszczak@yetiforce.com</a></div>
<hr>
<p><span>YetiForce 3.0 LTS has arrived! </span><a
moz-do-not-send="true"
href="https://gitdeveloper.yetiforce.com/" rel="noreferrer">Test</a><span> the
latest, most innovative open source system in the world, and
</span><a moz-do-not-send="true"
href="https://github.com/YetiForceCompany/YetiForceCRM"
rel="noreferrer">join</a><span> our community.</span></p>
</div>
<p> </p>
<p>W dniu 2016-07-26 16:42, Alan Bell napisał(a):</p>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0"><!-- html ignored --><!-- head ignored --><!-- meta ignored -->
<p> </p>
<br>
<div class="moz-cite-prefix">On 26/07/16 15:10, Błażej
Pabiszczak wrote:</div>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0">
<ol>
<li>Can you fill some test system with data, eg. Add
10.000.000 records into the Accounts module?</li>
</ol>
</blockquote>
yeah, I might have a go at that. I guess numbered data and some
random stuff should be fairly easy to generate. I am not
expecting it to affect the performance much.<br>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0">2. Does your search engine allow
to search parts of words, eg. I want to search for "EFG" in
the word "ABCDEFGHI"?</blockquote>
it should do, yes but there may be a performance hit for putting
a wildcard at the start of the query. Looking at it now, I am
not entirely sure we are handling wildcards correctly, but the
underlying engine should work with them.<br>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0">
<p lang="en-US">3. Can you add 10.000 users to the tested
system?</p>
</blockquote>
won't make any different to the search performance as such -
however when we get the search results back we do a vtiger API
query to see if each crmid should be seen by the current user,
that way you can't find things that are private and assigned to
other people. I am not sure if that scales to 10,000 users
however if it doesn't then list views in vtiger don't work
either!<br>
<br>
<blockquote type="cite" style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0">
<div>---<br>
<div>Z poważaniem / Regards</div>
<div> </div>
<div><strong>Błażej Pabiszczak</strong></div>
<div><em>Chief Executive Officer</em></div>
<div>M: +48.884999123<br>
E: <a moz-do-not-send="true" title="Mail do Błażej
Pabiszczak" href="mailto:b.pabiszczak@yetiforce.com">b.pabiszczak@yetiforce.com</a></div>
<hr>
<p><span>YetiForce 3.0 LTS has arrived! </span><a
moz-do-not-send="true"
href="https://gitdeveloper.yetiforce.com/">Test</a><span> the
latest, most innovative open source system in the world,
and </span><a moz-do-not-send="true"
href="https://github.com/YetiForceCompany/YetiForceCRM">join</a><span>
our community.</span></p>
</div>
<p> </p>
<p>W dniu 2016-07-20 14:32, Alan Lord napisał(a):</p>
<blockquote style="padding: 0 0.4em; border-left: #1010ff 2px
solid; margin: 0;">
<div class="pre" style="margin: 0; padding: 0; font-family:
monospace;"><span style="white-space: nowrap;">On 20/07/16 11:50, Alan Bell wrote:</span>
<blockquote style="padding: 0 0.4em; border-left: #1010ff
2px solid; margin: 0;"><br>
<blockquote style="padding: 0 0.4em; border-left:
#1010ff 2px solid; margin: 0;"><span
style="white-space: nowrap;">3. Does the search engine need any special search attributes?</span><br>
<br>
</blockquote>
<span style="white-space: nowrap;">not really, we do get a bit of a query language for free by using</span><br>
<span style="white-space: nowrap;">elastic search, but these are users who just want to type in words or</span><br>
<span style="white-space: nowrap;">phone numbers or dates and have the system find stuff.</span></blockquote>
<br>
<span style="white-space: nowrap;">For anyone interested, you can try it out on our demo server:</span><br>
<br>
<span style="white-space: nowrap;"><a
moz-do-not-send="true"
href="http://geotools.libertus.co.uk/index.php">http://geotools.libertus.co.uk/index.php</a></span><br>
<br>
<span style="white-space: nowrap;">Just start typing anything in the normal search box at the top of any page.</span><br>
<br>
After three characters it will start sending stuff to the
Elasticsearch server and displaying results in a popup.<br>
<br>
If you want to you can go to the Vlastic Search view,
which shows multiple ListViews of results on the same page
with pagination too.<br>
<br>
e.g.: <a moz-do-not-send="true"
href="http://geotools.libertus.co.uk/index.php?module=LSVlasticSearch&view=Search&value=smithf*">http://geotools.libertus.co.uk/index.php?module=LSVlasticSearch&view=Search&value=smithf*</a><br>
<br>
This demo system has about 120,000 records in it but it
could have lots more...<br>
<br>
The manual explains some of the search language
capabilities, such as AND, OR, parenthesis, wildcards
etc...<br>
<br>
But if you type in "smith" it will find all records that
has the word smith in them somewhere.<br>
<br>
If you wanted to find something like smithfields (a meat
market in London) you could type smith* or smithf* for
example.<br>
<br>
Cheers<br>
<br>
Al<br>
<br>
<br>
_______________________________________________<br>
<span style="white-space: nowrap;"><a
moz-do-not-send="true" href="http://www.vtiger.com/">http://www.vtiger.com/</a></span></div>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.vtiger.com/">http://www.vtiger.com/</a></pre>
</blockquote>
<br>
<!-- html ignored --><br>
<div class="pre" style="margin: 0; padding: 0; font-family:
monospace">_______________________________________________<br>
<a moz-do-not-send="true" href="http://www.vtiger.com/">http://www.vtiger.com/</a></div>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
<a class="moz-txt-link-freetext" href="http://www.vtiger.com/">http://www.vtiger.com/</a></pre>
</blockquote>
<br>
</body>
</html>