https://homeproj.cesnet.cz/https://homeproj.cesnet.cz/httpauth-login/favicon.ico?16194486082016-03-01T08:02:40ZHomeproj: Redmine for CESNETMentat - Bug #2808: Python and batch_size constrainthttps://homeproj.cesnet.cz/issues/2808?journal_id=93372016-03-01T08:02:40ZRadomír Orkáčorkac@cesnet.cz
<ul></ul><p>"Problem" jsem vypozoroval v python 3 na Ubuntu 12.04.<br />Dokud byl mentat-hub na wheezym, problem se neprojevil.<br />Na Debian Jessie se tato "vlastnost" opakuje.</p>
<pre>
The MongoDB server returns the query results in batches. Batch size will not exceed the maximum BSON document size. For most queries, the first batch returns 101 documents or just enough documents to exceed 1 megabyte. Subsequent batch size is 4 megabytes. To override the default size of the batch, see batchSize() and limit().
For queries that include a sort operation without an index, the server must load all the documents in memory to perform the sort and will return all documents in the first batch.
As you iterate through the cursor and reach the end of the returned batch, if there are more results, cursor.next() will perform a getmore operation to retrieve the next batch.
</pre>
<p>Tucny vysledek je ve vterinach a doklada, ze jakmile se dosahne limitu 101 dokumentu, vse se zpomali.</p>
<ol>
<li>./mongotest -f <strong>-r 100</strong> -q -c cache.cfg<br />I am going to use MongoDB version 24<br />Total elapsed time of script: <strong>0:00:01.726564</strong></li>
</ol>
<ol>
<li>./mongotest -f <strong>-r 102</strong> -q -c cache.cfg<br />I am going to use MongoDB version 24<br />Total elapsed time of script: <strong>0:00:46.054243</strong></li>
</ol>
<p>Zvetsim-li limit, je to lepsi:<br /><strong>cursor = alerts.find(f["filter"]).batch_size(200)</strong></p>
<ol>
<li>./mongotest -f <strong>-r 102</strong> -q -c cache.cfg<br />I am going to use MongoDB version 24<br />Total elapsed time of script: <strong>0:00:03.790171</strong></li>
</ol> Mentat - Bug #2808: Python and batch_size constrainthttps://homeproj.cesnet.cz/issues/2808?journal_id=93382016-03-01T08:03:20ZRadomír Orkáčorkac@cesnet.cz
<ul></ul><p>Pharook:<br /><em>Chápu to dobře? Zmiňuješ tam i něco o úpravě kódu, ale nevím přesně, co tím myslíš<br />- úpravou pythonového kódu se to asi nevyřeší, ne?</em></p>
<p>Upravou kodu myslim toto:<br />cursor = alerts.find(f["filter"]).batch_size(200)</p> Mentat - Bug #2808: Python and batch_size constrainthttps://homeproj.cesnet.cz/issues/2808?journal_id=93392016-03-01T08:03:44ZRadomír Orkáčorkac@cesnet.cz
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul> Mentat - Bug #2808: Python and batch_size constrainthttps://homeproj.cesnet.cz/issues/2808?journal_id=93492016-03-01T10:22:04ZPavel Káchaph@cesnet.cz
<ul><li><strong>Assignee</strong> changed from <i>Radomír Orkáč</i> to <i>Anonymous</i></li></ul><p>Tohle by si chtělo poznamenat pro budoucí pythoní verzi. Meku, jak nazveme pythoní verzí tady v Redmine? 3.0? <span class="wiking smiley smiley-smiley" title=":)"></span> Nechal bych v ní tenhle bug otevřený, ať na to nezapomeneme.</p> Mentat - Bug #2808: Python and batch_size constrainthttps://homeproj.cesnet.cz/issues/2808?journal_id=111362017-03-22T15:46:05ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Tracker</strong> changed from <i>Task</i> to <i>Bug</i></li><li><strong>Subject</strong> changed from <i>Python a batch_size omezeni</i> to <i>Python and batch_size constraint</i></li><li><strong>Category</strong> changed from <i>Development - Tools</i> to <i>Development - Core</i></li><li><strong>Target version</strong> set to <i>2.0</i></li></ul> Mentat - Bug #2808: Python and batch_size constrainthttps://homeproj.cesnet.cz/issues/2808?journal_id=146612018-07-27T09:53:51ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Closed</i></li><li><strong>Assignee</strong> set to <i>Jan Mach</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>After successfull migration from MongoDB to PostgreSQL is this issue not relevant anymore.</p> Mentat - Bug #2808: Python and batch_size constrainthttps://homeproj.cesnet.cz/issues/2808?journal_id=146622018-07-27T09:54:01ZJan Machjan.mach@cesnet.cz
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-5 priority-4 priority-default closed" href="/issues/3752">Task #3752</a>: Migration from MongoDB to PostgreSQL</i> added</li></ul>