Bug #2808
closed
Python and batch_size constraint
Added by Radomír Orkáč over 8 years ago.
Updated over 6 years ago.
Category:
Development - Core
Description
V python 3 se vypisuje pouze 101 dokumentu z mongodb, pak se zpracovani jakoby zasekne a pokracuje az po chvilce.
"Problem" jsem vypozoroval v python 3 na Ubuntu 12.04.
Dokud byl mentat-hub na wheezym, problem se neprojevil.
Na Debian Jessie se tato "vlastnost" opakuje.
The MongoDB server returns the query results in batches. Batch size will not exceed the maximum BSON document size. For most queries, the first batch returns 101 documents or just enough documents to exceed 1 megabyte. Subsequent batch size is 4 megabytes. To override the default size of the batch, see batchSize() and limit().
For queries that include a sort operation without an index, the server must load all the documents in memory to perform the sort and will return all documents in the first batch.
As you iterate through the cursor and reach the end of the returned batch, if there are more results, cursor.next() will perform a getmore operation to retrieve the next batch.
Tucny vysledek je ve vterinach a doklada, ze jakmile se dosahne limitu 101 dokumentu, vse se zpomali.
- ./mongotest -f -r 100 -q -c cache.cfg
I am going to use MongoDB version 24
Total elapsed time of script: 0:00:01.726564
- ./mongotest -f -r 102 -q -c cache.cfg
I am going to use MongoDB version 24
Total elapsed time of script: 0:00:46.054243
Zvetsim-li limit, je to lepsi:
cursor = alerts.find(f["filter"]).batch_size(200)
- ./mongotest -f -r 102 -q -c cache.cfg
I am going to use MongoDB version 24
Total elapsed time of script: 0:00:03.790171
Pharook:
Chápu to dobře? Zmiňuješ tam i něco o úpravě kódu, ale nevím přesně, co tím myslíš
- úpravou pythonového kódu se to asi nevyřeší, ne?
Upravou kodu myslim toto:
cursor = alerts.find(f["filter"]).batch_size(200)
- Status changed from In Progress to Feedback
- Assignee changed from Radomír Orkáč to Anonymous
Tohle by si chtělo poznamenat pro budoucí pythoní verzi. Meku, jak nazveme pythoní verzí tady v Redmine? 3.0? Nechal bych v ní tenhle bug otevřený, ať na to nezapomeneme.
- Tracker changed from Task to Bug
- Subject changed from Python a batch_size omezeni to Python and batch_size constraint
- Category changed from Development - Tools to Development - Core
- Target version set to 2.0
- Status changed from Feedback to Closed
- Assignee set to Jan Mach
- % Done changed from 0 to 100
After successfull migration from MongoDB to PostgreSQL is this issue not relevant anymore.
- Related to Task #3752: Migration from MongoDB to PostgreSQL added
Also available in: Atom
PDF