Bug #2808
closedPython and batch_size constraint
100%
Description
V python 3 se vypisuje pouze 101 dokumentu z mongodb, pak se zpracovani jakoby zasekne a pokracuje az po chvilce.
Related issues
Updated by Radomír Orkáč over 8 years ago
"Problem" jsem vypozoroval v python 3 na Ubuntu 12.04.
Dokud byl mentat-hub na wheezym, problem se neprojevil.
Na Debian Jessie se tato "vlastnost" opakuje.
The MongoDB server returns the query results in batches. Batch size will not exceed the maximum BSON document size. For most queries, the first batch returns 101 documents or just enough documents to exceed 1 megabyte. Subsequent batch size is 4 megabytes. To override the default size of the batch, see batchSize() and limit(). For queries that include a sort operation without an index, the server must load all the documents in memory to perform the sort and will return all documents in the first batch. As you iterate through the cursor and reach the end of the returned batch, if there are more results, cursor.next() will perform a getmore operation to retrieve the next batch.
Tucny vysledek je ve vterinach a doklada, ze jakmile se dosahne limitu 101 dokumentu, vse se zpomali.
- ./mongotest -f -r 100 -q -c cache.cfg
I am going to use MongoDB version 24
Total elapsed time of script: 0:00:01.726564
- ./mongotest -f -r 102 -q -c cache.cfg
I am going to use MongoDB version 24
Total elapsed time of script: 0:00:46.054243
Zvetsim-li limit, je to lepsi:
cursor = alerts.find(f["filter"]).batch_size(200)
- ./mongotest -f -r 102 -q -c cache.cfg
I am going to use MongoDB version 24
Total elapsed time of script: 0:00:03.790171
Updated by Radomír Orkáč over 8 years ago
Pharook:
Chápu to dobře? Zmiňuješ tam i něco o úpravě kódu, ale nevím přesně, co tím myslíš
- úpravou pythonového kódu se to asi nevyřeší, ne?
Upravou kodu myslim toto:
cursor = alerts.find(f["filter"]).batch_size(200)
Updated by Radomír Orkáč over 8 years ago
- Status changed from In Progress to Feedback
Updated by Pavel Kácha over 8 years ago
- Assignee changed from Radomír Orkáč to Anonymous
Tohle by si chtělo poznamenat pro budoucí pythoní verzi. Meku, jak nazveme pythoní verzí tady v Redmine? 3.0? Nechal bych v ní tenhle bug otevřený, ať na to nezapomeneme.
Updated by Jan Mach over 7 years ago
- Tracker changed from Task to Bug
- Subject changed from Python a batch_size omezeni to Python and batch_size constraint
- Category changed from Development - Tools to Development - Core
- Target version set to 2.0
Updated by Jan Mach about 6 years ago
- Status changed from Feedback to Closed
- Assignee set to Jan Mach
- % Done changed from 0 to 100
After successfull migration from MongoDB to PostgreSQL is this issue not relevant anymore.
Updated by Jan Mach about 6 years ago
- Related to Task #3752: Migration from MongoDB to PostgreSQL added