Bug #2808

Python and batch_size constraint

Added by Radomír Orkáč about 3 years ago. Updated 9 months ago.

Status:ClosedStart date:03/01/2016
Priority:NormalDue date:
Assignee:Jan Mach% Done:

100%

Category:Development - Core
Target version:2.0

Description

V python 3 se vypisuje pouze 101 dokumentu z mongodb, pak se zpracovani jakoby zasekne a pokracuje az po chvilce.


Related issues

Related to Mentat - Task #3752: Migration from MongoDB to PostgreSQL Closed 10/10/2017

History

#1 Updated by Radomír Orkáč about 3 years ago

“Problem” jsem vypozoroval v python 3 na Ubuntu 12.04.
Dokud byl mentat-hub na wheezym, problem se neprojevil.
Na Debian Jessie se tato “vlastnost” opakuje.

The MongoDB server returns the query results in batches. Batch size will not exceed the maximum BSON document size. For most queries, the first batch returns 101 documents or just enough documents to exceed 1 megabyte. Subsequent batch size is 4 megabytes. To override the default size of the batch, see batchSize() and limit().

For queries that include a sort operation without an index, the server must load all the documents in memory to perform the sort and will return all documents in the first batch.

As you iterate through the cursor and reach the end of the returned batch, if there are more results, cursor.next() will perform a getmore operation to retrieve the next batch.

Tucny vysledek je ve vterinach a doklada, ze jakmile se dosahne limitu 101 dokumentu, vse se zpomali.

  1. ./mongotest -f -r 100 -q -c cache.cfg
    I am going to use MongoDB version 24
    Total elapsed time of script: 0:00:01.726564
  1. ./mongotest -f -r 102 -q -c cache.cfg
    I am going to use MongoDB version 24
    Total elapsed time of script: 0:00:46.054243

Zvetsim-li limit, je to lepsi:
cursor = alerts.find(f[“filter"]).batch_size(200)

  1. ./mongotest -f -r 102 -q -c cache.cfg
    I am going to use MongoDB version 24
    Total elapsed time of script: 0:00:03.790171

#2 Updated by Radomír Orkáč about 3 years ago

Pharook:
Chápu to dobře? Zmiňuješ tam i něco o úpravě kódu, ale nevím přesně, co tím myslíš
- úpravou pythonového kódu se to asi nevyřeší, ne?

Upravou kodu myslim toto:
cursor = alerts.find(f[“filter"]).batch_size(200)

#3 Updated by Radomír Orkáč about 3 years ago

  • Status changed from In Progress to Feedback

#4 Updated by Pavel Kácha about 3 years ago

  • Assignee changed from Radomír Orkáč to Anonymous

Tohle by si chtělo poznamenat pro budoucí pythoní verzi. Meku, jak nazveme pythoní verzí tady v Redmine? 3.0? Nechal bych v ní tenhle bug otevřený, ať na to nezapomeneme.

#5 Updated by Jan Mach about 2 years ago

  • Tracker changed from Task to Bug
  • Subject changed from Python a batch_size omezeni to Python and batch_size constraint
  • Category changed from Development - Tools to Development - Core
  • Target version set to 2.0

#6 Updated by Jan Mach 9 months ago

  • Status changed from Feedback to Closed
  • Assignee set to Jan Mach
  • % Done changed from 0 to 100

After successfull migration from MongoDB to PostgreSQL is this issue not relevant anymore.

#7 Updated by Jan Mach 9 months ago

  • Related to Task #3752: Migration from MongoDB to PostgreSQL added

Also available in: Atom PDF