Project

General

Profile

Actions

Bug #2808

closed

Python and batch_size constraint

Added by Radomír Orkáč about 8 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Development - Core
Target version:
Start date:
03/01/2016
Due date:
% Done:

100%

Estimated time:
To be discussed:

Description

V python 3 se vypisuje pouze 101 dokumentu z mongodb, pak se zpracovani jakoby zasekne a pokracuje az po chvilce.


Related issues

Related to Mentat - Task #3752: Migration from MongoDB to PostgreSQLClosedJan Mach10/10/2017

Actions
Actions #1

Updated by Radomír Orkáč about 8 years ago

"Problem" jsem vypozoroval v python 3 na Ubuntu 12.04.
Dokud byl mentat-hub na wheezym, problem se neprojevil.
Na Debian Jessie se tato "vlastnost" opakuje.

The MongoDB server returns the query results in batches. Batch size will not exceed the maximum BSON document size. For most queries, the first batch returns 101 documents or just enough documents to exceed 1 megabyte. Subsequent batch size is 4 megabytes. To override the default size of the batch, see batchSize() and limit().

For queries that include a sort operation without an index, the server must load all the documents in memory to perform the sort and will return all documents in the first batch.

As you iterate through the cursor and reach the end of the returned batch, if there are more results, cursor.next() will perform a getmore operation to retrieve the next batch.

Tucny vysledek je ve vterinach a doklada, ze jakmile se dosahne limitu 101 dokumentu, vse se zpomali.

  1. ./mongotest -f -r 100 -q -c cache.cfg
    I am going to use MongoDB version 24
    Total elapsed time of script: 0:00:01.726564
  1. ./mongotest -f -r 102 -q -c cache.cfg
    I am going to use MongoDB version 24
    Total elapsed time of script: 0:00:46.054243

Zvetsim-li limit, je to lepsi:
cursor = alerts.find(f["filter"]).batch_size(200)

  1. ./mongotest -f -r 102 -q -c cache.cfg
    I am going to use MongoDB version 24
    Total elapsed time of script: 0:00:03.790171
Actions #2

Updated by Radomír Orkáč about 8 years ago

Pharook:
Chápu to dobře? Zmiňuješ tam i něco o úpravě kódu, ale nevím přesně, co tím myslíš
- úpravou pythonového kódu se to asi nevyřeší, ne?

Upravou kodu myslim toto:
cursor = alerts.find(f["filter"]).batch_size(200)

Actions #3

Updated by Radomír Orkáč about 8 years ago

  • Status changed from In Progress to Feedback
Actions #4

Updated by Pavel Kácha about 8 years ago

  • Assignee changed from Radomír Orkáč to Anonymous

Tohle by si chtělo poznamenat pro budoucí pythoní verzi. Meku, jak nazveme pythoní verzí tady v Redmine? 3.0? Nechal bych v ní tenhle bug otevřený, ať na to nezapomeneme.

Actions #5

Updated by Jan Mach about 7 years ago

  • Tracker changed from Task to Bug
  • Subject changed from Python a batch_size omezeni to Python and batch_size constraint
  • Category changed from Development - Tools to Development - Core
  • Target version set to 2.0
Actions #6

Updated by Jan Mach over 5 years ago

  • Status changed from Feedback to Closed
  • Assignee set to Jan Mach
  • % Done changed from 0 to 100

After successfull migration from MongoDB to PostgreSQL is this issue not relevant anymore.

Actions #7

Updated by Jan Mach over 5 years ago

  • Related to Task #3752: Migration from MongoDB to PostgreSQL added
Actions

Also available in: Atom PDF