Project

General

Profile

Actions

Bug #7591

closed

Tests fail because remaining connection slots are reserved for non-replication superuser connections

Added by Rajmund Hruska 6 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Testing
Target version:
Start date:
06/06/2022
Due date:
% Done:

0%

Estimated time:
To be discussed:
No

Description

This happens both in vagrantbox and GitLab CI/CD.

======================================================================
ERROR: Perform the tests of module service.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect
    return fn()
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
    return _ConnectionFairy._checkout(self)
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 773, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 657, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 652, in __connect
    connection = pool._invoke_creator(self)
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/var/mentat/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 488, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/var/mentat/venv/lib/python3.7/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "localhost" (::1), port 5432 failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections


Files

Actions #1

Updated by Rajmund Hruska 6 months ago

Alchemist/Buildbot has in the /etc/postgresql/12/main/postgresql.conf max_connections = 200 and default is 100.

Actions #2

Updated by Rajmund Hruska 6 months ago

  • Status changed from New to Feedback
  • Assignee set to Rajmund Hruska
  • To be discussed changed from No to Yes

A simple solution is just changing the number in configuration file like this: sed -i '/max_connections/c\max_connections = 200' /etc/postgresql/13/main/postgresql.conf. I used that in the GitLab CI/CD config file and it can be added in the vagrant script. Is that good enough?

Actions #3

Updated by Rajmund Hruska 6 months ago

In the Vagrant, I changed max_connections to 25, I ran tests and then I executed SELECT * FROM pg_stat_activity; in psql. It seems that the connections are in the state idle as shown on the attached picture.

Actions #4

Updated by Radko Krkoš 6 months ago

Rajmund Hruska wrote in #note-3:

In the Vagrant, I changed max_connections to 25, I ran tests and then I executed SELECT * FROM pg_stat_activity; in psql. It seems that the connections are in the state idle as shown on the attached picture.

Based on the successive timestamps "state_change" it seems that connection is made for each test separately and then not closed.

Actions #5

Updated by Rajmund Hruska 6 months ago

It seems that the problem lies in the lib/hawat/test/__init__.py. I put the session.close() at in the teardown_db method but that didn't help.

def teardown_db(self):
        with self.app.app_context():
            hawat.db.db_get().drop_all()
            hawat.db.db_session().close()
Actions #6

Updated by Rajmund Hruska 6 months ago

The issue is also mentioned here.

Actions #7

Updated by Rajmund Hruska 6 months ago

Honestly, I don't know how to solve this one. Disposing of connection pool with hawat.db.db_get().get_engine(self.app).dispose() also didn't help.

Actions #8

Updated by Radko Krkoš 6 months ago

Rajmund Hruska wrote in #note-7:

Honestly, I don't know how to solve this one. Disposing of connection pool with hawat.db.db_get().get_engine(self.app).dispose() also didn't help.

OK. Do you want someone else to look at it?

Actions #9

Updated by Rajmund Hruska 6 months ago

Radko Krkoš wrote in #note-8:

Rajmund Hruska wrote in #note-7:

Honestly, I don't know how to solve this one. Disposing of connection pool with hawat.db.db_get().get_engine(self.app).dispose() also didn't help.

OK. Do you want someone else to look at it?

Yes, that would be very helpful.

Actions #10

Updated by Rajmund Hruska 5 months ago

  • Assignee changed from Rajmund Hruska to Radko Krkoš
Actions #11

Updated by Pavel Kácha 4 months ago

  • Target version changed from 2.10 to 2.11
Actions #12

Updated by Radko Krkoš 4 months ago

  • To be discussed changed from Yes to No
Actions #13

Updated by Rajmund Hruska 4 months ago

I ran the tests with nose2 and unittest and default Postgres configuration file, where max_connections = 100. I didn't come across the problem with the connections, so it really is an issue with nose. I really thought I had the same issue when I ran tests with nose2 a couple of weeks ago, but it seems fine now.

Actions #14

Updated by Radko Krkoš 4 months ago

Rajmund Hruska wrote in #note-13:

I ran the tests with nose2 and unittest and default Postgres configuration file, where max_connections = 100. I didn't come across the problem with the connections, so it really is an issue with nose. I really thought I had the same issue when I ran tests with nose2 a couple of weeks ago, but it seems fine now.

Good news then.

Actions #15

Updated by Pavel Kácha 4 months ago

  • Target version changed from 2.11 to 2.10
Actions #16

Updated by Pavel Kácha 4 months ago

  • Status changed from Feedback to Closed
Actions

Also available in: Atom PDF