Mentat

The issue seems to stem from #7642.

Out of despair I asked the chatbot.

From the provided outputs, we can observe the following:

• The total function calls in the first output are 301,963,730, while in the second output, it is 1,229,618,712. This indicates a significant increase in the number of function calls when using the updated SQLAlchemy and Flask-SQLAlchemy libraries.

• The cumulative time in the first output is 420.533 seconds, while in the second output, it is 1,645.624 seconds. Again, there is a substantial increase in the execution time when using the updated libraries.

• The most time-consuming functions in both cases are related to the import mechanism and module loading, as shown by the entries with high cumulative time in the importlib._bootstrap and importlib._bootstrap_external modules.

• In the second output (with the updated libraries), there are significantly more calls to the infer method of the astroid.nodex.node_ng module. This indicates that the updated SQLAlchemy and Flask-SQLAlchemy libraries might be performing more type inference or static analysis during linting, which can contribute to the increased execution time.

Based on these observations, it seems that the updated SQLAlchemy and Flask-SQLAlchemy libraries have introduced changes that result in a higher number of function calls and longer execution time during pylint profiling. This can be a result of increased type inference or additional analysis performed by the updated libraries. It may be worth investigating the specific changes made in the updated libraries to understand the impact on pylint's performance and determine if any optimizations can be made.

This would support my conjecture about the libraries being more complex.

The linter grinds also through external libraries (SQLAlchemy/Flask-SQLAlchemy)? Do we need/want it? Can't we narrow down the linting only to "our" code?

Pavel Kácha wrote in #note-4:

The linter grinds also through external libraries (SQLAlchemy/Flask-SQLAlchemy)? Do we need/want it? Can't we narrow down the linting only to "our" code?

I guess that would be an option. When I disabled all checks (python -m pylint --disable=all --rcfile .pylintrc-lib lib), the execution time was about 50 seconds on my computer. Now, I just need to find the 'right' options to disable.

I don't think we can exclude checking the dependencies as it is built in the code and it seems there is no way of disabling that. Here is a link where it's described in more detail.

What we can do is to run pylint in parallel.

From today's meeting:

• we could patch out critical SQLAlchemy use ("compile" usage?) before linting (and patch back after)
• we should make minimal case and raise a bug report to PyLint

Pavel Kácha wrote in #note-8:

From today's meeting:

• we should make minimal case and raise a bug report to PyLint

The bug report actually exists already (I am surprised no one mentioned this before) - https://github.com/pylint-dev/pylint/issues/8150.