Mentat

Some additions:

• https://nanomsg.org/ (successor of ZeroMQ, same author)
• https://redis.io/ (supports push/pop, used by IntelMQ)

Heretic thought:

• Message queue within PostgreSQL, while we're already running it..?

Pavel Kácha wrote:

Heretic thought:

• Message queue within PostgreSQL, while we're already running it..?

My intuitive reaction would be a strict NO, as that would never work. But then a simple search provides this: https://gist.github.com/chanks/7585810

Then, if we want to use a SQL DB, we could use sqlite. According to [1], it should work fine. The communication is on localhost only, there is just one writer as it is a pipeline (so exclusive write lock is not an issue), the amount of blocks should be lower than with files (and you can still use caching) - but lower than with PostgreSQL - no WAL.

[1] https://www.sqlite.org/whentouse.html

Radko Krkoš wrote:

Then, if we want to use a SQL DB, we could use sqlite. According to [1], it should work fine. The communication is on localhost only, there is just one writer as it is a pipeline (so exclusive write lock is not an issue), the amount of blocks should be lower than with files (and you can still use caching) - but lower than with PostgreSQL - no WAL.

[1] https://www.sqlite.org/whentouse.html

Sure, we could use all sorts of things. PQ just came to my mind as thing we are already heavily use, and also, you can use different "queues" in compeletely different dbs (and machines, when we are at it). However, should I consider lighter db, I'd go for (for example) Berkeley DB, which is much simpler, solves locking, and provides queueing primitives itself.

I think, that the goal of using some better queue mechanism is also to move to the producer-consumer design pattern, use subscriptions, simplify configuration and enable arbitrary module ordering/coupling/chaining. This will be considerable amount of work, so I think the payout should be worth it. And it should definitely enable process level paralelism, single processes are not enough...we are already running five enrichers to speed up the event enrichment process.

We can of course discuss this at length, but I think, that there is no point of using some overly simple tool and then end up implementing all the safety features yourself on top of that. And we would implement them eventually, we need safe communication protocol without the data loss.

Jan Mach wrote:

I think, that the goal of using some better queue mechanism is also to move to the producer-consumer design pattern

Maybe.

, use subscriptions,

Why? Seems not relevant to me. But I may be wrong.

simplify configuration and enable arbitrary module ordering/coupling/chaining.

Maybe, however not sure about configuration simplification, as I hoped for minimal changes.

This will be considerable amount of work, so I think the payout should be worth it. And it should definitely enable process level paralelism, single processes are not enough...we are already running five enrichers to speed up the event enrichment process.

Maybe.

Do mentioned "simple" tools preclude us from any of these?

We can of course discuss this at length, but I think, that there is no point of using some overly simple tool and then end up implementing all the safety features yourself on top of that. And we would implement them eventually, we need safe communication protocol without the data loss.

And safe we do NOT have now. That's why I'm thinking about something simple enough to replace current state with relative ease.

Couple of forgotten notes from some old discussion:

• do we want memory based one (to avoid disk trashing)?
• do we really want network based one (for possibility of running some daemons on separate machine)?
• do we want broker based one (may bring own ready to use ecosystem and tooling)?
• do we want centralised one (brings single point of failure)?
• do we want some standard protocol based one (brings possible remote interoperability)?