Mentat

Just a remark, according to [1], tmpfs content can be swapped to disk if low on memory, so in the end we might only save on inode manupulation. Nevertheless, our import pipeline is the intended use case of tmpfs, so this was a good idea. Also, according to [1], no RAM is actually wasted if the ramdisk is empty, the ramdisk size is technically just an upper limit.

[1] https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

Radko Krkoš wrote:

Just a remark, according to [1], tmpfs content can be swapped to disk if low on memory, so in the end we might only save on inode manupulation. Nevertheless, our import pipeline is the intended use case of tmpfs, so this was a good idea. Also, according to [1], no RAM is actually wasted if the ramdisk is empty, the ramdisk size is technically just an upper limit.

[1] https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

Thank you for the remarks, I think I will add them to the documentation page, because then it will be clear we have taken this into consideration.

Radko, please feel free to extend and update the documentation as you see fit. This is not the fist time you have provided valuable information and I am not sure we have written all of it to appropriate documentation pages (I am talking specifically about your outstanding database work).

I think we won't get much more "observations" about benefits/drawbacks on mentat-alt. I'd go for pushing this to production and observe behaviour during crises (maybe ask Dan to turn the Proki on again *grin*). Your opinion, Radko?

Pavel Kácha wrote:

I think we won't get much more "observations" about benefits/drawbacks on mentat-alt. I'd go for pushing this to production and observe behaviour during crises (maybe ask Dan to turn the Proki on again *grin*). Your opinion, Radko?

I would still like to test this. Quite a bit can be learned by looking at io-wait and user times. This got deferred because the amount of data on mentat-alt fitted into the memory and I could not model the out-of-cache scenario reliably. The amount of data has risen since, we can proceed with the observations. Also, reusing the "Proki load-test" and monitoring the import behaviour on mentat-hub and mentat-alt in parallel could be highly beneficial.
That being said, the drawbacks have been looked into extensively and the only one remaining is ensuring that import is re-run since the last commited event to not lose data. I would like a solution for that. Except this, I see no other drawbacks, as was already discussed.
If this issue blocks the 2.3 release, please move it to the next. I think it is important to have an understanding of the performance difference before rolling it out to world/production.

Radko Krkoš wrote:

Also, reusing the "Proki load-test" and monitoring the import behaviour on mentat-hub and mentat-alt in parallel could be highly beneficial.

Ok. I have chatted with Dan right now, he is eager to hit us again. Radko, please, fire up some mail to him (dans), Mek (would be find to have him around for possible mitigation on Mentat-hub) and me and let's come up with time.

That being said, the drawbacks have been looked into extensively and the only one remaining is ensuring that import is re-run since the last commited event to not lose data. I would like a solution for that.

Not sure I get it, what do you mean by ensuring import re-run?

Pavel Kácha wrote:

Radko Krkoš wrote:

Also, reusing the "Proki load-test" and monitoring the import behaviour on mentat-hub and mentat-alt in parallel could be highly beneficial.

Ok. I have chatted with Dan right now, he is eager to hit us again. Radko, please, fire up some mail to him (dans), Mek (would be find to have him around for possible mitigation on Mentat-hub) and me and let's come up with time.

Will do.

That being said, the drawbacks have been looked into extensively and the only one remaining is ensuring that import is re-run since the last commited event to not lose data. I would like a solution for that.

Not sure I get it, what do you mean by ensuring import re-run?

If the data in the Mentat import pipeline is stored on volatile memory, all is lost on reboot. A mechanism needs to be in place that allows reading the data from Warden since the last successfully commited event. Otherwise there will be a window of lost events.

Radko Krkoš wrote:

If the data in the Mentat import pipeline is stored on volatile memory, all is lost on reboot. A mechanism needs to be in place that allows reading the data from Warden since the last successfully commited event. Otherwise there will be a window of lost events.

Ah, see what you mean. This is not easy, that would need some mechanism which feeds last Warden id from mentat-storage back to warden-filer - but there is no info, or rather notion of warden event id in the pipeline after warden-filer.

We could do some dances with init scripts (like save ramdisks' contents when shutting down, recover on start), but not sure if its worth it. If system hits the pavement, we cannot do much.

I have a quite nice solution in mind, but please let's discuss this on a VC, it is too lengthy to explain properly in writing. This issue is on the agenda for the next meeting anyways.

Radko Krkoš wrote:

I have a quite nice solution in mind, but please let's discuss this on a VC, it is too lengthy to explain properly in writing. This issue is on the agenda for the next meeting anyways.

Wrap-up from 2019-02-07: We could learn Warden server to get id by event id, or even replace Warden numeric last-id by event id. Then we could push some sophistication into filer initscripts - forget your own notion of id and get last event id for querying Warden server from Mentat alerts database.

But - seems like lot of work for future, let's stay realistic for now - if we lose, we lose.

Easier (and less reliable) way could be to save non-empty contents of ramdisks during system shutdown and restore during boot. Saving must run after all Mentat daemons (using filesystem queues) are already down, restoring can be parallel. Not sure we have made a decision. Is it worth it? Opinions?

Another option that was tested during mentat-alt preparation for stress-testing (PROKI) is shutting down warden-filer first and waiting for the Mentat import pipeline to finish processing. This generally takes only few seconds and covers the same use cases as saving the RAMdisk to non-volatile storage. It should also be comparatively easy to implement. Maybe by adding another shutdown signal?
System restarts are infrequent and the added shutdown latency will not matter significantly. Of course this still leaves the "recovery from improper shutdown" use case which would require a more complex solution such as the one discussed.

1. The RAM disk approach seems to lead to lower total writes (about 1/2). This seems counter-intuitive and has not been analyzed further yet.
2. The sheer number of events per second is not the defining factor for overload. The case of extra large events seems to be the true problem. This was not encountered during the test. The expectation is that the RAMdisk would help in this case.

Let's discuss this sometimes, but that should not be viewed as a blocker for deployment on production.

Radko Krkoš wrote:

Another option that was tested during mentat-alt preparation for stress-testing (PROKI) is shutting down warden-filer first and waiting for the Mentat import pipeline to finish processing. This generally takes only few seconds and covers the same use cases as saving the RAMdisk to non-volatile storage. It should also be comparatively easy to implement. Maybe by adding another shutdown signal?
System restarts are infrequent and the added shutdown latency will not matter significantly. Of course this still leaves the "recovery from improper shutdown" use case which would require a more complex solution such as the one discussed.

That's it, one hung daemon in the pipeline would mean it does not work. When writing/changing start/shutdown scripts already, simple cp -r (or find ... | cp ...) seems quite simple to me...

Pavel Kácha wrote:

That's it, one hung daemon in the pipeline would mean it does not work.

True. Is that worse than current state? Does this ever happen? (I am just curious).

When writing/changing start/shutdown scripts already, simple cp -r (or find ... | cp ...) seems quite simple to me...

I cannot disagree.

Radko Krkoš wrote:

Pavel Kácha wrote:

That's it, one hung daemon in the pipeline would mean it does not work.

True. Is that worse than current state? Does this ever happen? (I am just curious).

I hope that during standard operation not. However reboot can be sometimes used as a last resort measure in the case of hard-to-debug-or-solve problems, where something in system is congested or hung.

When writing/changing start/shutdown scripts already, simple cp -r (or find ... | cp ...) seems quite simple to me...

I cannot disagree.

... aand seems to me that simpler "copy out - copy in" solves both cases.

After a brief talk about draining queues:

• we could use different signal for graceful stop (finish queue first, then shutdown)
  • either replace SIGUSR1
  • or use SIGINT (Ctrl-C) for immediate shutdown (so user is able to stop the script immediately when run from command line as one-shot) and change SIGTERM to do immediate shutdown (finish what is in memory or rollback)
• then we can have default target in controller for graceful stop and another for immediate stop
• filer would need some more care - then we can learn him to also use these signals, until then some timeouts may work
• we would have to create dependencies (fetching filer shuts first, mentat second, sending filer last)

(Doesn't cp out and cp in look better now? )