Mentat

From telco brainstorming with Radko - upgrade of the db and code could be done with relatively short downtime (the longest will be CLUSTER, which can be delayed to off-hours).

• create new empty table
• enable API 'extend' code to be able to deal with jsonb in separate table, but resort to jsonb in the original table (to be able to cope with the transition period)
• stop incoming data
• run the last run of the reporter and statistician
• alter table jsonb NULL (allow NULL data in the old table)
• set up new storage code (to split into metadata table and jsonb data table)
• enable incoming data (last column in metadata table now gets NULLs)
• enable reporter and statistician
• repeat: select old jsonb into new jsonb limit XXX; until done (limit is to prevent lock congestion)
• alter table drop old jsonb; vacuum
• CLUSTER when suitable
• get rid of dual code

Statistics gathering was performed to better understand this. Following numbers are for mentat-hub on 20181024.
Current table size: 171GB
Number of rows: 125.3M (125 314 536)
Aggregate size of event BYTEA: 110GB (118 092 392 057) - obtained by SELECT sum(octet_length(event)) FROM events;
Aggregate indices size: 34.8GB (REM: No REINDEX was performed for a long time so the indices are inflated considerably)
ID index size: 11GB (this would be duplicated for the data table)

That leaves us with (171-110) + 34.8 + 11 = 106.8GB per 125.3M of events, effectively 0.85GB per 1M rows. With current physical memory and long term memory load, this would get us to at least 260M events fully cached in RAM. A considerable improvement.