Bug #6497
closedMentat does not store events with missing Node[*].Name
100%
Description
Node[*].Name is not mandatory, however internal Idea converter seems to expect it and crumbles.
2020-09-03 13:50:28,291 mentat-storage.py [38488] ERROR: Component 'storage': Unable to store IDEA message 'mentat-hub.5009.1598934233.799019.42.159097562.idea' into database: 'Traceback (most recent call last): File "/var/mentat/venv/lib/python3.7/site-packages/mentat/daemon/component/storage.py", line 260, in cbk_event_message_process self.event_gateway(daemon, args) File "/var/mentat/venv/lib/python3.7/site-packages/mentat/daemon/component/storage.py", line 104, in _event_insert_bulk self.event_service.insert_event_bulkci(args['idea']) File "/var/mentat/venv/lib/python3.7/site-packages/mentat/services/eventstorage.py", line 951, in wrapped_f return func(other_self, *args, **kwargs) File "/var/mentat/venv/lib/python3.7/site-packages/mentat/services/eventstorage.py", line 1275, in insert_event_bulkci self.cursor.insert_event(idea_event) File "/var/mentat/venv/lib/python3.7/site-packages/mentat/services/eventstorage.py", line 593, in insert_event idea_pgsql = mentat.idea.sqldb.Idea(idea_event) File "/var/mentat/venv/lib/python3.7/site-packages/mentat/idea/sqldb.py", line 178, in __init__ self.node_name = idea_event.get_detectors() File "/var/mentat/venv/lib/python3.7/site-packages/mentat/idea/internal.py", line 264, in get_detectors return [node['Name'] for node in self.get('Node', [])] File "/var/mentat/venv/lib/python3.7/site-packages/mentat/idea/internal.py", line 264, in <listcomp> return [node['Name'] for node in self.get('Node', [])] File "/var/mentat/venv/lib/python3.7/site-packages/typedcols.py", line 192, in __getitem__ return self.data[key] KeyError: 'Name' '
Files
Related issues
Updated by Rajmund Hruška about 4 years ago
- Status changed from New to In Progress
- Assignee changed from Rajmund Hruška to Pavel Kácha
Node.Name is expected even at the database level - the column is set to be not null. What solution should I choose: making the column nullable or rejecting events with missing Node.Name?
Updated by Pavel Kácha about 4 years ago
- Assignee changed from Pavel Kácha to Rajmund Hruška
Hmm, I see... thinking...
It happened in message with two Node fields, and only (last) one of them had Name missing. As for CESNET, it does not make sense to accept events without at least one Node.Name - and it even should not happen, because Warden server checks whether first Node.Name corresponds with client name in certificate.
So it seems to me we can settle on a compromise: Reject events with no Node.Name whatsoever (require at least one), but don't break on events with missing Name in some of the Node's. (Bonus: no change in db needed.)
Updated by Rajmund Hruška about 4 years ago
- Status changed from In Progress to Feedback
- Assignee changed from Rajmund Hruška to Pavel Kácha
I made a change so that Nodes with no names are accepted and stored in database as NULL. If there are no node names, an exception is raised.
Updated by Pavel Kácha about 4 years ago
Wouldn't it make sense to change get_detectors to not return empty Names (None) at all - omit them completely? Code at internal.py:430 would not be needed and sqldb.py:179 would be a tad simpler. Is get_detector used anywhere else, where "change" in semantics may interfere?
Updated by Rajmund Hruška about 4 years ago
Indeed, it gets simpler by not storing empty Names. That was my original idea, but I didn't manage to make it work. Here is the updated code:
Updated by Pavel Kácha about 4 years ago
Ok. What exactly happens when this exception is triggered?
Updated by Rajmund Hruška about 4 years ago
The exception is stored in the log and the message is moved to errors.
Updated by Pavel Kácha about 4 years ago
- Assignee changed from Pavel Kácha to Rajmund Hruška
Could we wedge in some small unit test?
Updated by Rajmund Hruška about 4 years ago
- Assignee changed from Rajmund Hruška to Pavel Kácha
Updated by Rajmund Hruška about 4 years ago
- Status changed from Feedback to Resolved
Updated by Rajmund Hruška about 4 years ago
- Related to Bug #6698: Use of deprecated function in test_sqldb.py added
Updated by Jan Mach over 3 years ago
- Status changed from Resolved to Closed
Closing, so that the version can also be closed.