Project

General

Profile

Actions

Bug #1460

closed

Kešování selektorů pro alerts

Added by Pavel Kácha over 10 years ago. Updated over 10 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
Development - GUI
Target version:
Start date:
03/25/2014
Due date:
% Done:

100%

Estimated time:
To be discussed:

Description

Kesovani selektoru pro alerts se zmenilo a to tak, ze jsem zacal
pouzivat concat:

source => '$Alert.Analyzer.@analyzerid',
name => '$Alert.Analyzer.@name'
'$concat' => ['$source', '/', '$name']

Kesovaci skript bohuzel nyni konci s timeoutem (testovano na
mentat-dev), coz v minulosti nekoncil:
Selectors:
Unable call aggregate function of MongoDB: recv timed out (800000 ms)
at /usr/local/lib/perl/5.14.2/MongoDB/Cursor.pm line 160.

Aktualni stav pipeline:

my $class = [
{
'$project' => {
source => '$Alert.Classification.@text'
}
},
{
'$group' => {
_id => '$source'
}
},
{
'$sort' => { sum => -1 }
},
];

my $detector = [
{
'$project' => {
source => '$Alert.Analyzer.@analyzerid',
name => '$Alert.Analyzer.@name'
}
},
{
'$group' => {
_id => {
'$concat' => ['$source', '/', '$name']
}
}
},
{
'$sort' => { sum => -1 }
},
];

Pipeline pro $Alert.Classification.@text je jednoduche nahradit, ale
to az tak treba neni:
db.alerts.distinct('Alert.Analyzer.@name')

Pipeline pro '$concat' => ['$source', '/', '$name'] bych asi doplnil o
match (posledni mesic, nebo posledni 3 mesice).
Vytvoril jsem si i jiny dotaz v mongu, ale bez podminky na cas to je
opet nepouzitelne.

db.alerts.group( {
key: { 'Alert.Analyzer.@analyzerid' : 1, 'Alert.Analyzer.@name' : 1},
cond: {ts_u: {$gte: 1394751643}},
reduce: function ( curr, result ) { },
initial: { }
})

Mozna by bylo nejlepsi to spoustet tydne a ziskane selektory jen doplnovat.
Cili jednou na zacatku to spustit pres vsechna data a pak je tydne
doplnovat.

Actions #1

Updated by Pavel Kácha over 10 years ago

Zkusit si říct o id distinct a pak pro jednotlivá id pro name distinct.

Pokud i to bude dlouhé, tak tři měsíce budou muset stačit.

Actions #2

Updated by Pavel Kácha over 10 years ago

  • Priority changed from Normal to High
Actions #3

Updated by Radomír Orkáč over 10 years ago

  • Status changed from New to Feedback

Hotovo, pushnuto.

Pomohly i indexy, ktere jsem zapsal do Message/Storage/Mongo.db (tykaji se kolekce Alerts). Chybely dve z techto (uz nevim ktere):

                           {'Alert.Analyzer.@name' => 1}, # creating selectors
                           {'Alert.Classification.@text' => 1}, # creating selectors
                           {'Alert.Analyzer.@analyzerid' => 1}, # creating selector

Na vsb se "kesovani" zrychlylo o 9s (z 13s kleslo na 4s).
Na devu (nize uvedeny vypis) to uz nestihalo dojet, nyni to trva neco pres 2 minuty.

---
Selectors:
$VAR1 = {
          'alert-class-select' => [
                                    '(D)DoS',
                                    'Botnet Command and Control',
                                    'Botnet Drone',
                                    'Bruteforce',
                                    'Connection attempt',
                                    'EPMAPPER exploitation attempt',
                                    'Malware',
                                    'Open DNS Resolver',
                                    'Open Proxy',
                                    'Other',
                                    'Portscan',
                                    'Probe',
                                    'Remote Login',
                                    'Remote login',
                                    'SMB exploitation attempt',
                                    'SQL query attack attempt',
                                    'SQL query attempt',
                                    'Sandbox URL',
                                    'Spam',
                                    'Spam URL',
                                    'URL attack attempt',
                                    'Webattack'
                                  ],
          'alert-detector-select' => [
                                       'au1/SSERV',
                                       'au1-pickup/LaBrea',
                                       'au2-pickup/LaBrea',
                                       'au2-pickup/sshd',
                                       'holly-pickup/Kippo',
                                       'kryten-pickup/Dionaea',
                                       'mentat-wardenin/DionaeaTUL',
                                       'mentat-wardenin/HPScan',
                                       'mentat-wardenin/HostStats',
                                       'mentat-wardenin/KippoTUL',
                                       'mentat-wardenin/hihat',
                                       'mentat-wardenin/honeyscan',
                                       'mentat-wardenin/labrea'
                                     ]
        };
---

real    2m13.010s
user    0m0.632s
sys     0m0.044s

Actions #4

Updated by Pavel Kácha over 10 years ago

  • Assignee changed from Radomír Orkáč to Jan Mach

Potřeba vytvořit indexy.

Actions #5

Updated by Jan Mach over 10 years ago

  • Status changed from Feedback to Closed
  • % Done changed from 0 to 100

Vybuildil jsem chybějící indexy na ostrém serveru (chyběl pouze Alert.Analyzer.@analyzerid), úkol zavírám.

Actions

Also available in: Atom PDF