Bug #1460
closedKešování selektorů pro alerts
100%
Description
Kesovani selektoru pro alerts se zmenilo a to tak, ze jsem zacal
pouzivat concat:
source => '$Alert.Analyzer.@analyzerid', name => '$Alert.Analyzer.@name' '$concat' => ['$source', '/', '$name']
Kesovaci skript bohuzel nyni konci s timeoutem (testovano na
mentat-dev), coz v minulosti nekoncil:
Selectors:
Unable call aggregate function of MongoDB: recv timed out (800000 ms)
at /usr/local/lib/perl/5.14.2/MongoDB/Cursor.pm line 160.
Aktualni stav pipeline:
my $class = [
{
'$project' => {
source => '$Alert.Classification.@text'
}
},
{
'$group' => {
_id => '$source'
}
},
{
'$sort' => { sum => -1 }
},
];
my $detector = [
{
'$project' => {
source => '$Alert.Analyzer.@analyzerid',
name => '$Alert.Analyzer.@name'
}
},
{
'$group' => {
_id => {
'$concat' => ['$source', '/', '$name']
}
}
},
{
'$sort' => { sum => -1 }
},
];
Pipeline pro $Alert.Classification.@text je jednoduche nahradit, ale
to az tak treba neni:
db.alerts.distinct('Alert.Analyzer.@name')
Pipeline pro '$concat' => ['$source', '/', '$name'] bych asi doplnil o
match (posledni mesic, nebo posledni 3 mesice).
Vytvoril jsem si i jiny dotaz v mongu, ale bez podminky na cas to je
opet nepouzitelne.
db.alerts.group( {
key: { 'Alert.Analyzer.@analyzerid' : 1, 'Alert.Analyzer.@name' : 1},
cond: {ts_u: {$gte: 1394751643}},
reduce: function ( curr, result ) { },
initial: { }
})
Mozna by bylo nejlepsi to spoustet tydne a ziskane selektory jen doplnovat.
Cili jednou na zacatku to spustit pres vsechna data a pak je tydne
doplnovat.
Updated by Pavel Kácha about 10 years ago
Zkusit si říct o id distinct a pak pro jednotlivá id pro name distinct.
Pokud i to bude dlouhé, tak tři měsíce budou muset stačit.
Updated by Radomír Orkáč about 10 years ago
- Status changed from New to Feedback
Hotovo, pushnuto.
Pomohly i indexy, ktere jsem zapsal do Message/Storage/Mongo.db (tykaji se kolekce Alerts). Chybely dve z techto (uz nevim ktere):
{'Alert.Analyzer.@name' => 1}, # creating selectors
{'Alert.Classification.@text' => 1}, # creating selectors
{'Alert.Analyzer.@analyzerid' => 1}, # creating selector
Na vsb se "kesovani" zrychlylo o 9s (z 13s kleslo na 4s).
Na devu (nize uvedeny vypis) to uz nestihalo dojet, nyni to trva neco pres 2 minuty.
---
Selectors:
$VAR1 = {
'alert-class-select' => [
'(D)DoS',
'Botnet Command and Control',
'Botnet Drone',
'Bruteforce',
'Connection attempt',
'EPMAPPER exploitation attempt',
'Malware',
'Open DNS Resolver',
'Open Proxy',
'Other',
'Portscan',
'Probe',
'Remote Login',
'Remote login',
'SMB exploitation attempt',
'SQL query attack attempt',
'SQL query attempt',
'Sandbox URL',
'Spam',
'Spam URL',
'URL attack attempt',
'Webattack'
],
'alert-detector-select' => [
'au1/SSERV',
'au1-pickup/LaBrea',
'au2-pickup/LaBrea',
'au2-pickup/sshd',
'holly-pickup/Kippo',
'kryten-pickup/Dionaea',
'mentat-wardenin/DionaeaTUL',
'mentat-wardenin/HPScan',
'mentat-wardenin/HostStats',
'mentat-wardenin/KippoTUL',
'mentat-wardenin/hihat',
'mentat-wardenin/honeyscan',
'mentat-wardenin/labrea'
]
};
---
real 2m13.010s
user 0m0.632s
sys 0m0.044s
Updated by Pavel Kácha about 10 years ago
- Assignee changed from Radomír Orkáč to Jan Mach
Potřeba vytvořit indexy.
Updated by Jan Mach about 10 years ago
- Status changed from Feedback to Closed
- % Done changed from 0 to 100
Vybuildil jsem chybějící indexy na ostrém serveru (chyběl pouze Alert.Analyzer.@analyzerid), úkol zavírám.