Actions
Bug #1638
closedKesovani selektoru
Start date:
05/28/2014
Due date:
% Done:
0%
Estimated time:
To be discussed:
Description
Problem nastava pouze v danou dobu (spousteno cronem).
V jinou dobu ci rucni spusteni je kesovani v poradku.
S Mekem se domluvim, jestli budeme hybat s casem zalohovani (pravdepodobne vytizi disky), nebo s casem kesovani.
Wed May 28 02:00:03 2014 mentat hawat-precache[28298]: NOTICE: hawat-precache has been executed
Wed May 28 02:00:03 2014 mentat hawat-precache[28298]: NOTICE: Start: caching selectors
Wed May 28 02:26:58 2014 mentat hawat-precache[28298]: ERROR: Error in getting detector selector data from db. recv timed out (800000 ms) at /usr/local/lib/perl/5.14.2/MongoDB/Cursor.pm line 160.
Wed May 28 02:29:44 2014 mentat hawat-precache[28298]: NOTICE: Stop: caching selectors, time: 1781.68079400063
Wed May 28 02:29:44 2014 mentat hawat-precache[28298]: NOTICE: Selectors data: $VAR1 = {
'error' => [
'Error in getting detector selector data from db. recv timed out (800000 ms) at /usr/local/lib/perl/5.14.2/MongoDB/Cursor.pm line 160.
'
],
'alert-class-select' => [
'(D)DoS',
'Arakis Report',
'Backscatter Report',
'Bot Zeus P2P Report',
'Botnet B157-N3',
'Botnet B157-O3',
'Botnet B157-R0',
'Botnet B157-R1',
'Botnet B54-CODE',
'Botnet B54-CONFIG',
'Botnet B58-DGA2',
'Botnet B68-1-64',
'Botnet B68-2-32',
'Botnet B68-2-64',
'Botnet B68-DNS',
'Botnet Command and Control',
'Botnet Conficker',
'Botnet Drone',
'Botnet zbot',
'Bots',
'Bots Zeroaccess Report',
'Bruteforce',
'CERT PL Sinkhole Report',
'Connection attempt',
'EPMAPPER exploitation attempt',
'Malware',
'Malware URL',
'Open DNS Resolver',
'Open DNS resolver',
'Open Proxy',
'Other',
'Ping probe',
'Portscan',
'Probe',
'Proxy server',
'Remote Login',
'Remote login',
'SMB exploitation attempt',
'SQL query attack attempt',
'SQL query attempt',
'SYN/ACK scan or DOS attack',
'Sandbox URL',
'Scan CHARGEN',
'Scan NETBIOS',
'Scan NTP',
'Scan QOTD',
'Scan SNMP',
'Scan SSDP',
'Scanners',
'Security vulnerability',
'Spam',
'Spam Report',
'URL attack attempt',
'Webattack'
],
'alert-detector-select' => [
'au1/X2',
'au1/SSERV',
'au1/X4',
'au1/N6',
'au1/UCEPROT',
'au1/LaBrea',
'au1-pickup/LaBrea',
'au2/LaBrea'
]
};
Wed May 28 02:29:44 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00
Wed May 28 02:30:40 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 55.9859671592712
Wed May 28 02:30:40 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00
Wed May 28 02:32:39 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 118.214454889297
Wed May 28 02:32:39 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00
Wed May 28 02:35:33 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 174.206841945648
Wed May 28 02:35:33 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00
Wed May 28 02:39:55 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 262.102180957794
Wed May 28 02:39:55 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00
Wed May 28 02:50:53 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 658.330244779587
Wed May 28 02:50:53 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 147.251.0.0-147.251.255.255
Wed May 28 02:51:32 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 38.4185900688171
Wed May 28 02:51:32 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 195.178.86.0-195.178.87.255
Wed May 28 02:51:34 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 2.36591291427612
Wed May 28 02:51:34 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 2001:718:800:1::/64
Wed May 28 02:52:07 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 32.8774719238281
Wed May 28 02:52:07 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 2001:718:800:5::/64
Wed May 28 02:52:40 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 32.9891510009766
Wed May 28 02:52:40 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 2001:718:801::/48
Wed May 28 02:53:12 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 32.4385488033295
Wed May 28 02:53:12 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 2001:718:805::/48
Wed May 28 02:53:45 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 32.5431821346283
Wed May 28 02:53:45 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 147.251.0.0-147.251.255.255
Wed May 28 02:53:54 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 9.05734801292419
Wed May 28 02:53:54 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 195.178.86.0-195.178.87.255
Wed May 28 02:53:56 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 2.12722611427307
Wed May 28 02:53:56 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 2001:718:800:1::/64
Wed May 28 02:55:16 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 80.073392868042
Wed May 28 02:55:16 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 2001:718:800:5::/64
Wed May 28 02:56:37 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 80.5492069721222
Wed May 28 02:56:37 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 2001:718:801::/48
Wed May 28 02:57:58 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 81.2128429412842
Wed May 28 02:57:58 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 2001:718:805::/48
Wed May 28 02:59:19 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 81.2612619400024
Wed May 28 02:59:19 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 147.251.0.0-147.251.255.255
Wed May 28 02:59:27 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 8.21130180358887
Wed May 28 02:59:27 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 195.178.86.0-195.178.87.255
Wed May 28 02:59:29 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 2.12078499794006
Wed May 28 02:59:29 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 2001:718:800:1::/64
Wed May 28 03:02:20 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 170.63827085495
Wed May 28 03:02:20 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 2001:718:800:5::/64
Wed May 28 03:05:12 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 172.002375125885
Wed May 28 03:05:12 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 2001:718:801::/48
Wed May 28 03:09:51 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 278.838557958603
Wed May 28 03:09:51 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 2001:718:805::/48
Wed May 28 03:14:08 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 256.617415904999
Wed May 28 03:14:08 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 147.251.0.0-147.251.255.255
Wed May 28 03:14:18 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 10.5731279850006
Wed May 28 03:14:18 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 195.178.86.0-195.178.87.255
Wed May 28 03:14:21 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 2.91054487228394
Wed May 28 03:14:21 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 2001:718:800:1::/64
Wed May 28 03:21:11 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 409.90402007103
Wed May 28 03:21:11 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 2001:718:800:5::/64
Wed May 28 03:25:42 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 271.320389986038
Wed May 28 03:25:42 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 2001:718:801::/48
Wed May 28 03:30:13 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 270.326061010361
Wed May 28 03:30:13 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 2001:718:805::/48
Wed May 28 03:34:43 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 270.821288824081
Wed May 28 03:34:43 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 147.251.0.0-147.251.255.255
Wed May 28 03:35:23 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 39.1071059703827
Wed May 28 03:35:23 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 195.178.86.0-195.178.87.255
Wed May 28 03:35:25 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 2.4498929977417
Wed May 28 03:35:25 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 2001:718:800:1::/64
Wed May 28 03:44:55 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 570.433768987656
Wed May 28 03:44:55 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 2001:718:800:5::/64
Wed May 28 03:52:44 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 468.325734138489
Wed May 28 03:52:44 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 2001:718:801::/48
Wed May 28 04:00:19 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 455.121767997742
Wed May 28 04:00:19 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 2001:718:805::/48
Wed May 28 04:08:32 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 492.995522975922
Wed May 28 04:08:32 2014 mentat hawat-precache[28298]: NOTICE: Total finished time: 7709.16201090813
Related issues
Updated by Pavel Kácha almost 10 years ago
Možná by Mek mohl upravit zálohování na ty dumpy inkrementů, o kterých jsme mluvili - pro pro přechod na datová úložiště by to nejspíš stejně dělal. Pak bude záloha řádově rychlejší a nebudeme si pálit procesor/disky.
Updated by Pavel Kácha almost 10 years ago
(Ale domluvte se, co zatím uzná za rozumné - vím, že má hromadu jiné práce. )
Updated by Radomír Orkáč almost 10 years ago
- Status changed from New to In Progress
jj, ja to ale beru z pohledu VC, kde jsme tomu davali malou prioritu:
- příležitostně: zálohování? inkrementy na du
Ja se s nim domluvim.., treba to udelam ja.
V konecne fazi bude stacit neco takoveho (unix epoch), co pouzivam ja pro presun dat na mentat-dev:
# mongodump -d 'mentat' -c 'alerts' -q '{ts_u: {$gte: 1398895200}}'
Updated by Radomír Orkáč almost 10 years ago
Nevim, kam to zapsat, tak to pisu sem.
Nechce se mi otevirat dalsi task.
0 0 * * * /root/bin/backupdb
root@mentat:~# /root/bin/backupdb Tue May 27 00:00:00 CEST 2014 Wed May 28 00:00:00 CEST 2014 connected to: 127.0.0.1 2014-05-28T14:00:02.339+0200 DATABASE: mentat to /var/tmp/20140528/mentat 2014-05-28T14:00:02.340+0200 mentat.alerts to /var/tmp/20140528/mentat/alerts.bson 2014-05-28T14:00:05.001+0200 Collection File Writing Progress: 461400/91832945 0% (documents) 2014-05-28T14:00:08.005+0200 Collection File Writing Progress: 855500/91832945 0% (documents) 2014-05-28T14:00:09.369+0200 1035154 documents 2014-05-28T14:00:09.369+0200 Metadata for mentat.alerts to /var/tmp/20140528/mentat/alerts.metadata.json connected to: 127.0.0.1 2014-05-28T14:00:09.404+0200 DATABASE: mentat to /var/tmp/20140528/mentat 2014-05-28T14:00:09.404+0200 mentat.groups to /var/tmp/20140528/mentat/groups.bson 2014-05-28T14:00:09.413+0200 289 documents 2014-05-28T14:00:09.413+0200 Metadata for mentat.groups to /var/tmp/20140528/mentat/groups.metadata.json connected to: 127.0.0.1 2014-05-28T14:00:09.437+0200 DATABASE: mentat to /var/tmp/20140528/mentat 2014-05-28T14:00:09.437+0200 mentat.users to /var/tmp/20140528/mentat/users.bson 2014-05-28T14:00:09.438+0200 8 documents 2014-05-28T14:00:09.438+0200 Metadata for mentat.users to /var/tmp/20140528/mentat/users.metadata.json tar: Removing leading `/' from member names
root@mentat:~# ls -alh /var/backups/20140528.tgz -rw-r--r-- 1 root root 114M May 28 14:00 /var/backups/20140528.tgz
Updated by Radomír Orkáč almost 10 years ago
Pomohla uprava zalohovaci politiky:
#------------------------- Backup /var
what=(
"."
)
exclude=(
"backups"
"lock"
"lost+found"
"run"
"spool"
"tmp"
# added by Orkac
"lib/mongodb"
)
backup var "/var"
Po nekolika dnech testovani jsem exclude aktivoval az vcera. V noci zalohovani probehlo bez chyb:
Mon Jun 2 03:00:02 2014 mentat hawat-precache[22818]: NOTICE: hawat-precache has been executed
Mon Jun 2 03:00:02 2014 mentat hawat-precache[22818]: NOTICE: Start: caching selectors
Mon Jun 2 03:41:34 2014 mentat hawat-precache[22818]: NOTICE: Stop: caching selectors, time: 2491.93398785591
Mon Jun 2 03:41:34 2014 mentat hawat-precache[22818]: NOTICE: Selectors data: $VAR1 = {
'alert-class-select' => [
'(D)DoS',
'Arakis Report',
'Backscatter Report',
Updated by Radomír Orkáč almost 10 years ago
Opet zacal problem s kesovanim selectoru.
Wed Jun 18 02:28:55 2014 mentat hawat-precache[3472]: NOTICE: Selectors data: $VAR1 = {
'error' => [
'Error in getting detector selector data from db. recv timed out (800000 ms) at /usr/local/lib/perl/5.14.2/MongoDB/Cursor.pm line 160.
'
],
Updated by Pavel Kácha almost 10 years ago
- Related to Bug #1498: hawat - dashboard - nespravne hodnoty u kese 3m added
Updated by Jan Mach about 7 years ago
- Status changed from In Progress to Closed
Již neaktuální, úklid v úkolech, zavírám.
Actions