Bug #1638
closedKesovani selektoru
0%
Description
Problem nastava pouze v danou dobu (spousteno cronem).
V jinou dobu ci rucni spusteni je kesovani v poradku.
S Mekem se domluvim, jestli budeme hybat s casem zalohovani (pravdepodobne vytizi disky), nebo s casem kesovani.
Wed May 28 02:00:03 2014 mentat hawat-precache[28298]: NOTICE: hawat-precache has been executed Wed May 28 02:00:03 2014 mentat hawat-precache[28298]: NOTICE: Start: caching selectors Wed May 28 02:26:58 2014 mentat hawat-precache[28298]: ERROR: Error in getting detector selector data from db. recv timed out (800000 ms) at /usr/local/lib/perl/5.14.2/MongoDB/Cursor.pm line 160. Wed May 28 02:29:44 2014 mentat hawat-precache[28298]: NOTICE: Stop: caching selectors, time: 1781.68079400063 Wed May 28 02:29:44 2014 mentat hawat-precache[28298]: NOTICE: Selectors data: $VAR1 = { 'error' => [ 'Error in getting detector selector data from db. recv timed out (800000 ms) at /usr/local/lib/perl/5.14.2/MongoDB/Cursor.pm line 160. ' ], 'alert-class-select' => [ '(D)DoS', 'Arakis Report', 'Backscatter Report', 'Bot Zeus P2P Report', 'Botnet B157-N3', 'Botnet B157-O3', 'Botnet B157-R0', 'Botnet B157-R1', 'Botnet B54-CODE', 'Botnet B54-CONFIG', 'Botnet B58-DGA2', 'Botnet B68-1-64', 'Botnet B68-2-32', 'Botnet B68-2-64', 'Botnet B68-DNS', 'Botnet Command and Control', 'Botnet Conficker', 'Botnet Drone', 'Botnet zbot', 'Bots', 'Bots Zeroaccess Report', 'Bruteforce', 'CERT PL Sinkhole Report', 'Connection attempt', 'EPMAPPER exploitation attempt', 'Malware', 'Malware URL', 'Open DNS Resolver', 'Open DNS resolver', 'Open Proxy', 'Other', 'Ping probe', 'Portscan', 'Probe', 'Proxy server', 'Remote Login', 'Remote login', 'SMB exploitation attempt', 'SQL query attack attempt', 'SQL query attempt', 'SYN/ACK scan or DOS attack', 'Sandbox URL', 'Scan CHARGEN', 'Scan NETBIOS', 'Scan NTP', 'Scan QOTD', 'Scan SNMP', 'Scan SSDP', 'Scanners', 'Security vulnerability', 'Spam', 'Spam Report', 'URL attack attempt', 'Webattack' ], 'alert-detector-select' => [ 'au1/X2', 'au1/SSERV', 'au1/X4', 'au1/N6', 'au1/UCEPROT', 'au1/LaBrea', 'au1-pickup/LaBrea', 'au2/LaBrea' ] }; Wed May 28 02:29:44 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00 Wed May 28 02:30:40 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 55.9859671592712 Wed May 28 02:30:40 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00 Wed May 28 02:32:39 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 118.214454889297 Wed May 28 02:32:39 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00 Wed May 28 02:35:33 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 174.206841945648 Wed May 28 02:35:33 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00 Wed May 28 02:39:55 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 262.102180957794 Wed May 28 02:39:55 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00 Wed May 28 02:50:53 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 658.330244779587 Wed May 28 02:50:53 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 147.251.0.0-147.251.255.255 Wed May 28 02:51:32 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 38.4185900688171 Wed May 28 02:51:32 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 195.178.86.0-195.178.87.255 Wed May 28 02:51:34 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 2.36591291427612 Wed May 28 02:51:34 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 2001:718:800:1::/64 Wed May 28 02:52:07 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 32.8774719238281 Wed May 28 02:52:07 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 2001:718:800:5::/64 Wed May 28 02:52:40 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 32.9891510009766 Wed May 28 02:52:40 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 2001:718:801::/48 Wed May 28 02:53:12 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 32.4385488033295 Wed May 28 02:53:12 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-27 00:00:00, 2001:718:805::/48 Wed May 28 02:53:45 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 32.5431821346283 Wed May 28 02:53:45 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 147.251.0.0-147.251.255.255 Wed May 28 02:53:54 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 9.05734801292419 Wed May 28 02:53:54 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 195.178.86.0-195.178.87.255 Wed May 28 02:53:56 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 2.12722611427307 Wed May 28 02:53:56 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 2001:718:800:1::/64 Wed May 28 02:55:16 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 80.073392868042 Wed May 28 02:55:16 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 2001:718:800:5::/64 Wed May 28 02:56:37 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 80.5492069721222 Wed May 28 02:56:37 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 2001:718:801::/48 Wed May 28 02:57:58 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 81.2128429412842 Wed May 28 02:57:58 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-25 00:00:00, 2001:718:805::/48 Wed May 28 02:59:19 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 81.2612619400024 Wed May 28 02:59:19 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 147.251.0.0-147.251.255.255 Wed May 28 02:59:27 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 8.21130180358887 Wed May 28 02:59:27 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 195.178.86.0-195.178.87.255 Wed May 28 02:59:29 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 2.12078499794006 Wed May 28 02:59:29 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 2001:718:800:1::/64 Wed May 28 03:02:20 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 170.63827085495 Wed May 28 03:02:20 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 2001:718:800:5::/64 Wed May 28 03:05:12 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 172.002375125885 Wed May 28 03:05:12 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 2001:718:801::/48 Wed May 28 03:09:51 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 278.838557958603 Wed May 28 03:09:51 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-21 00:00:00, 2001:718:805::/48 Wed May 28 03:14:08 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 256.617415904999 Wed May 28 03:14:08 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 147.251.0.0-147.251.255.255 Wed May 28 03:14:18 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 10.5731279850006 Wed May 28 03:14:18 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 195.178.86.0-195.178.87.255 Wed May 28 03:14:21 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 2.91054487228394 Wed May 28 03:14:21 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 2001:718:800:1::/64 Wed May 28 03:21:11 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 409.90402007103 Wed May 28 03:21:11 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 2001:718:800:5::/64 Wed May 28 03:25:42 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 271.320389986038 Wed May 28 03:25:42 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 2001:718:801::/48 Wed May 28 03:30:13 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 270.326061010361 Wed May 28 03:30:13 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-05-14 00:00:00, 2001:718:805::/48 Wed May 28 03:34:43 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 270.821288824081 Wed May 28 03:34:43 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 147.251.0.0-147.251.255.255 Wed May 28 03:35:23 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 39.1071059703827 Wed May 28 03:35:23 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 195.178.86.0-195.178.87.255 Wed May 28 03:35:25 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 2.4498929977417 Wed May 28 03:35:25 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 2001:718:800:1::/64 Wed May 28 03:44:55 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 570.433768987656 Wed May 28 03:44:55 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 2001:718:800:5::/64 Wed May 28 03:52:44 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 468.325734138489 Wed May 28 03:52:44 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 2001:718:801::/48 Wed May 28 04:00:19 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 455.121767997742 Wed May 28 04:00:19 2014 mentat hawat-precache[28298]: NOTICE: Start: 2014-05-28 00:00:00 - 2014-04-28 00:00:00, 2001:718:805::/48 Wed May 28 04:08:32 2014 mentat hawat-precache[28298]: NOTICE: Finished time: 492.995522975922 Wed May 28 04:08:32 2014 mentat hawat-precache[28298]: NOTICE: Total finished time: 7709.16201090813
Related issues
Updated by Pavel Kácha over 10 years ago
Možná by Mek mohl upravit zálohování na ty dumpy inkrementů, o kterých jsme mluvili - pro pro přechod na datová úložiště by to nejspíš stejně dělal. Pak bude záloha řádově rychlejší a nebudeme si pálit procesor/disky.
Updated by Pavel Kácha over 10 years ago
(Ale domluvte se, co zatím uzná za rozumné - vím, že má hromadu jiné práce. )
Updated by Radomír Orkáč over 10 years ago
- Status changed from New to In Progress
- příležitostně: zálohování? inkrementy na du
Ja se s nim domluvim.., treba to udelam ja.
V konecne fazi bude stacit neco takoveho (unix epoch), co pouzivam ja pro presun dat na mentat-dev:
# mongodump -d 'mentat' -c 'alerts' -q '{ts_u: {$gte: 1398895200}}'
Updated by Radomír Orkáč over 10 years ago
Nevim, kam to zapsat, tak to pisu sem.
Nechce se mi otevirat dalsi task.
0 0 * * * /root/bin/backupdb
root@mentat:~# /root/bin/backupdb Tue May 27 00:00:00 CEST 2014 Wed May 28 00:00:00 CEST 2014 connected to: 127.0.0.1 2014-05-28T14:00:02.339+0200 DATABASE: mentat to /var/tmp/20140528/mentat 2014-05-28T14:00:02.340+0200 mentat.alerts to /var/tmp/20140528/mentat/alerts.bson 2014-05-28T14:00:05.001+0200 Collection File Writing Progress: 461400/91832945 0% (documents) 2014-05-28T14:00:08.005+0200 Collection File Writing Progress: 855500/91832945 0% (documents) 2014-05-28T14:00:09.369+0200 1035154 documents 2014-05-28T14:00:09.369+0200 Metadata for mentat.alerts to /var/tmp/20140528/mentat/alerts.metadata.json connected to: 127.0.0.1 2014-05-28T14:00:09.404+0200 DATABASE: mentat to /var/tmp/20140528/mentat 2014-05-28T14:00:09.404+0200 mentat.groups to /var/tmp/20140528/mentat/groups.bson 2014-05-28T14:00:09.413+0200 289 documents 2014-05-28T14:00:09.413+0200 Metadata for mentat.groups to /var/tmp/20140528/mentat/groups.metadata.json connected to: 127.0.0.1 2014-05-28T14:00:09.437+0200 DATABASE: mentat to /var/tmp/20140528/mentat 2014-05-28T14:00:09.437+0200 mentat.users to /var/tmp/20140528/mentat/users.bson 2014-05-28T14:00:09.438+0200 8 documents 2014-05-28T14:00:09.438+0200 Metadata for mentat.users to /var/tmp/20140528/mentat/users.metadata.json tar: Removing leading `/' from member names
root@mentat:~# ls -alh /var/backups/20140528.tgz -rw-r--r-- 1 root root 114M May 28 14:00 /var/backups/20140528.tgz
Updated by Radomír Orkáč over 10 years ago
Pomohla uprava zalohovaci politiky:
#------------------------- Backup /var what=( "." ) exclude=( "backups" "lock" "lost+found" "run" "spool" "tmp" # added by Orkac "lib/mongodb" ) backup var "/var"
Po nekolika dnech testovani jsem exclude aktivoval az vcera. V noci zalohovani probehlo bez chyb:
Mon Jun 2 03:00:02 2014 mentat hawat-precache[22818]: NOTICE: hawat-precache has been executed Mon Jun 2 03:00:02 2014 mentat hawat-precache[22818]: NOTICE: Start: caching selectors Mon Jun 2 03:41:34 2014 mentat hawat-precache[22818]: NOTICE: Stop: caching selectors, time: 2491.93398785591 Mon Jun 2 03:41:34 2014 mentat hawat-precache[22818]: NOTICE: Selectors data: $VAR1 = { 'alert-class-select' => [ '(D)DoS', 'Arakis Report', 'Backscatter Report',
Updated by Radomír Orkáč over 10 years ago
Opet zacal problem s kesovanim selectoru.
Wed Jun 18 02:28:55 2014 mentat hawat-precache[3472]: NOTICE: Selectors data: $VAR1 = { 'error' => [ 'Error in getting detector selector data from db. recv timed out (800000 ms) at /usr/local/lib/perl/5.14.2/MongoDB/Cursor.pm line 160. ' ],
Updated by Pavel Kácha over 10 years ago
- Related to Bug #1498: hawat - dashboard - nespravne hodnoty u kese 3m added
Updated by Jan Mach almost 8 years ago
- Status changed from In Progress to Closed
Již neaktuální, úklid v úkolech, zavírám.