Implementacni poznamky¶
Struktura Warden databaze pro warden-server-2.2¶
Tabulka 'clients'
+--------------------+--------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------------------+----------------+
| client_id | int(11) | NO | PRI | NULL | auto_increment |
| hostname | varchar(256) | YES | | NULL | |
| registered | timestamp | NO | | 0000-00-00 00:00:00 | |
| requestor | varchar(256) | YES | | NULL | |
| service | varchar(64) | YES | | NULL | |
| client_type | varchar(1) | YES | | NULL | |
| type | varchar(64) | YES | | NULL | |
| receive_own_events | varchar(1) | YES | | NULL | |
| description_tags | varchar(256) | YES | | NULL | |
| ip_net_client | varchar(256) | YES | | NULL | |
| valid | varchar(1) | YES | | NULL | |
+--------------------+--------------+------+-----+---------------------+----------------+
Tabulka 'events'
+--------------+-----------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-----------------+------+-----+---------------------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| detected | timestamp | NO | | 0000-00-00 00:00:00 | |
| received | timestamp | NO | | 0000-00-00 00:00:00 | |
| type | varchar(64) | YES | | NULL | |
| source_type | varchar(64) | YES | | NULL | |
| source | varchar(256) | YES | | NULL | |
| target_proto | varchar(16) | YES | | NULL | |
| target_port | int(2) unsigned | YES | | NULL | |
| attack_scale | int(4) unsigned | YES | | NULL | |
| note | text | YES | | NULL | |
| priority | int(1) unsigned | YES | | NULL | |
| timeout | int(2) unsigned | YES | | NULL | |
| valid | varchar(1) | YES | | NULL | |
| client_id | int(11) | YES | | NULL | |
+--------------+-----------------+------+-----+---------------------+----------------+
Vyvoj funkcionality¶
Typy Subject Alternate Names v SSL certifikatu:¶
#define GEN_OTHERNAME 0
#define GEN_EMAIL 1
#define GEN_DNS 2
#define GEN_X400 3
#define GEN_DIRNAME 4
#define GEN_EDIPARTY 5
#define GEN_URI 6
#define GEN_IPADD 7
#define GEN_RID 8
Odmazavani databaze¶
- /opt/warden-server/bin/warden-server.pl
our $MAX_DB_SIZE = undef;
our $LWM = undef;
.
.
.
#-------------------------------------------------------------------------------
# vacuumDb - start vacuum process in SQLite database
#-------------------------------------------------------------------------------
sub vacuumDb {
local $DBH->{AutoCommit} = 1;
$DBH->do('VACUUM');
return;
}
#-------------------------------------------------------------------------------
# cleanDb - delete old received events from table 'events' after exceed of
# $MAX_DB_SIZE
#-------------------------------------------------------------------------------
sub cleanDb
{
my $db = shift;
my $db_size_human = shift;
my $db_size_bytes = shift;
my $max_db_size_bytes = shift;
my $lwm_bytes = ($max_db_size_bytes / 100) * $LWM;
my ($sth, $last_date, $last_timestamp, $delete_threshold_db, $db_size_bytes_old, $sum_deleted_bytes);
write2log("info", "Starting cleanup of DB => DB_SIZE: $db_size_human ($db_size_bytes bytes), MAX_DB_SIZE: $MAX_DB_SIZE" . "MB ($max_db_size_bytes bytes), LWM: $LWM" . "% ($lwm_bytes bytes)");
LOOP: {
do {
# obtain timestamp of last received event
$sth = $DBH->prepare("SELECT received FROM events WHERE id = (SELECT min(id) FROM events);");
if ( !defined $sth ) {die("Cannot prepare statement in cleanDb: $DBI::errstr\n")}
$sth->execute;
$last_timestamp = $sth->fetchrow();
# prepare threshold for deletion
last LOOP if !defined $last_timestamp; # break if $last_timestamp is not defined => DB is empty
$last_date = substr($last_timestamp,0,10); # only 2012-01-16 from 2012-01-16T13:44:45
$delete_threshold_db = $DBH->quote($last_date . "T23:59:59");
# delete events latest than $delete_threshold
$DBH->do("DELETE FROM events WHERE received <= $delete_threshold_db;");
if ($DBH->err()) {die("Cannot do delete statement in cleanDb: $DBI::errstr\n")}
$DBH->commit();
vacuumDb;
# obtain new DB size and sum of deleted bytes
$db_size_bytes_old = $db_size_bytes;
$db_size_bytes = -s $db;
$sum_deleted_bytes = $db_size_bytes_old - $db_size_bytes;
write2log("info", "$last_date ($sum_deleted_bytes bytes) was deleted; DB size: $db_size_bytes bytes;");
} while ($db_size_bytes >= $lwm_bytes);
}
$db_size_human = Format::Human::Bytes::base10(-s $db);
write2log("info", "Stopping cleanup of DB => DB_SIZE: $db_size_human ($db_size_bytes bytes), MAX_DB_SIZE: $MAX_DB_SIZE" . "MB ($max_db_size_bytes bytes), LWM: $LWM" . "% ($lwm_bytes bytes)");
} # End of cleanDb
.
.
.
# check if size of DB file exceed $MAX_DB_SIZE
#my $db_size_bytes = -s $db;
#my $max_db_size_bytes = $MAX_DB_SIZE * 1024 * 1024;
#if ($db_size_bytes >= $max_db_size_bytes) { cleanDb($db, $db_size_human, $db_size_bytes, $max_db_size_bytes) }
- /opt/warden-server/etc/warden-server.conf
#-------------------------------------------------------------------------------
# MAX_DB_SIZE - maximal size of database file (in MB)
#-------------------------------------------------------------------------------
$MAX_DB_SIZE = 790;
#-------------------------------------------------------------------------------
# LWM - delete received events from table 'events' until size of database
# file = $LWM % of $MAX_DB_SIZE; typically 90
#-------------------------------------------------------------------------------
$LWM = 90;
Poznamky¶
Prenasene XML schema¶
Poznatky o odmazavani databaze SQLite¶
- pro periodicke odmazavani databaze jsem pouzil algoritmus, ktery:
- zacal odmazavat udalosti z DB po dosazeni nastavene maximalni velikosti db v promenne $max_db_size
- odmazavani skoncilo v okamzik, kdy aktualni velikost databaze $current_db_size <= hodnote promenne $low_water_mark, ktera by nastavena na 90% (90% z $max_db_size)
- z databaze byly odmazany udalosti vzdy po 1 dni
- z DB byla vzdy odmazana posledni udalost v DB spolecne se vsemi, ktere prisly na warden server ve stejny den
- z casu prijeti posledni udalosti, napr. 2012-01-16T13:44:45 se vzala cast 2012-01-16 a udalosti, ktere prisly v tento den byly take smazany
- po odmazani dostatecneho mnozstvi udalosti byl spusten prikaz vacuum, ktery skutecne data z databaze smazal
- zde jsem narazil na problem, jelikoz beh vaccum na vetsi DB (na DB ve ktere se odmazalo vice dat) trval nekolik minut
- zkousel jsem i odhadovat cas, ktery vaccum spotrebuje, nicmene tento pristup se neosvedcil, jelikoz kazdy zaznam v DB ma jinou velikost
- pro tento algoritmus by bylo lepsi kdyby warden jel na databazi umoznujici partitioning (PostgreSQL), jelikoz by bylo mozne odmazavat data po partitionach o velikosti napr. 1 den
*
- dale, pokud se nepletu, tak vaccum v PostgreSQL nezamyka tabulku a je tedy mozne s ni pracovat i po dobu behu vacuua