Warden 3¶

Motivation¶

The main goal of Warden 3 is to address the shortcomings, which emerged during several years of Warden 2.X operation. Warden 3 uses flexible and descriptive event format, based on JSON. Warden 3 protocol is based on plain HTTPS queries with help of JSON (Warden 2 SOAP is heavyweight, outdated and draws in many dependencies). Clients can be multilanguage, unlike SOAP/HTTPS, plain HTTPS and JSON is mature in many mainstream programming languages. Server is written in Python - mature language with consistent and coherent libraries and many skilled developers.

Concepts¶

Event description format¶

IDEA - Intrusion Detection Extensible Alert, flexible extensible format for security events, see: https://csirt.cesnet.cz/IDEA

Event serial ID¶

Each received event gets assigned integer serial number. These numbers are sequential, so each recipient can keep track of the last event id it received and next time ask only for following events.

Authentication¶

In Warden 2, clients get authenticated by server certificate, however server certificate is usually same for the whole machine, so individual clients are differentiated only by telling its own name. However, client name is widely known, so this allows for client impersonation within one machine. Warden 3 slightly improves this schema by replacing client name in authentication phase by "secret", random string, shared among particular client and main server, which makes it harder to forge client identity (be it by mistake or intentional).

However, best solution for these cases is of course specific certificate for each particular client (which is also fully supported).

Client also has to have server CA certificate (or chain) at its disposal to be able to verify server authenticity.

Client name¶

Unlike Warden 2, client names in Warden 3 have hierarchy. Modelled after Java class names, client name is dot separated list of labels, with significance from left to right – leftmost denoting largest containing realm, rightmost denoting single entity. Country.organisation.suborganizations.machine.local realm scheme akin to "org.example.csirt.northwest.honeypot.jabberwock" is strongly recommended. Label case is significant, label can contain only letters, numbers or underscore and must not start with number.

The reason is the possibility to filter incoming events based not only on particular client, or (for some recipients flawed) notion of "own" messages, but based on wider units.

HTTP/JSON API¶

Client must know the base URL of the Warden server. Warden 3 accepts queries on paths under base URL (which correspond to called method), with usual query string variable=data pairs separated by ampersand as arguments. Multivalues are specified by repeating same variable with each value several times.

   https://warden.example.org/warden3/getEvents?secret=PwD&cat=Abusive.Spam&cat=Fraud.Phishing
   \________________ _______________/ \___ ___/ \____ ___/ \______ _______/ \________ _______/
                    V                     V          V            V                  V
   Base URL       --'                     |          |            |                  |
   Called method  ------------------------'          |            |                  |
   Key/value pair -----------------------------------'            |                  |
   Multivalue     ------------------------------------------------'------------------'

Method may expect bulk data (events to save, for example) - query then must be POST, with POST JSON data, formed appropriately as documented in particular method.

If HTTPS call succeeds (200 OK), method returns JSON object containing requested data.

Error handling¶

Should the call fail, server returns HTTP status code, together with JSON object, describing the errors (there may be multiple ones, especially when sending events). The keys of the object, which may be available, are:

method - name of the method called
req_id - unique identifier or the request (for troubleshooting, Warden administrator can uniquely identify related log lines)
errors - always present list of JSON objects, which contain:
- error - HTTP status code
- message - human readable error description
- Other context dependent fields may appear, see particular method description.

Client errors (4xx) are considered permanent - client must not try to send same event again as it will get always rejected - client administrator will need to inspect logs and rectify the cause.

Server errors (5xx) may be considered by client as temporary and client is advised to try again after reasonable recess.

Common arguments¶

secret - shared secret, assigned to client during registration
client - client name, optional, can be used to mimic Warden 2 authentication behavior if explicitly allowed for this client by server administrator

getEvents¶

Fetches events from server.

Arguments¶

count - number of requested events
id - starting serial number requested, id of all received events will be greater
cat, nocat - selects only events with categories, which are/are not present in the event Category field (mutually exclusive)
group, nogroup - selects only events originated/not originated from this realms and/or client names, as denoted in the event Node.Name field (mutually exclusive)
tag, notag - selects only events with/without this client description tags, as denoted in the event Node.Type field (mutually exclusive)

Returns¶

lastid - serial number of the last received event
events - array of Idea events

Example¶

$ curl \
    --key key.pem \
    --cert cert.pem \
    --cacert ca.pem \
    --connect-timeout 3 \
    --request POST \
    \
"https://warden.example.org/getEvents?\
secret=SeCrEt\
&count=1\
&nogroup=org.example\
&cat=Abusive.Spam\
&cat=Fraud.Phishing" 

{"lastid": 581,
 "events": [{
  "Format": "IDEA0",
  "DetectTime": "2015-02-03T09:55:21.563638Z",
  "Target": [{"URL": ["http://example.com/kocHq"]}],
  "Category": ["Fraud.Phishing"],
  "Note": "Example event"}]}

sendEvents¶

Uploads events to server.

Arguments¶

POST data - JSON array of Idea events

Returns¶

Returns object with number of saved messages in "saved" attribute.

In case of error, multiple errors may be returned in "errors" list (see Error handling section). Each of the error objects may contain "events" key with list of indexes of events affected by this particular error. If there is error object without "events" key, caller must consider all events affected.

Note¶

Should the call fail because of errors in just couple of events, error message will contain JSON object in "detail.errors" section. The keys of the object are indexes into POST data array, values are error messages for each particular failed Idea event.

Example¶

$ eventid=$RANDOM$RANDOM$RANDOM$RANDOM$RANDOM
$ detecttime=$(date --rfc-3339=seconds|tr " " "T")
$ client="cz.example.warden.test" 
$ printf '
[
 {
  "Format": "IDEA0",
  "ID": "%s",
  "DetectTime": "%s",
  "Category": ["Test"],
  "Node": [{"Name": "%s"}]
 }
]' $eventid $detecttime $client |\
curl \
    --key $keyfile \
    --cert $certfile \
    --cacert $cafile \
    --connect-timeout 3 \
    --request POST \
    --data-binary "@-" \
    "https://warden.example.org/sendEvents?client=$client&secret=SeCrEt" 

{}

(However note that this is not the best way to generate Idea messages. )

Example with error:

$ curl \
    --key $keyfile \
    --cert $certfile \
    --cacert $cafile \
    --connect-timeout 3 \
    --request POST \
    --data-binary '[{"Format": "IDEA0","ID":"ASDF","Category":[],"DetectTime":"asdf"}]' \
    "https://warden.example.org/sendEvents?client=cz.example.warden.test&secret=SeCrEt" 

{"errors":
 [
  {"message": "Validation error: key \"DetectTime\", value \"asdf\", expected - RFC3339 timestamp.",
   "events": [0],
   "error": 460
  }
 ],
 "method": "sendEvents",
 "req_id": 3726454025
}

getInfo¶

Returns basic server information.

Returns¶

version - Warden server version string
description - server greeting
send_events_limit - sendEvents will be rejected if client sends more events in one call
get_events_limit - getEvents will return at most that much events

Example¶

$ curl \
    --key key.pem \
    --cert cert.pem \
    --cacert ca.pem \
    --connect-timeout 3 \
    --request POST \
    "https://warden.example.org/getInfo?secret=SeCrEt" 

{"version": "3.0-not-even-alpha",
 "send_events_limit": 500,
 "get_events_limit": 1000,
 "description": "Warden 3 not even alpha development server"}

Python API¶

Python API tries to abstract from raw HTTPS/URL/JSON details. User instantiates Client class with necessary settings (certificates, secret, client name, logging, limits, ...) and then uses its method to access server.

Client constructor¶

wclient = warden.Client(
              url,
              certfile=None,
              keyfile=None,
              cafile=None,
              timeout=60,
                          retry=3,
                          pause=5,
              recv_events_limit=6000,
                          send_events_limit=500,
              errlog={},
              syslog=None,
              filelog=None,
              idstore=None,
              name="org.example.warden_client",
              secret=None)

url - Warden server base URL
certfile, keyfile, cafile - paths to X509 material
timeout - network timeout value in seconds
retry - number retries on transitional errors during sending events
pause - wait time in seconds between transitional error retries
recv_events_limit - maximum number of events to receive (note that server may have its own notion)
send_events_limit - when sending, event lists will be split and sent by chunks of at most this size (note that used value will get adjusted according to the limit reported by server)
errlog - stderr logging configuration dict
- level - most verbose loglevel to log
syslog - syslog logging configuration dict
- level - most verbose loglevel to log
- socket - syslog socket path (defaults to "/dev/log")
- facility - syslog facility (defaults to "local7")
filelog - file logging configuration dict
- level - most verbose loglevel to log
- file - path to log file
idstore - path to simple text file, in which last received event ID gets stored. If None, server notion is used
name - client name
secret - authentication secret

Configuration file helper¶

warden.read_cfg(cfgfile)

Warden object can get initialized from JSON like configuration file. It's essentially JSON, but full line comments, starting with "#" or "//", are allowed. read_cfg reads the configuration file and returns dict suitable for passing as Client constructor arguments.

Usage example:

wclient = warden.Client(**warden.read_cfg("warden_client.cfg"))

warden.Client.getEvents¶

wclient.getEvents(
    id=None,
    idstore=None,
    count=1,
    cat=None, nocat=None,
    tag=None, notag=None,
    group=None, nogroup=None)

id - can be used to explicitly override value from idstore file
idstore - can be used to explicitly override idstore for this request
count - number of requested events
cat, nocat - selects only events with categories, which are/are not present in the event Category field (mutually exclusive)
group, nogroup - selects only events originated/not originated from this realms and/or client names, as denoted in the event Node.Name field (mutually exclusive)
tag, notag - selects only events with/without this client description tags, as denoted in the event Node.Type field (mutually exclusive)

Returns¶

list of Idea events

warden.Client.sendEvents¶

wclient.sendEvents(self, events=[], retry=None, pause=None):

events - list of Idea events to be sent to server
retry - use this retry value just for this call instead from value from constructor
pause - use this pause value just for this call instead from value from constructor

Returns¶

dict with number of sent events under "saved" key

Note¶

events list length is limited only by available resources, sendEvents will split it and send separately in at most send_events_limit long chunks (however note that sendEvents will also need additional memory for its internal data structures).

Server errors (5xx) are considered transitional and sendEvents will do retry number of attempts to deliver corresponding events, delayed by pause seconds.

Should the call fail because of errors, particular errors may contain "events" list. Values of the list are then indexes into POST data array. If no "events" list is present, all events attempted to send must be considered as failed (with this particular error). See also Error handling section.

Errors may also contain event IDs from Idea messages in "events_id" list. This is primarily for logging - client administrator may identify offending messages by stable identifiers.

warden.Client.getInfo¶

wclient.getInfo()

Returns dictionary of information from getInfo Warden call.

Error class¶

Error(
    message,
    logger=None,
    error=None,
    prio="error",
    method=None,
    req_id=None,
    detail=None,
    exc=None)

Class, which gets returned in case of client or server error. Caller can test whether it received data or error by checking:

isinstance(res, Error).

However if he does not want to deal with errors altogether, this error object also returns False value if used in Bool context and acts as an empty iterator - in following examples do_stuff() is not evaluated:

if res:
    do_stuff(res)

for e in res:
    do_stuff(e)

str(Error_Instance) outputs formatted error, info_str() and debug_str() output increasingly more detailed info.

Files (0)

Project

General

Profile

Warden

Wiki

Warden 3¶

Motivation¶

Concepts¶

Event description format¶

Event serial ID¶

Authentication¶

Client name¶

HTTP/JSON API¶

Error handling¶

Common arguments¶

getEvents¶

Arguments¶

Returns¶

Example¶

sendEvents¶

Arguments¶

Returns¶

Note¶

Example¶

getInfo¶

Returns¶

Example¶

Python API¶

Client constructor¶

Configuration file helper¶

warden.Client.getEvents¶

Returns¶

warden.Client.sendEvents¶

Returns¶

Note¶

warden.Client.getInfo¶

Error class¶