Usage of Monitoring Kafka extension

To try out there is a docker-compose.yml file in the test/resources directory. This will start up a Kafka broker and a schema registry instance. To start it up you need docker and docker-compose installed then run:

# start kafka
cd src/community/monitor-kafka/src/test/resources
docker-compose up -d

This will start up the Kafka broker and the schema registry.

Once Kafka is running start geoserver with monitor and monitor-kafka extension from src/web/app:

mvn jetty:run -P monitor,monitor-kafka

This will initialize the data directory in src/web/app/src/main/webapp/data and start up geoserver. Stop the geoserver again.

With this you will get the default monitor config installed automatically in src/web/app/src/main/webapp/data/monitoring/. Edit it to use the Kafka storage.

Then you need to configure the monitor-kafka extension for it with:

storage=kafka
kafka.bootstrap.servers=localhost:9092
kafka.schema.registry.url=http://localhost:8081

Create your topic with:

docker exec -ti broker kafka-topics --bootstrap-server localhost:9092 --create --topic geoserver-monitor --partitions 1 --replication-factor 1 --if-not-exists

Start geoserver again with:

mvn jetty:run -P monitor,monitor-kafka

Check that the monitoring extension is actually enabled by looking for the following log line:

INFO   [geoserver.monitor] - Kafka connection established and topic geoserver-monitor exists

Head over to http://localhost:8080/geoserver and hit some map, so you get some data into the topic.

If you want to consume the data you need a consumer which has the right Deserializer configured (io.confluent.kafka.serializers.KafkaAvroDeserializer).

The easiest way to do this is to enter the schema registry container with:

docker exec -ti schema-registry bash
# from within the container run:
  kafka-avro-console-consumer \
    --bootstrap-server broker:29092 \
    --topic geoserver-monitor \
    --from-beginning

# or directly from the host machine so you can pipe it into jq or a file...
docker exec -ti schema-registry kafka-avro-console-consumer \
    --bootstrap-server broker:29092 \
    --topic geoserver-monitor \
    --from-beginning | tee >(grep -E "^{" | jq) | grep -vE "^{"

Then you will see messages like this:

{
    "id": 2,
    "status": "FINISHED",
    "category": "OWS",
    "path": "/tiger/wms",
    "queryString": "SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&FORMAT=image/png&TRANSPARENT=true&STYLES&LAYERS=tiger:giant_polygon&exceptions=application/vnd.ogc.se_inimage&SRS=EPSG:4326&WIDTH=768&HEIGHT=384&BBOX=3.7958354296875,-40.4131489453125,71.2627583203125,-6.6962260546875",
    "body": "",
    "bodyContentLength": 0,
    "bodyContentType": "",
    "httpMethod": "GET",
    "startTime": 1697172148809,
    "endTime": 1697172148902,
    "totalTime": 93,
    "remoteAddress": "[0:0:0:0:0:0:0:1]",
    "remoteHost": "0:0:0:0:0:0:0:1",
    "host": "localhost",
    "internalHost": "client",
    "remoteUser": "anonymous",
    "remoteUserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    "remoteCountry": "",
    "remoteCity": "",
    "remoteLat": 0.0,
    "remoteLon": 0.0,
    "service": "WMS",
    "operation": "GetMap",
    "owsVersion": "1.1.1",
    "subOperation": "",
    "resources": [
        "tiger:giant_polygon"
    ],
    "responseLength": 6594,
    "responseContentType": "image/png",
    "errorMessage": "",
    "responseStatus": 0,
    "httpReferer": "",
    "coordinateReferenceSystem": "EPSG:WGS 84",
    "minx": 3.7958354296875,
    "miny": -40.4131489453125,
    "maxx": 71.2627583203125,
    "maxy": -6.6962260546875,
    "cacheResult": "",
    "missReason": "",
    "resourceProcessingTimes": [],
    "labelingProcessingTime": 0
}
{
    "id": 3,
    "status": "FINISHED",
    "category": "OWS",
    "path": "/tiger/wms",
    "queryString": "SERVICE=WMS&VERSION=1.1.1&REQUEST=GetFeatureInfo&FORMAT=image/png&TRANSPARENT=true&QUERY_LAYERS=tiger:giant_polygon&STYLES&LAYERS=tiger:giant_polygon&exceptions=application/vnd.ogc.se_inimage&INFO_FORMAT=text/html&FEATURE_COUNT=50&X=50&Y=50&SRS=EPSG:4326&WIDTH=101&HEIGHT=101&BBOX=39.23891004908502,-32.22561743843973,48.11586317408502,-23.348664313439727",
    "body": "",
    "bodyContentLength": 0,
    "bodyContentType": "",
    "httpMethod": "GET",
    "startTime": 1697172178181,
    "endTime": 1697172178213,
    "totalTime": 32,
    "remoteAddress": "[0:0:0:0:0:0:0:1]",
    "remoteHost": "",
    "host": "localhost",
    "internalHost": "client",
    "remoteUser": "anonymous",
    "remoteUserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    "remoteCountry": "",
    "remoteCity": "",
    "remoteLat": 0.0,
    "remoteLon": 0.0,
    "service": "WMS",
    "operation": "GetFeatureInfo",
    "owsVersion": "1.1.1",
    "subOperation": "",
    "resources": [
        "giant_polygon"
    ],
    "responseLength": 803,
    "responseContentType": "text/html;charset=utf-8",
    "errorMessage": "",
    "responseStatus": 0,
    "httpReferer": "",
    "coordinateReferenceSystem": "EPSG:WGS 84",
    "minx": 43.63344129908502,
    "miny": -27.743195563439727,
    "maxx": 43.63344129908502,
    "maxy": -27.743195563439727,
    "cacheResult": "",
    "missReason": "",
    "resourceProcessingTimes": [],
    "labelingProcessingTime": 0
}