Overview | Documentation
BeatLog parses NGINX reverse proxy and fail2ban logs into readable tables and reports. Use BeatLog to assess server traffic and tailor fail2ban filters.
BeatLog is a Python based web application, and utilizes a PostgreSQL database for storage. Log files are parsed line-by-line using regex. BeatLog provides default patterns for parsing, but each log's regex scheme can be customized.
In addition to parsing data from log files, it categorizes each IP address as coming from "Home" (same IP as server) or "Outside" (anywhere else). Location information (coordinates, city, country) can be added to all "Outside" entries, using MaxMind's GeoLite2 database (SWAG users, see: mod installation). If only coordinates are provided from the MaxMind database, locations can be ascertained using the Nominatim reverse geocoding API, based on OpenStreetMap data.
See the BeatLog Documentation for a full description of features. The Database, Report, and Visitor Map are briefly highlighted here.
Data is saved in a PostgreSQL database and can be used for your own purposes. See the Processed Data section in the Setup Guide to see the table and field schema used for parsed log data, and the Database Explorer page to see how data can be queried and viewed within BeatLog. A simple JSON API provides daily summaries and bandwidth statistics.
Database Query - fail2ban Log
Adminer can be installed to facilitate interaction with the database.
Report |demo|
A report synthesizes all log data from the previous few days or a custom date range. Charts are integrated using CanvasJS, and Bootstrap is used for tables and styling. Within the report, you can:
- Analyze home and outside connections against fail2ban finds, bans, and ignores to assess efficacy of fail2ban filters.
- Scrutinize traffic from frequent visitors and monitor popular client requests
- Known Devices can be identified and separated from other outside connections
Visitor Map |demo|
Visitor locations are plotted onto an interactive map using LeafletJS and OpenStreetMap tiles.
- Tool tips show location names and total connections or unique visitors (IPs) over the selected time range
- Location marker sizes are scaled by total connections or unique visitors
- Tabular data is presented beneath the map
BeatLog docker images are created via workflows with two different tags.
"Stable" images are built and pushed with each release, and "Latest" images are built and pushed with each commit.
Therefore, the stable or arm32v7-stable tags are recommended, unless there are pending updates that are desired.
Current release: alpha-0.1.7, In development: alpha-0.1.8
| Architecture | Latest Tags | Stable Tags |
|---|---|---|
| x86-64 | latest | stable, alpha-0.1.7 |
| arm64 | latest | stable, alpha-0.1.7 |
| armhf | (built on request) | arm32v7-stable, arm32v7-alpha-0.1.7 |
A PostgreSQL database is required, and can be included in the same docker deployment, as shown below.
Or, connect to an existing database, by providing connection settings under environment:.
Logs and other files are added to the container via volumes.
See the data sources section for specific files in the mounted directories and their usage.
In the example below, the specified directories, and their contents, will be available within a created /import/ directory.
It is important to mount files that may change (log turnover, changed fail2ban settings, MaxMindDB updates) indirectly via their parent directories. Directly mounted files will not update within the container.
The compose parameters are detailed in the next section. Optional healthcheck and adminer additions are shown on the Installation Extras page.
With an existing, connectable database, depends_on: and the following lines are not needed. Ensure a database with the name specified in BeatLog's environment exists.
Docker Compose
---
version: "2.1"
services:
beatlog:
image: nbpub/beatlog:stable
container_name: beatlog
user: 1000:1000 # optional
ports:
- 5000:8000 # access from 5000 instead of 8000, for demonstration
environment:
- TZ=Pacific/Galapagos
- db_host=<IP>
- db_password=changeme
- FLASK_SECRET_KEY=<secretkey>
- check_IP=12
- check_Log=3
volumes:
- /path_to/swag_config/log:/import/log # NGINX and fail2ban logs
- /path_to/swag_config/fail2ban:/import/fail2ban # fail2ban jail.local
- /path_to/swag_config/geoip2db:/import/geoip2db # MaxMindDB
depends_on:
- db
db:
image: postgres:15
container_name: beatlog_db
ports:
- 5432:5432
environment:
- POSTGRES_USER=beatlog
- POSTGRES_PASSWORD=changeme
- TZ=Pacific/Galapagos
volumes:
- /path_to/beatlog_conifg/db:/var/lib/postgresql/data # recommended
restart: unless-stopped Container images are configured using parameters passed at runtime (such as those above). These parameters are separated by a colon and indicate <external>:<internal> respectively.
For example, setting ports to 5433:5432 would expose port 5432 from inside the container to be accessible from the host's IP on port 5433 outside the container.
Sensitive data can be passed to compose using secrets, if desired.
Docker Compose Parameters
| Parameter | Function |
|---|---|
| user | |
1000:1000 |
Optional setting to change the user used for the docker container. See also |
| ports | |
5000:8000 |
Example of changing external access port. Internal port, 8000, should not be changed. |
| environment | |
TZ=Pacific/Galapagos |
Timezone should match log files. Defaults to UTC. time zone list |
db_host=<IP> |
IP address or host name of PostgreSQL database, defaults to localhost which should fail |
db_password=changeme |
PostgreSQL database password, should match POSTGRES_PASSWORD |
db_port= |
Only needed if database port is changed, defaults to 5432 |
db_user= |
Only needed if POSTGRES_USER is changed, defaults to beatlog |
db_database= |
Only needed if POSTGRES_USER is changed, defaults to beatlog |
FLASK_SECRET_KEY=<secretkey> |
Generate a secret key for deployment. Default, dev, is not suitable |
check_IP=12 |
Interval (hours) for checking / updating the home IP address, 12 hr default. Specify as integer, or 0 to disable |
check_Log=3 |
Interval (hours) for checking / parsing the Log Files, 3 hr default. Specify as integer, or 0 to disable |
| volumes | |
/path/to/log_directory:/path/in/container |
Add log files, fail2ban jail, and MaxMindDB to container, to be read by BeatLog. See data sources below |
/swag_config/<subdirectory>:/import/<subdirectory> |
example for SWAG structure in setup guide |
| postgres db ports | |
5432:5432 |
Default database port. If the external port is changed, then db_port must be specifed for the beatlog container |
| postgres db environment | |
POSTGRES_USER=beatlog |
If changed from beatlog, db_user and db_database must be specified for the beatlog container |
POSTGRES_PASSWORD=changeme |
PostgreSQL database password, should match db_password |
TZ=Pacific/Galapagos |
See above |
| postgres db volumes | |
/path_to/beatlog_conifg/db:/var/lib/postgresql/data |
Recommended: Store database files in a location of your choosing. Facilitates PostgreSQL updates. |
See the Installation Extras page for more, including how to add healthchecks to the docker containers and how to update the database.
BeatLog reads the following files for the information described. See the Parsing and Processed Data sections of the Guide for more information.
- NGINX reverse proxy
- access.log - client requests to the server
- error.log - client request errors and associated severity levels
- fail2ban
- fail2ban.log - all activity of fail2ban service, relevant information is parsed and the rest ignored
- jail.local - fail2ban settings and activated filters, checks ignored IPs
- MaxMindDB
- GeoLite2-City.mmdb - database to match IP addressess to locations, updated twice monthly
Create the container, monitor logs for proper startup, and then navigate to the WebUI at the port specified http://<your-ip>:5000.
If a database connection error is presented, check the parameters provided in your compose file and consult the container logs for more information.
Get the most information from BeatLog following these steps (detailed in Setup Guide):
- Specify Log File locations
*/Logs/add/ - Load default regex methods or create and test your own
*/Logs/add_regex/ - Associate regex methods with Log Files to prepare for parsing
*/Logs/<log_file>/regex/ - Specify fail2ban jail.local location
*/jail/ - Specify MaxMind GeoLite2-City database location, in geography settings
*/settings#geography - Add a user-agent for the Nominatim API to fill unnamed locations, in geography settings
- Parse Logs! Schedule parsing!
Submit bugs or feedback.
- fixed links on some BeatLog pages
show/hide
- Design
- utilize row factories with psycopg to make cleaner database selections
- use template file for SQL commands to clean up code
- add tests for code
- consider smarter way to gather regex methods across functions
- solve possible issues with SQL creation: Home Ignores
- Features
- visitor maps, pan to location from table entry
- fail2ban filter testing
- Failed Regex
- add ability to test Regex methods
- JSON API
- handle geography data better / provide calls to retrieve geography data
- other ideas listed on page?
- allow for SSL in deployment, likely just changes to Gunicorn start command
- Copy to clipboard button won't work in most browsers without connecting BeatLog as
localhostor adding a certificate for LAN connections. - Enable via environmental variable, if enabled Gunicorn should look for cerificates / keyfiles. This way a key can be added and "activated" after initial setup.
- Even with a certificate, BeatLog should only be hosted on trusted network and accessed locally
- Copy to clipboard button won't work in most browsers without connecting BeatLog as
- Deployment
- will using server hooks instead of pre-load for scheduled tasks and management of database connection pool improve anything?
- add NGINX? utilize LSIO base images?
| Version (Docker Hub) | Notes |
|---|---|
| alpha-0.1.0 | Initial release, testing docker deployment. Flask App environmental variables must be used with this image, similar to Local Installation. Internal port is 5000 for this container. |
| alpha-0.1.1 | Switched WSGI from Werkzeug to Gunicorn, updated compose example. Minor fixes / tweaks. Working to properly implement Gunicorn, APScheduler, psycopg3 together. |
| alpha-0.1.1t | NullConnectionPool version of alpha-0.1.1. may be more stable and less load on postgresql, might be slower. psycogp3 ConnectionPool vs. NullConnectionPool |
| alpha-0.1.2, alpha-0.1.2t | Improved contruction of SQL queries across all functions and pages, with care for SQL Injection risks. Docker images built via Github workflow. Added demo page. Bugfixes and improvements. |
| alpha-0.1.3, alpha-0.1.3t | BREAKING: altered database schema! notes. Added DB Query and View pages to access database within BeatLog. Improved handling of Known Devices. Added location fill to scheduled log parsing. Bugfixes and improvements. Last NullConnectionPool release. |
| alpha-0.1.4 | Removed support for unauthorized log, link for migration. Added simple API. No more NullConnectionPool tag. All docker images built and pushed via github actions now. |
| alpha-0.1.5 | Additional API features, and API code organization. Bugfixes / aesthetic improvements. Expanding API documentation and help page. |
| alpha-0.1.6 | BIG documentation reorganization. Modified bandwidth API, no longer rounding to nearest day. Added Failed Regex features. Bugfixes, aesthetic, and navigation improvements. |
| alpha-0.1.7 | Upgrade to Python 3.11, minor feature additions and improvements: API, Report, Data Clean |
| alpha-0.1.8 | minor fixes |