Skip to content

Midtrans/maxwell

 
 

Repository files navigation

This is Maxwell's daemon, an application that reads MySQL binlogs and writes row updates to Kafka as JSON. Maxwell has a low operational bar and produces a consistent, easy to ingest stream of updates. It allows you to easily "bolt on" some of the benefits of stream processing systems without going through your entire code base to add (unreliable) instrumentation points. Common use cases include ETL, cache building/expiring, metrics collection, and search indexing.

advanced features:

  • Can do SELECT * from table (bootstrapping) initial loads of a table
  • supports automatic position recover on master promotion
  • flexible partitioning schemes for Kakfa - by database, table, primary key, or column
  • Maxwell pulls all this off by acting as a full mysql replica, including a SQL parser for create/alter/drop statements (nope, there was no other way).

→ Download: https://github.com/zendesk/maxwell/releases/download/v1.5.2/maxwell-1.5.2.tar.gz
→ Source: https://github.com/zendesk/maxwell


  mysql> insert into `test`.`maxwell` set id = 1, daemon = 'Stanislaw Lem';
  maxwell: {
    "database": "test",
    "table": "maxwell",
    "type": "insert",
    "ts": 1449786310,
    "xid": 940752,
    "commit": true,
    "data": { "id":1, "daemon": "Stanislaw Lem" }
  }
  mysql> update test.maxwell set daemon = 'firebus!  firebus!' where id = 1;
  maxwell: {
    "database": "test",
    "table": "maxwell",
    "type": "update",
    "ts": 1449786341,
    "xid": 940786,
    "commit": true,
    "data": {"id":1, "daemon": "Firebus!  Firebus!"},
    "old":  {"daemon": "Stanislaw Lem"}
  }
<script> jQuery(document).ready(function () { jQuery("#maxwell-header").append( jQuery("

The Daemon, maybe

") ); jQuery("pre").addClass("home-code"); }); </script>

About

Maxwell's daemon, a mysql-to-json kafka producer

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 89.1%
  • ANTLR 5.1%
  • Ruby 3.6%
  • Makefile 1.1%
  • Shell 1.1%