Simple and fast marshalling of Lisp datastructures. Convert any object into a string representation, put it on a stream an revive it from there. Only minimal changes required to make your CLOS objects serializable. Actually you only need to add 1 method per baseclass.
Just to make that clear: project is not dead - the functionality is simply finished and working.
MIT License. See included LICENSE file.
For this ASDF needs to be installed and has to have access to marshal.asd. Otherwise do a load *.lisp/tests-.lisp.
(require :marshal)
Serialization of simple examples:
$ (ms:marshal (list 1 2 3 "Foo" "Bar" (make-array '(3) :initial-contents '(a b c))))
--> (:PCODE 1
(:LIST 1 1 2 3 (:SIMPLE-STRING 2 "Foo") (:SIMPLE-STRING 3 "Bar")
(:ARRAY 4 (3) T (A B C))))
Deserialization:
$ (ms:unmarshal '(:PCODE 1
(:LIST 1 1 2 3 (:SIMPLE-STRING 2 "Foo") (:SIMPLE-STRING 3 "Bar")
(:ARRAY 4 (3) T (A B C)))))
--> (1 2 3 "Foo" "Bar" #(A B C))
That means that a
(ms:unmarshal (ms:marshal myobject))
returns a deep clone of myobject.
Definition of a class
(defclass ship ()
((name :initform "" :initarg :name :accessor name)
(dimensions :initform '(:width 0 :length 0) :initarg :dimensions :accessor dimensions)
(course :initform 0 :initarg :course :accessor course)
(cruise :initform 0 :initarg :cruise :accessor cruise) ; shall be transient
(dinghy :initform NIL :initarg :dinghy :accessor dinghy)) ; another ship -> ref
(:documentation "A democlass. Some 'persistent slots', one transient.
Some numbers, string, lists and object references."))
(defparameter ark (make-instance 'ship :name "Ark" :course 360
:dimensions '(:width 30 :length 90)))
Let's try to serialize this:
$ (ms:marshal ark)
--> (:PCODE 1 NIL)
Actually nothing happens.
Next we define the method class-persistent-slots for this class. The method has to be defined
in the package :marshal.
(defmethod ms:class-persistent-slots ((self ship))
'(name dimensions course dinghy))
--> #<STANDARD-METHOD MARSHAL:CLASS-PERSISTENT-SLOTS (SHIP) {1002B16B31}
Note that the slot cruise is not listed. Therefore it will not be serialized.
$ (ms:marshal ark)
--> (:PCODE 1
(:OBJECT 1 SHIP (:SIMPLE-STRING 2 "Ark") (:LIST 3 :WIDTH 30 :LENGTH 90) 360
(:LIST 4)))
Fine. Try a (ms:unmarshal (ms:marshal ark)) and you will get a clone of the object ark.
Let's define some subclasses (yes, it's Lisp, we use multiple inheritance here).
(defclass sailingship (ship)
((sailarea :initform 0 :initarg :sailarea :accessor sailarea))
)
(defclass motorship (ship)
((enginepower :initform 0 :initarg :enginepower :accessor enginepower))
)
(defclass motorsailor (motorship sailingship)
()
)
Some instances:
(defparameter ship2 (make-instance 'sailingship :name "Pinta" :course 270 :cruise 9
:dimensions '(:width 7 :length 21) :sailarea 400))
(defparameter ship3 (make-instance 'motorship :name "Titanic" :course 320 :cruise 21
:dimensions '(:width 28 :length 269) :enginepower 51000))
(defparameter ship4 (make-instance 'motorsailor :name "Krusenstern" :course 180
:cruise 17.4 :dimensions '(:width 12 :length 82)
:sailarea 3400 :enginepower 2000))
Let's try
$ (ms:marshal ship4)
--> (:PCODE 1
(:OBJECT 1 MOTORSAILOR (:SIMPLE-STRING 2 "Krusenstern")
(:LIST 3 :WIDTH 12 :LENGTH 82) 180 (:LIST 4)))
Note that the slots to be marshalled are determined by the function ms:class-peristant-slots of
the baseclass ship.
One last class and an instance. please note the backreference to another ship. That's a circular reference.
(defclass dinghy (ship)
((aboard :initform NIL :initarg :aboard :accessor aboard)) ; another ship -> circular ref
)
(defparameter ship5 (make-instance 'dinghy :name "Gig" :course 320 :cruise 5
:dimensions '(:width 2 :length 6) :aboard ship4))
(setf (dinghy ship4) ship5)
$ (ms:marshal ship4)
--> (:PCODE 1
(:OBJECT 1 MOTORSAILOR (:SIMPLE-STRING 2 "Krusenstern")
(:LIST 3 :WIDTH 12 :LENGTH 82) 180
(:OBJECT 4 DINGHY (:SIMPLE-STRING 5 "Gig") (:LIST 6 :WIDTH 2 :LENGTH 6)
320 (:LIST 7))))
We see the reference to the dhingy (the "sub ship"), but not the
backreference. Simply, because so far the back link aboard is still transient.
(defmethod ms:class-persistent-slots ((self dinghy))
(append (call-next-method) '(aboard)))
$ (marshall ship4)
--> (:PCODE 1
(:OBJECT 1 MOTORSAILOR (:SIMPLE-STRING 2 "Krusenstern")
(:LIST 3 :WIDTH 12 :LENGTH 82) 180
(:OBJECT 4 DINGHY (:SIMPLE-STRING 5 "Gig") (:LIST 6 :WIDTH 2 :LENGTH 6)
320 (:LIST 7) (:REFERENCE 1))))
Brilliant! References, circles et. are working regardless these are references from and to lists, objects, hashtables, array etc.
Everything is in the package :marshal, nickname :ms.
To enable the serialization of a class, you need to specialise the
method ms:class-persistent-slots for this class, or one of its
baseclasses. This method must reside in the package :marshal! It has
to return a list of slotnames. These slots will be serialized.
A call to ms:marshal on an object will generate the string (actually
a sexp) representation. It will try to find the method ms:class-persistent-slots
to see which slots have to be serialized.
A call to ms:unmarshal with sexp generated by a ms:marshal will
revive an object.
You may have different implementations of classes to be serialized
and deserialized. For examples different classes on the endpoints of
a net work connections. Or simply different classes as time passes
between the persistent storage of a serialization and its retrieval.
It is important to understand that the classes that are serialized and
the one of the object that will be deserialized need to have the same
name and need to have the same slotnames as listed in ms:class-persistent-slots.
If you define a method ms:initialize-unmarshalled-instance for your
class, then this method will be called in the end of the
deserialization process. This gives you the chance to initialize
transient slots, that were not serialized, or to do other
initialization tricks.
There is a function called 'coding-idiom', that defines the language of the
marshalling. The default vocabulary is quite verbose. In case you are
going to send the objects through a network, you may want to change
that to a shorter set of verbs. Well, I think there are better ways
to speed that up, e.g. by adding a nginx proxy with automaic gzip
compression in front of your lisp webserver. Anyway, you will find an
alternative, shorter implementation in coding-idiom.lisp, it is fairly
straight-forward.
The most simple and recommended way to install cl-marshal is by using Quicklisp. If you installed Quicklisp a simple
(ql:quickload :marshal)
will download the package and actually load it. You only need to do this once per machine. Later a
(require :marshal)
will be enough.
Alternatively you may get the code with:
git clone git://github.com/wlbr/cl-marshal.git
Either you add this to your asdf repository, then you will only need
to do a (require :marshal) in your source.
Or, you may put the source in a subdirectory of your project and add
the file marshal.asd wih its full path to your own asdf definition.
Or, you may put the source in a subdirectory of your project and load
the file "marshal.asd" directly. After that a (asdf:load-system "marshal")
should be sufficient.
Or, as a kind of worst case, you simply do a direct (load <file>) of the files package.lisp, coding-idiom.lisp, marshal.lisp,
and unmarshal.lisp.
None except for asdf.
xlunit for the unit tests only (the tests are not included in the asdf system).
Tested with SBCL and CCL. No rocket science required, should run in any environment.
A set of unit tests is included in tests.lisp.
If you run into any trouble or find bugs, please report them via the Github issue tracker.
First written as encode/decode for CLOS objects only during a diploma thesis in '95. Major rework/enhancements during a research project in the end of the '90s. Refactoring in 2011 when revisiting Lisp.
Written by Michael Wolber. Major fixes and enhancements by Christoph Oechslein.