pub.mk.nw

\section{Introduction and usage}

Sometimes we wish to easily publish a release of the material we work with.
Here we provide the functionality of publishing files, we have two ways:
\begin{enumerate}
\item the [[upload]] target which uploads files to a server, and
\item the [[gh-release]] target which creates a release on the GitHub repo.
\end{enumerate}
To use the [[publish]] target, we must add either [[publish: upload]] or 
[[publish: gh-release]] to our [[Makefile]].

\subsection{Specifying files}

The idea is to publish files, and this is common between all publication 
methods.
This is controlled with the [[PUB_FILES]] variable, which is set to a space 
separated list of file names.
<<variables>>=
PUB_FILES?=
@ The [[upload]] target will take the files in [[PUB_FILES]] and upload them to 
the target server (more below).
The [[gh-release]] on the other hand will use the [[PUB_FILES]] as attachment 
to the release.

For convenience, we can also control files to ignore.
<<variables>>=
IGNORE_FILES?=      \(\.svn\|\.git\|CVS\)
PUB_IGNORE?=        ${IGNORE_FILES}
@ However, this only applies to the [[upload]] target.

\subsection{Automatically tag on publication}

Since the published files usually are stripped of their versioning information,
it can be a good idea to keep track of the corresponding version in the version
management system.
One way is to create a tag every time a publication is made.

To enable this feature we set the variable [[PUB_AUTOTAG]] to true.
By default we let it be false, \ie this feature is disabled.
<<variables>>=
PUB_AUTOTAG?=       false
@ Note that for [[gh-release]] this doesn't matter, a tag will be created on 
the GitHub side if it doesn't exist already.

The first thing we need is to know which \ac{VCS} is used.
We control this with [[PUB_VCS]].
<<variables>>=
PUB_VCS?=           git
@ The only thing needed more than this is any options that the user want to 
use.
<<variables>>=
PUB_TAG_OPTS?=
@ The tag name is controlled with the following variable.
The default value is today's date and the current time.
<<variables>>=
PUB_TAG_NAME?=      $(shell date +%Y%m%d-%H%M)
@

The tagging will be wrong if we have forgotten to commit the files we were 
working on.
For this reason we also provide a similar feature which automatically makes 
a commit.
This feature is also disabled by default.
<<variables>>=
PUB_AUTOCOMMIT?=    false
@ The command and options are similarly set with the following.
<<variables>>=
PUB_COMMIT_OPTS?=   -av
@


\section{Configuration for publishing files on a server, [[upload]]}

Publication means that we upload the files somewhere.
This is controlled by the following variable.
<<variables>>=
PUB_SERVER?=        localhost
@

We are also interested in where on the server the files are written.
<<variables>>=
PUB_DIR?=           ${PUBDIR}/${CATEGORY}
@ Once written to the location, we must consider the owner, group and access 
rights.
<<variables>>=
PUB_USER?=          ${USER}
PUB_GROUP?=         ${GROUP}
PUB_CHMOD?=         o+r
@

\subsection{Publication methods}

There are currently three methods for publication:
\begin{enumerate*}
\item [[ssh]],
\item [[git]], and
\item [[at]].
\end{enumerate*}
The default method is [[ssh]].
<<variables>>=
PUB_METHOD?=        ssh
@ The remaining parts of the configuration depends on which publication method 
is used.

\paragraph{ssh}

The [[ssh]] method will use the \ac{SSH} protocol to transfer the files.
It will compress the files, pipe the output to the [[ssh]] process which runs 
the decompression on the server --- in the specified directory.
After successful transfer it will try to change the access rights to what is 
given by the settings above.

\paragraph{at}

The [[at]] method works similarly to [[ssh]], the difference is that it 
postpones the publication until a certain time.
The time is given by the [[PKG_AT]] variable, or [[at]] as a shortcut for the 
command-line ([[make at=tomorrow]]).
<<variables>>=
at?=                tomorrow
PKG_AT?=            ${at}
@

The way this works is that instead of writing the files to [[PUB_DIR]] on the 
server, we write the files to [[PUB_TMP]] and then add an [[at]] job that will 
move the files from the temporary to the final directory.
<<variables>>=
PUB_TMPDIR?=        /var/tmp
@

\paragraph{git}

The [[git]] method uses Git's archive functionality.
This means that Git will export an archive made from a branch in the 
repository, which branch is used is controlled by the following variable.
<<variables>>=
PUB_BRANCH?=        master
@

\subsection{Publishing to multiple sites}

We might also be interested in publishing files to several places, \eg to 
a set of mirrors.
The variable [[PUB_SITES]] contains a list of sites.
<<variables>>=
PUB_SITES?=         main
@ We supply one by default, this allows us to simply use the general variables 
above.
This way, site-specific overrides can be specified by appending the variable 
with the site name, \eg [[-main]].
All other values are copied from the defaults, \ie the general variables.
<<variables>>=
define variables
PUB_METHOD-$(1)?=   ${PUB_METHOD}

PUB_SERVER-$(1)?=   ${PUB_SERVER}
PUB_DIR-$(1)?=      ${PUB_DIR}
PUB_FILES-$(1)?=    ${PUB_FILES}
PUB_IGNORE-$(1)?=   ${PUB_IGNORE}

PUB_USER-$(1)?=     ${PUB_USER}
PUB_GROUP-$(1)?=    ${PUB_GROUP}
PUB_CHMOD-$(1)?=    ${PUB_CHMOD}

PUB_AT-$(1)?=       ${PUB_AT}
PUB_TMPDIR-$(1)?=   ${PUB_TMPDIR}

PUB_BRANCH-$(1)?=   ${PUB_BRANCH}
endef

$(foreach site,${PUB_SITES},$(eval $(call variables,${site})))
@

\begin{example}
To publish the same material to three different mirrors, we can do the 
following.
\begin{lstlisting}
PUB_SITES=            main mirror1 mirror2
PUB_SERVER =          foo.bar
PUB_SERVER-mirror1 =  foo.bar.mirror1
PUB_SERVER-mirror2 =  foo.bar.mirror2
\end{lstlisting}
\end{example}


\section{Implementation}

This is an include file, so we will first use the C-style technique to prevent 
inclusion more than once.
Thus the structure is as follows.
<<pub.mk>>=
ifndef PUB_MK
PUB_MK=true

INCLUDE_MAKEFILES?=.
include ${INCLUDE_MAKEFILES}/portability.mk

<<variables>>

.PHONY: publish upload gh-release

ifeq (${PUB_AUTOTAG},true)
upload: autotag
gh-release: autotag
else ifeq (${PUB_AUTOCOMMIT},true)
upload: autocommit
gh-release: autocommit
endif

<<upload target>>
<<gh-release target>>
<<autotag and autocommit targets>>

endif
@

We will now cover the different parts below.
The [[<<variables>>]] block has been covered in the usage section, but the 
remaining are discussed below.

\subsection{The upload publication mechanism, [[upload]]}

The upload target consists of two parts.
<<upload target>>=
<<target for uploading>>
<<publication methods>>
@

We have a general publication mechanism that drives the publication process and
uses the methods described below.
We have a general target [[upload]] to be invoked by the user.
Then we have a specific [[upload-site]] target for each site, which does the 
actual publication.
We add all those as prerequisites to the main target.
<<target for uploading>>=
.PHONY: upload
upload: $(foreach site,${PUB_SITES},upload-${site})
@ Depending on the settings for automatic commits and tags, we also add targets
for those functionalities as prerequisites.
<<target for uploading>>=
ifeq (${PUB_AUTOTAG},true)
upload: autotag
else ifeq (${PUB_AUTOCOMMIT},true)
upload: autocommit
endif
@

Next up is the actual site-specific targets.
The prerequisites are the files that should be uploaded.
Then the recipe is simply a call to the relevant publication method.
<<target for uploading>>=
define upload_target
.PHONY: upload-$(1)
upload-$(1): $(foreach file,${PUB_FILES-$(1)},${file})
	$$(call upload-${PUB_METHOD-$(1)},$(1))
endef

$(foreach site,${PUB_SITES},$(eval $(call upload_target,${site})))
@

\subsection{Publication methods}
\label{PubMethods}

We will now cover the different publication methods.
The outline is as follows.
<<publication methods>>=
<<helper functions>>
<<ssh method>>
<<at method>>
<<git method>>
@ We will first discuss two helper functions, [[chown]] and [[chmod]].
Then we will process with the different methods discussed in the introduction.

Both [[chown]] and [[chmod]] takes one argument, the name of the site.
Then each function can use the site name to find the relevant configuration.
The [[chown]] function simply runs chown(1) on the [[PUB_DIR]] directory on the 
server.
<<helper functions>>=
define chown
$(if ${PUB_GROUP-$(1)},\
  $(if ${PUB_SERVER-$(1)},${SSH} ${PUB_SERVER-$(1)})\
  ${CHOWN} ${PUB_USER-$(1)}:$(strip ${PUB_GROUP-$(1)})\
  $(foreach f,${PUB_FILES-$(1)},${PUB_DIR-$(1)}/$f );\
  ,)
endef
@ Conversely, the [[chmod]] function does the same but with the chmod(1) 
command.
Note, however, that we do not run these commands if [[PUB_GROUP]] or 
[[PUB_CHMOD]], respectively, are empty.
<<helper functions>>=
define chmod
$(if ${PUB_CHMOD-$(1)},\
  $(if ${PUB_SERVER-$(1)},${SSH} ${PUB_SERVER-$(1)})\
  ${CHMOD} ${PUB_CHMOD-$(1)}\
  $(foreach f,${PUB_FILES-$(1)},${PUB_DIR-$(1)}/$f );\
  ,)
endef
@

\paragraph{ssh}

Now to the first publication method, the one using copying over \ac{SSH}.
We define the method as a make function which takes one argument, the name of 
the site.
<<ssh method>>=
define upload-ssh
<<create directory on server>>; \
<<pack the files and pipe them to the server>>; \
$(call chown,$(1)) \
$(call chmod,$(1))
endef
@ To create the directory on the server is straight-forward, we simply run the 
command over \ac{SSH}.
<<create directory on server>>=
$(if ${PUB_SERVER-$(1)},${SSH} ${PUB_SERVER-$(1)}) ${MKDIR} ${PUB_DIR-$(1)}
@

Next is the packing of the files.
<<pack the files and pipe them to the server>>=
<<generate file list>> | \
<<pack the files>> | \
<<extract the files on the server>>
@ Before we do anything with the files, we must ensure that the list of files 
is not empty --- if it was empty, that would break all of the following 
commands.
If not, we will use find(1) to generate a list of files to include.
We do this in case there is a directory in the list [[PUB_FILES]].
If there is a directory in there, we cannot filter it using [[PKG_IGNORE]], so 
we must generate a list of the entire hierarchy included.
<<generate file list>>=
[ -n "${PUB_FILES-$(1)}" ] && find ${PUB_FILES-$(1)} -type f -or -type l
@ Once we have the list we can use pax(1) to put them into an archive, an 
archive which is written to standard out.
<<pack the files>>=
xargs ${PAX} \
  $(foreach regex,${PUB_REGEX-$(1)},-s ${regex}) \
  -s "|^.*/$(strip ${PUB_IGNORE-$(1)})/.*$$||p"
@ We also filter the file list through a series of regular expressions.
The user may add regular expressions as a space-separated list in the following
variable.
<<variables>>=
PUB_REGEX?=     "|^(.*)$$$$|\1|p"
$(foreach site,${PKG_SITES},$(eval PUB_REGEX-${site}?=${PUB_REGEX}))
@

Finally, we extract the files on the server by running the corresponding pax(1) 
instance over \ac{SSH}.
<<extract the files on the server>>=
$(if ${PUB_SERVER-$(1)},${SSH} ${PUB_SERVER-$(1)}) ${UNPAX} \
  -s "\"|^|$(strip ${PUB_DIR-$(1)})/|p\""
@

\paragraph{at}

The next method is very similar to the first.
The difference here is a middle step where we copy the files to a temporary 
place on the server and an additional final step where we upload them in the 
destination at some predefined time.
<<at method>>=
define upload-at
<<create directory on server>>; \
<<create temporary directory>>; \
<<generate file list>> | \
<<pack the files>> | \
<<extract the files in the temporary directory>>; \
<<add at-job on the server>>
endef
@ We have already seen some of these code blocks above, we will now cover the 
new ones.

The first thing we want to do is to create a temporary directory on the server.
We do this in the proper way.
<<create temporary directory>>=
TMPPUB=$$($(if ${PUB_SERVER-$(1)},${SSH} ${PUB_SERVER-$(1)}) \
  "export TMPDIR=${PUB_TMPDIR-$(1)} && ${MKTMPDIR-$(1)}")
@ We allow the user to override the [[mktemp]] command per server, since this 
command might differ on different servers.
<<variables>>=
$(foreach site,${PUB_SITES},$(eval MKTMPDIR-${site}?=${MKTMPDIR}))
@

Next we upload the files to the temporary directory on the server.
The difference between this and previous upload is the extraction.
We will now use a different regular expression, one which prepends the 
temporary directory to all files.
<<extract the files in the temporary directory>>=
$(if ${PUB_SERVER-$(1)},${SSH} ${PUB_SERVER-$(1)}) ${UNPAX} \
  -s "\"|^|$${TMPPUB}/|p\""
@

Finally, we must add the at(1) job on the server.
This is done by changing the directory to the temporary directory, then we echo
the commands we want to execute later and pipe those to the at(1) command.
<<add at-job on the server>>=
$(if ${PUB_SERVER-$(1)},${SSH} ${PUB_SERVER-$(1)}) "cd $${TMPPUB} && (\
  echo 'mv ${PUB_FILES-$(1)} ${PUB_DIR-$(2)};' \
  $(if ${PUB_CHMOD-$(1)},\
    echo '${CHMOD-$(1)} ${PUB_CHMOD-$(1)} ${PUB_DIR-$(1)};',) \
  $(if ${PUB_GROUP-$(1)},\
    echo '${CHOWN-$(1)} ${PUB_USER-$(1)}:$(strip ${PUB_GROUP-$(1)}) ${PUB_DIR-$(1)};',) \
  ) | at ${PKG_AT}"
@ We note that we allow the user to specify different [[CHOWN]] and [[CHMOD]] 
variables for different servers, since these commands might differ per server.
<<variables>>=
define chown_and_chmod
CHOWN-$(1)?=  ${CHOWN}
CHMOD-$(1)?=  ${CHMOD}
endef
$(foreach site,${PUB_SITES},$(eval $(call chown_and_chmod,${site})))
@

\paragraph{git}

The last method uses Git's functionality to pack the files.
We simply use [[git archive]] and specify which branch to use.
Then we pipe the archive to the server, unpack as before and finally run 
[[chown]] and [[chmod]].
<<git method>>=
define upload-git
git archive ${PUB_BRANCH-$(1)} ${PUB_FILES-$(1)} \
  | $(if ${PUB_SERVER-$(1)},${SSH} ${PUB_SERVER-$(1)}) \
    ${UNPAX} -s ",^,$(strip ${PUB_DIR-$(1)}),"; \
$(call chown,$(1)) \
$(call chmod,$(1))
endef
@


\subsection{GitHub releases, [[gh-release]]}

Now let's turn our attention to the [[gh-release]] target.
It's important that we push any changes, since the tag and release are created 
on the server.
<<gh-release target>>=
.PHONY: gh-release
gh-release: ${PUB_FILES}
	git push --all
	git push --tags
	gh release create -t ${PUB_TAG_NAME} ${PUB_TAG_NAME} ${PUB_FILES}
@


\subsection{Automatically committing and tagging,
  [[autotag]] and [[autocommit]]}

The last feature allows us to automatically commit and make a tag when we 
publish.
We accomplish this by two targets that we have already seen above.
These targets use functions specific to the selected \ac{VCS}.
<<autotag and autocommit targets>>=
<<commit and tag functions>>

.PHONY: autocommit
autocommit:
	$(call autocommit-${PUB_VCS})

.PHONY: autotag
autotag:
	$(call autotag-${PUB_VCS})
@ Below we will cover the different \acp{VCS}.

For now there are two functions, one for committing and one for tagging.
The commit functions are quite straight-forward for all three \acp{VCS}.
The tagging is similarly straight-forward for two, but not the third.
<<commit and tag functions>>=
<<autocommit for git, svn and cvs>>
<<autotag for git and cvs>>
<<autotag for svn>>
@

The commit functions are as expected for all three \acp{VCS}.
<<autocommit for git, svn and cvs>>=
autocommit-git = git diff --quiet || git commit ${PUB_COMMIT_OPTS}
autocommit-svn = svn commit ${PUB_COMMIT_OPTS}
autocommit-cvs = cvs commit ${PUB_COMMIT_OPTS}
@ The tagging is similarly straight-forward for Git and \ac{CVS}.
<<autotag for git and cvs>>=
autotag-git = git tag ${PUB_TAG_OPTS} ${PUB_TAG_NAME}
autotag-cvs = cvs tag ${PUB_TAG_OPTS} ${PUB_TAG_NAME}
@

The tagging function for \ac{SVN} is not as easy though.
The outline is as follows.
<<autotag for svn>>=
<<helper functions for svn tagging>>

define autotag-svn
<<find the root of repo>>
<<go to root and create tag>>
endef
@ To find the root of the repository, or more exactly where the directories 
[[trunk]] and [[tags]] are located, we must search through the parent 
directories.
We start in the current working directory and add one level per iteration.
<<find the root of repo>>=
ROOT=.
while ! [ -d $${ROOT}/trunk ]; do \
  $(call exit_if_fs_root,$${ROOT})
  ROOT=$${ROOT}/.. \
done \
@ We must check if we reach the root of the file system.
We use the function [[exit_if_fs_root]] for this.
This function exits with value [[1]] if the current directory examined is the 
root of the file system.
If this happens, make(1) will abort the recipe and the code after will not be 
executed.
The way we check for equality is to check that the device identifiers and the 
inode numbers are equal, we can do that using stat(1).
<<helper functions for svn tagging>>=
define exit_if_fs_root
if [ $(stat -c %i $(1)) = $(stat -c %i /) \
     -a $(stat -c %d $(1)) = $(stat -c %d /) ]; then \
    exit 1; \
fi
endef
@

Finally, if the recipe is still executing, this means that we have found the 
root and we can copy the trunk to tags.
<<go to root and create tag>>=
cd ${ROOT} \
  && svn copy trunk tags/${PUB_TAG_NAME} \
  && svn commit ${PUB_COMMIT_OPTS};
@