Skip to content

Commit

Permalink
GraalVM native image (#38)
Browse files Browse the repository at this point in the history
* Partially working GraalVM native image

Investigation to determine if GraalVM can be used to fix #33.
So that Clojure/Java impl of data.xml is used instead of ClojureScript.

Using Clojure 1.10.2-alpha1 and spec-tools 0.10.0
Using xfthhxk fix for luchiniatwork/cambada#9
Working for files that don't exist (Traktor to Rekordbox)
Error for files that exist, from mp3-parser (Traktor to Rekordbox)
Error for all files? (Rekordbox to Traktor)

* ClojureScript tests passing again

* Partially working GraalVM native image

Investigation to determine if GraalVM can be used to fix #33.
Now working for files that exist (Traktor to Rekordbox)
Fixed error from mp3-parser

* Partially working GraalVM native image

Investigation to determine if GraalVM can be used to fix #33.
Now working for all files (Rekordbox to Traktor)

* Add clojure setup to build job steps

* Add native-image-linux job

Use Java 11 instead of Java 8

* Use sudo where needed

* Graalvm Native Image for Windows (#35)

* Add native-image-windows job

Doesn't build the native image yet
Need to determine installation directory first

* Reorganised to create an uberjar in the first step

The Clojure setup action doesn't work on Windows;
This is an experiment to just use GraalVM Native image directly instead

* Enable verbose output to determine paths

* Debugging

* Fixed name argument

* Add --initialize-at-build-time option

* No target directory

* Add steps to native-image-windows job

* Install Visual C Build Tools Workload

For Visual Studio 2017

* Changes for Powershell

* Use cmd instead of Powershell

* Add package and publish steps

* Debugging

* Fixed GraalVM image name

* Additional smoke tests

* Don't use direct linking

It causes java.lang.ClassNotFoundException: clojure.data.xml$parse

* Add native-image-macos job (#37)

* Add release job

* Update clojurescript dep to 1.10.773

Avoid error running uberjar alias
Execution error (ExceptionInfo) at clojure.tools.deps.alpha.extensions.maven/get-artifact (maven.clj:118).
Could not find artifact org.clojure:tools.reader:jar:1.3.0
Separate job for running tests
Add support for skipping builds

* Add status badge for GitHub Actions

* Reverted from fork back to juxt repo for tick dep

Also removed js-joda ns, no longer required with latest tick version

* Update mp3-parser dep to 0.2.0

* Add link to status badge

* Remove Azure Pipelines definition

* Added 0.5.0 notes
  • Loading branch information
alza-bitz authored Nov 3, 2020
1 parent 3890144 commit 8f0bae5
Show file tree
Hide file tree
Showing 32 changed files with 279 additions and 176 deletions.
171 changes: 169 additions & 2 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
@@ -1,14 +1,181 @@
name: "Build"
on: [push, workflow_dispatch]
jobs:
build:
test:
runs-on: ubuntu-latest
if: github.event_name == 'push' && contains(toJson(github.event.commits), '[ci skip]') == false && contains(toJson(github.event.commits), '[skip ci]') == false
steps:
- name: "Checkout"
uses: actions/checkout@v2
- name: "Setup Java"
uses: actions/setup-java@v1
with:
java-version: 8
java-version: 11
- name: "Setup Clojure"
uses: DeLaGuardo/setup-clojure@master
with:
cli: "1.10.1.697"
- name: "Clojure Tests"
run: clojure -A:dev:test
- name: "Install Node Dependencies"
run: npm install
- name: "ClojureScript Tests"
run: clojure -A:dev:test-cljs
uberjar:
needs: [test]
runs-on: ubuntu-latest
steps:
- name: "Checkout"
uses: actions/checkout@v2
- name: "Setup Java"
uses: actions/setup-java@v1
with:
java-version: 11
- name: "Setup Clojure"
uses: DeLaGuardo/setup-clojure@master
with:
cli: "1.10.1.697"
- name: "Clojure Tests"
run: clojure -A:dev:test
- name: "Uberjar"
run: clojure -A:uberjar
- name: "Smoke Test"
run: |
java -jar target/dj-data-converter-SNAPSHOT-standalone.jar -h
java -jar target/dj-data-converter-SNAPSHOT-standalone.jar test-resources/collection.nml
java -jar target/dj-data-converter-SNAPSHOT-standalone.jar test-resources/rekordbox.xml
- name: "Upload"
uses: actions/upload-artifact@v2
with:
name: dj-data-converter-standalone
path: target/dj-data-converter-SNAPSHOT-standalone.jar
native-image-linux:
needs: [uberjar]
runs-on: ubuntu-latest
steps:
- name: "Checkout"
uses: actions/checkout@v2
- name: "Setup GraalVM"
uses: DeLaGuardo/setup-graalvm@3
with:
graalvm-version: "20.2.0.java11"
- name: "Install GraalVM Native Image Deps"
run: |
sudo apt -y update
sudo apt -y install gcc libc6-dev zlib1g-dev libstdc++-8-dev
- name: "Install GraalVM Native Image"
run: gu install native-image
- name: "Download"
uses: actions/download-artifact@v2
with:
name: dj-data-converter-standalone
- name: "Native Image"
run: native-image --verbose --no-server --no-fallback --allow-incomplete-classpath --report-unsupported-elements-at-runtime --initialize-at-build-time -J-Xmx8g -jar dj-data-converter-SNAPSHOT-standalone.jar -H:Name=dj-data-converter
- name: "Smoke Test"
run: |
chmod +x dj-data-converter
./dj-data-converter -h
- name: "Package"
run: tar -czvf dj-data-converter-linux.tar.gz dj-data-converter
- name: "Publish"
uses: actions/upload-artifact@v2
with:
name: dj-data-converter-linux
path: dj-data-converter-linux.tar.gz
native-image-windows:
needs: [uberjar]
runs-on: windows-latest
steps:
- name: "Setup GraalVM"
uses: DeLaGuardo/setup-graalvm@3
with:
graalvm-version: "20.2.0.java11"
- name: "Install GraalVM Native Image"
run: C:\hostedtoolcache\windows\GraalVM\20.2.0-java11\x64\bin\gu install native-image
- name: "Install Visual C Build Tools Workload for Visual Studio 2017"
run: choco install visualstudio2017-workload-vctools
- name: "Download"
uses: actions/download-artifact@v2
with:
name: dj-data-converter-standalone
- name: "Native Image"
shell: cmd
run: |
call "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
C:\hostedtoolcache\windows\GraalVM\20.2.0-java11\x64\bin\native-image --verbose --no-server --no-fallback --allow-incomplete-classpath --report-unsupported-elements-at-runtime --initialize-at-build-time -J-Xmx8g -jar dj-data-converter-SNAPSHOT-standalone.jar -H:Name=dj-data-converter
- name: "Smoke Test"
shell: cmd
run: dj-data-converter.exe -h
- name: "Package"
run: 7z a dj-data-converter-win.zip dj-data-converter.exe
- name: "Publish"
uses: actions/upload-artifact@v2
with:
name: dj-data-converter-win
path: dj-data-converter-win.zip
native-image-mac:
needs: [uberjar]
runs-on: macos-latest
steps:
- name: "Checkout"
uses: actions/checkout@v2
- name: "Setup GraalVM"
uses: DeLaGuardo/setup-graalvm@3
with:
graalvm-version: "20.2.0.java11"
- name: "Install GraalVM Native Image"
run: gu install native-image
- name: "Download"
uses: actions/download-artifact@v2
with:
name: dj-data-converter-standalone
- name: "Native Image"
run: native-image --verbose --no-server --no-fallback --allow-incomplete-classpath --report-unsupported-elements-at-runtime --initialize-at-build-time -J-Xmx8g -jar dj-data-converter-SNAPSHOT-standalone.jar -H:Name=dj-data-converter
- name: "Smoke Test"
run: |
chmod +x dj-data-converter
./dj-data-converter -h
- name: "Package"
run: tar -czvf dj-data-converter-macos.tar.gz dj-data-converter
- name: "Publish"
uses: actions/upload-artifact@v2
with:
name: dj-data-converter-macos
path: dj-data-converter-macos.tar.gz
release:
needs: [native-image-linux, native-image-windows, native-image-mac]
runs-on: ubuntu-latest
if: success() && contains(github.ref, 'refs/tags/')
steps:
- name: "Download"
uses: actions/download-artifact@v2
- name: "Release"
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: ${{ github.ref }}
release_name: ${{ github.ref }}
draft: true
prerelease: false
- name: "Upload"
uses: actions/upload-release-asset@v1
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: dj-data-converter-linux/dj-data-converter-linux.tar.gz
asset_name: dj-data-converter-linux.tar.gz
asset_content_type: application/gzip
- name: "Upload"
uses: actions/upload-release-asset@v1
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: dj-data-converter-win/dj-data-converter-win.zip
asset_name: dj-data-converter-win.zip
asset_content_type: application/zip
- name: "Upload"
uses: actions/upload-release-asset@v1
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: dj-data-converter-macos/dj-data-converter-macos.tar.gz
asset_name: dj-data-converter-macos.tar.gz
asset_content_type: application/gzip
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,11 @@ pom.xml.asc
.cljs_node_repl
package-lock.json
node_modules
/pkg/
/out/
/cljs-test-runner-out/
.nrepl-port
.calva/
.clj-kondo/
/pkg/
/*.xml
/*.nml
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Support the case where an entry has a tempo, but no cues (Traktor to Rekordbox)
- Support disabling the "Store Beatmarker as Hotcue" Traktor setting

## 0.5.0 (2020-11-03)
### Changed
- Migrated from JavaScript/NodeJS runtime to Java/GraalVM runtime, in order to fix [#33](https://github.com/digital-dj-tools/dj-data-converter/issues/33) and improve performance generally

## 0.4.1 (2020-02-04)
### Fixed
- Allow DateAdded blank string for Rekordbox tracks [#27](https://github.com/digital-dj-tools/dj-data-converter/issues/27)
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# dj-data-converter [![Build Status](https://dev.azure.com/digital-dj-tools/dj-data-converter/_apis/build/status/digital-dj-tools.dj-data-converter?branchName=master)](https://dev.azure.com/digital-dj-tools/dj-data-converter/_build/latest?definitionId=3&branchName=master)
# dj-data-converter [![Build](https://github.com/digital-dj-tools/dj-data-converter/workflows/Build/badge.svg)](https://github.com/digital-dj-tools/dj-data-converter/actions?query=workflow%3ABuild)

A command-line app for converting data files to and from different DJ software formats, such as [Traktor](https://www.native-instruments.com/en/products/traktor/dj-software/traktor-pro-3/), [Rekordbox](https://rekordbox.com/en/) and [Serato DJ](https://serato.com/dj).

Expand Down
113 changes: 0 additions & 113 deletions azure-pipelines.yml

This file was deleted.

3 changes: 2 additions & 1 deletion compile-opts.edn
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
{:output-to "out/cli.js"
{:output-dir "out"
:output-to "out/cli.js"
:target :nodejs
:optimizations :simple
:source-map "out/cli.js.map"}
36 changes: 26 additions & 10 deletions deps.edn
Original file line number Diff line number Diff line change
@@ -1,24 +1,22 @@
{:paths ["src"]
:deps
{org.clojure/clojure {:mvn/version "1.10.1"}
org.clojure/clojurescript {:mvn/version "1.10.520"}
{org.clojure/clojure {:mvn/version "1.10.2-alpha2"}
org.clojure/clojurescript {:mvn/version "1.10.773"}
camel-snake-kebab {:mvn/version "0.4.0"}
cljc.java-time {:mvn/version "0.1.6"}
cljc.java-time {:mvn/version "0.1.11"}
cljs-node-io {:mvn/version "1.1.2"}
com.taoensso/tufte {:mvn/version "2.1.0"}
digital-dj-tools/utils {:git/url "https://github.com/digital-dj-tools/utils.git"
:sha "a6e97edb71bdba724779c5118811a3cc56d6c877"}
digital-dj-tools/mp3-parser {:git/url "https://github.com/digital-dj-tools/mp3-parser.git"
:sha "515c4838dc0b1eadebe64ec5f0a9ed827cfed54c"}
:sha "207bd9f95b260659e050c20de90b7cbbf4376314"}
lambdaisland/uri {:mvn/version "1.4.54"}
org.clojure/data.xml {:mvn/version "0.2.0-alpha6"}
org.clojure/data.zip {:mvn/version "0.1.3"}
org.clojure/tools.cli {:mvn/version "0.4.2"}
metosin/spec-tools {:mvn/version "0.9.3"}
com.cemerick/url {:mvn/version "0.1.1" :exclusions [com.cemerick/clojurescript.test]}
tick {:mvn/version "0.4.19-alpha"
:exclusions [cljsjs/js-joda
cljsjs/js-joda-timezone
cljsjs/js-joda-locale-en-us]}}
metosin/spec-tools {:mvn/version "0.10.0"}
tick {:git/url "https://github.com/juxt/tick.git"
:sha "85aece71140a8a7372fb73f3dfad7cf87cdef9ab"}}
:aliases {:dev {:extra-paths ["test"]
:extra-deps {net.mikera/core.matrix {:mvn/version "0.62.0"}
org.clojure/test.check {:mvn/version "0.10.0"}
Expand All @@ -32,6 +30,24 @@
:sha "cb96e80f6f3d3b307c59cbeb49bb0dcb3a2a780b"}}
:main-opts ["-m" "cognitect.test-runner"
"-i" ":profile"]}
:uberjar {:extra-deps {luchiniatwork/cambada {:git/url "https://github.com/xfthhxk/cambada.git"
:sha "c8a01477c856fc28c53f34aa3f268f8eaf683869"}}
:main-opts ["-m" "cambada.uberjar"
"-m" "converter.cli"
"--app-artifact-id" "dj-data-converter"
"--app-version" "SNAPSHOT"]}
:native-image {:extra-deps {luchiniatwork/cambada {:git/url "https://github.com/xfthhxk/cambada.git"
:sha "c8a01477c856fc28c53f34aa3f268f8eaf683869"}}
:main-opts ["-m" "cambada.native-image"
"-m" "converter.cli"
"--graalvm-opt"
"-verbose"
"--graalvm-opt"
"-allow-incomplete-classpath"
"--graalvm-opt"
"-report-unsupported-elements-at-runtime"
"--graalvm-opt"
"J-Xmx8g"]}
:test-cljs {:extra-deps {olical/cljs-test-runner {:mvn/version "3.7.0"}}
:main-opts ["-m" "cljs-test-runner.main"
"-e" ":profile"]}
Expand Down
Loading

0 comments on commit 8f0bae5

Please sign in to comment.