Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce workflow run crate #32

Merged
merged 70 commits into from
Nov 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
7251cfa
starting to split the functions
inutano Nov 9, 2022
59d6c1b
createAction and more
inutano Nov 9, 2022
d806c58
add input/output in the workflow
inutano Nov 9, 2022
6d50fbe
make them arrays
inutano Nov 9, 2022
caada19
add mentions workflow run
inutano Nov 9, 2022
0c16bda
use subjectOf to link log files from a workflow run
inutano Nov 10, 2022
a1edde5
copy the original to split
inutano Nov 10, 2022
a5f8ebe
leave only run-crate related terms
inutano Dec 20, 2022
7846d1e
add extra terms
inutano Feb 9, 2023
6c588d2
use encodingFormat for specifying EDAM
inutano Feb 9, 2023
45db975
hide TESTING_EXTRA_TERMS
inutano Feb 9, 2023
68755ee
start refactoring
inutano Feb 9, 2023
d100524
define EDAM
inutano Feb 9, 2023
4e7e2ed
latest stable
inutano Feb 9, 2023
aaff313
comment out unused functions
inutano Feb 9, 2023
0ad018a
hide unused type defitinition
inutano Feb 9, 2023
e880bac
remove pid, uid, gid, mode
inutano Feb 9, 2023
01ade04
remove fileStats for main wf file but still remains
inutano Feb 9, 2023
2bf3a19
recovered necessary functions, fix main wf file object
inutano Feb 9, 2023
a220a3e
fix command not found
inutano Feb 9, 2023
2aa534b
fix ComputatinalWorkflow object initialization
inutano Jun 8, 2023
59d21bf
refactor
inutano Jun 8, 2023
609ccfb
refactor
inutano Jun 8, 2023
1499a4b
refactor wf run
inutano Jun 8, 2023
733ef3b
add example ro-crate output
inutano Jun 8, 2023
36efeda
fix conformsTo targets
inutano Jun 9, 2023
f1a7a9e
add exampleOfWork to input data entity
inutano Jun 9, 2023
46b212b
change type of readme as a description
inutano Jun 13, 2023
2d7d95d
remove yevis-id for now
inutano Jun 13, 2023
5066654
easy fix for duplicated v problem
inutano Jun 13, 2023
c345484
remove type WorkflowAttachment
inutano Jun 13, 2023
f885dd0
use actionStatus
inutano Jun 13, 2023
0529283
remove logs from output of mainEntity
inutano Jun 13, 2023
055c609
remove unused extra terms
inutano Jun 13, 2023
bbd94f4
rollback stats generation functions
inutano Jun 13, 2023
12d3df8
add status label to actionStatus
inutano Jun 13, 2023
dc2d867
Improve python package script
suecharo Jun 13, 2023
2af11c4
More generalized uwsgi config
suecharo Jun 13, 2023
663bbed
Update docker env
suecharo Jun 13, 2023
4372d04
Fix to use Sapporo version as gh_ref of ctx instead of main. Fix over…
suecharo Jun 14, 2023
e8fe62e
Fix mypy and pylint errors
suecharo Jun 15, 2023
26f1784
Remove prev terms csv file
suecharo Jun 15, 2023
21ce8a5
Fix for cromwell
suecharo Jun 15, 2023
23a875b
Fix test for cromwell
suecharo Jun 16, 2023
248a8b9
Add test for ro-crate feature
suecharo Jun 16, 2023
6fd6c77
Support for both path and location of file object in cwl params
suecharo Jul 22, 2023
8a96482
Add UserAgent header for GitHub
suecharo Jul 22, 2023
56348b7
Use hasPart instead to attachment
suecharo Jul 22, 2023
76f4b28
Change actionStatus to literal instead of object
suecharo Jul 22, 2023
7b9332c
use schema.org DataType
inutano Jul 22, 2023
f5a86dd
Fix tests
suecharo Jul 22, 2023
a906d09
Remove prev terms
suecharo Jul 22, 2023
327e5a9
Fix jq error
suecharo Jul 22, 2023
20f609f
Change extra terms to sapporo's ro-terms
suecharo Jul 25, 2023
e9a56a5
Remove prev local terms
suecharo Jul 25, 2023
362cd7f
Update ro-crate small example
suecharo Jul 25, 2023
9308c16
Fix CI/CD scripts ubuntu version
suecharo Jul 25, 2023
d9b5a79
Add run-dir from which ro-crate-metadata.json was generated
suecharo Jul 25, 2023
85241ea
Add log files to ro-crate_dir
suecharo Aug 3, 2023
17f2222
Fix extra context
suecharo Aug 3, 2023
31ea971
Add entities about run crate profiles
suecharo Aug 3, 2023
aa123d6
Link to from the root data entity to the CreateAction
suecharo Aug 3, 2023
cbb958e
Omit decription props
suecharo Aug 3, 2023
03681e7
Fix context terms
suecharo Aug 29, 2023
c9e8b36
Add license props
suecharo Aug 29, 2023
1658102
remove input/output from json file
inutano Oct 23, 2023
2399fe8
remove files from index
inutano Oct 23, 2023
79d2fc0
revert file remove
inutano Oct 24, 2023
0b7e990
Delete of input, output props and Formal Parameter in mainWorkflow en…
suecharo Oct 25, 2023
de0f18b
Fix flake8 error
suecharo Nov 2, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@
!README_ja.md
!README.md
!sapporo
!sapporo-wes-*
!sapporo-wes-1-0-1-openapi-spec.yml
!setup.py
!uwsgi.yml
6 changes: 3 additions & 3 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
jobs:
build_and_deploy_wheel:
if: github.repository == 'sapporo-wes/sapporo-service'
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
steps:
- name: "Checkout"
uses: actions/checkout@v3
Expand Down Expand Up @@ -48,7 +48,7 @@ jobs:

deploy_ghcr:
if: github.repository == 'sapporo-wes/sapporo-service'
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
steps:
- name: "Checkout"
uses: actions/checkout@v3
Expand Down Expand Up @@ -81,7 +81,7 @@ jobs:
create_release:
if: github.repository == 'sapporo-wes/sapporo-service'
needs: [build_and_deploy_wheel]
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
steps:
- name: "Download sdist"
uses: actions/download-artifact@v2
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/flake8.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ on:

jobs:
flake8:
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/isort.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ on:

jobs:
isort:
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/mypy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ on:

jobs:
mypy:
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v3
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ on:

jobs:
pytest:
runs-on: ubuntu-latest
runs-on: ubuntu-20.04

steps:
- uses: actions/checkout@v3
Expand Down
13 changes: 6 additions & 7 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,18 +1,17 @@
FROM python:3.8.13-buster as builder
FROM python:3.8.17-bullseye as builder

WORKDIR /app
COPY . .
RUN python3 -m pip install --no-cache-dir --progress-bar off -U pip setuptools wheel && \
python3 -m pip install --no-cache-dir --progress-bar off .

FROM python:3.8.13-slim-buster
FROM python:3.8.17-slim-bullseye

LABEL org.opencontainers.image.authors="Bioinformatics and DDBJ Center <t.ohta@nig.ac.jp>"
LABEL org.opencontainers.image.url="https://github.com/sapporo-wes/sapporo-service"
LABEL org.opencontainers.image.source="https://github.com/sapporo-wes/sapporo-service/blob/main/Dockerfile"
LABEL org.opencontainers.image.version="1.4.9"
LABEL org.opencontainers.image.description="sapporo-service is a standard implementation conforming to the \
Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API specification."
LABEL org.opencontainers.image.description="sapporo-service is a standard implementation conforming to the Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API specification."
LABEL org.opencontainers.image.licenses="Apache2.0"

RUN apt update && \
Expand All @@ -26,10 +25,10 @@ RUN apt update && \
rm -rf /var/lib/apt/lists/*

WORKDIR /tmp
RUN curl -O https://download.docker.com/linux/static/stable/$(uname -m)/docker-20.10.9.tgz && \
tar -xzf docker-20.10.9.tgz && \
RUN curl -O https://download.docker.com/linux/static/stable/$(uname -m)/docker-23.0.6.tgz && \
tar -xzf docker-23.0.6.tgz && \
mv docker/* /usr/bin/ && \
rm -rf docker docker-20.10.9.tgz
rm -rf docker docker-23.0.6.tgz

COPY --from=builder /usr/local/lib/python3.8/site-packages /usr/local/lib/python3.8/site-packages
COPY --from=builder /usr/local/bin/uwsgi /usr/local/bin/uwsgi
Expand Down
8 changes: 4 additions & 4 deletions Dockerfile-dev
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.8.13-buster
FROM python:3.8.17-bullseye

RUN apt update && \
apt install -y --no-install-recommends \
Expand All @@ -9,10 +9,10 @@ RUN apt update && \
rm -rf /var/lib/apt/lists/*

WORKDIR /tmp
RUN curl -O https://download.docker.com/linux/static/stable/$(uname -m)/docker-20.10.9.tgz && \
tar -xzf docker-20.10.9.tgz && \
RUN curl -O https://download.docker.com/linux/static/stable/$(uname -m)/docker-23.0.6.tgz && \
tar -xzf docker-23.0.6.tgz && \
mv docker/* /usr/bin/ && \
rm -rf docker docker-20.10.9.tgz
rm -rf docker docker-23.0.6.tgz

WORKDIR /app
COPY . .
Expand Down
9 changes: 8 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
include LICENSE
include sapporo-wes-spec.swagger.yml
include README_ja.md
include sapporo-wes-1-0-1-openapi-spec.yml
include sapporo/executable_workflows.json
include sapporo/executable_workflows.schema.json
include sapporo/ro-terms.csv
include sapporo/run.sh
include sapporo/service-info.json
include sapporo/service-info.schema.json
include uwsgi.yml
9 changes: 9 additions & 0 deletions dockerfile-for-wf-engines/Dockerfile-cromwell
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM broadinstitute/cromwell:80

# install docker-ce
RUN apt update && \
apt install -y apt-transport-https ca-certificates curl software-properties-common && \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - && \
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" && \
apt update && \
apt install -y docker-ce
31 changes: 31 additions & 0 deletions dockerfile-for-wf-engines/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Dockerfile for Workflow Engines

## Why?

When using the official docker image such as `broadinstitute/cromwell:85`,

```
docker: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by docker)
```

an error like the above appears.

In Sapporo, within `run.sh`,

```bash
local container="broadinstitute/cromwell:85"
docker run --rm ${D_SOCK} -v ${run_dir}:${run_dir} -v /tmp:/tmp -v /usr/bin/docker:/usr/bin/docker -w=${exe_dir} ${container} run ${wf_engine_params} ${wf_url} -i ${wf_params} -m ${exe_dir}/metadata.json
```

We mount the docker command and docker socket on the host side and use them, but because the glibc version required by Docker on the host side and the glibc provided by the container image are different, an error like the one above occurs.

Therefore, we create, provide, and use an workflow engine image with docker-ce installed, using the official image of the workflow engine as the base.

## Cromwell

To build and push:

```bash
$ docker build -f ./Dockerfile-cromwell -t ghcr.io/sapporo-wes/cromwell-with-docker:80 .
$ docker push ghcr.io/sapporo-wes/cromwell-with-docker:80
```
4 changes: 2 additions & 2 deletions release.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,10 @@ echo "Rewrite files."
sed -i "s/version=\"$PREV_VERSION\"/version=\"$NEW_VERSION\"/g" Dockerfile
sed -i "s/sapporo-service:$PREV_VERSION/sapporo-service:$NEW_VERSION/g" docker-compose.yml
sed -i "s/\"sapporo-version\": \"$PREV_VERSION\"/\"sapporo-version\": \"$NEW_VERSION\"/g" sapporo/service-info.json
sed -i "s/version=\"$PREV_VERSION\"/version=\"$NEW_VERSION\"/g" setup.py
sed -i "s/__version__ = \"$PREV_VERSION\"/__version__ = \"$NEW_VERSION\"/g" sapporo/__init__.py

echo "Commit and push."
git add Dockerfile docker-compose.yml sapporo/service-info.json setup.py
git add Dockerfile docker-compose.yml sapporo/service-info.json sapporo/__init__.py
git commit -m "Update version to $NEW_VERSION"
git push origin main

Expand Down
3 changes: 3 additions & 0 deletions sapporo/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/usr/bin/env python3
# coding: utf-8
__version__ = "1.4.9"
62 changes: 0 additions & 62 deletions sapporo/ro-terms.csv

This file was deleted.

Loading
Loading