At first, it is just a matter of exploration ... and then more exploration ... and then more informed exploration ... and then later ... it will always be a matter of exploration, because there's always some sort of gamechanging dev community or new adaptation of technology that matters ... we're not talking about hype or marketing pushes -- this is about emergent communities of researchers, scientists and knowledge professionals developing new communities / hubs around new methods ... this is about learning, it's not about the George Jetson futuristic projections ... learning is about adapting and finding out what IS working better ... not what will work, what IS working and that is always changing ... but we always get better by learning from others and sharing our skills for how we ARE learning.
This means that our "app" or learning and research toolchain will be constructed using a dogfooding approach.
We cannot emphasize the dogfooding nature of this enough ... our little Sci Ops knowledge graph project might eventually mature into a community project ... but maybe not ... at first, it's just us learning and dogfooding a toolchain and contributing to those learning open source projects which are most efficaciously advancing the learning cause.
We know that we are not alone in this ... we see research scientists [or their grad assistant Ph. D. candidates] developing their own SciOps workflow will rely upon dogfooding some sort of some-established, but still evolving and improving CI/CD process for Sci Ops ... of course that means fully drinking the Kool-Aid on continuous integration, continuous delivery, and continuous deployment architectures ... with some sort of fluid, evolving Sci Ops knowledge graph analytic data engineering lifecycle.
We are not interested in re-inventing any wheels ... this is all about simplifying reuse and dogfooding, accelerated dogfooding and then new ways of doing advanced [dogfooding](https://en.wikipAt first, it is just a matter of exploration ... and then more exploration ... and then more informed exploration ... and then later ... it will always be a matter of exploration, because there's always some sort of gamechanging dev community or new adaptation of technology that matters ... we're not talking about hype or marketing pushes -- this is about emergent communities of researchers, scientists and knowledge professionals developing new communities / hubs around new methods ... this is about learning, it's not about the George Jetson futuristic projections ... learning is about adapting and finding out what IS working better ... not what will work, what IS working and that is always changing ... but we always get better by learning from others and sharing our skills for how we ARE learning.
This means that our "app" or learning and research toolchain will be constructed using a dogfooding approach.
We cannot emphasize the dogfooding nature of this enough ... our little Sci Ops knowledge graph project might eventually mature into a community project ... but maybe not ... at first, it's just us learning and dogfooding a toolchain and contributing to those learning open source projects which are most efficaciously advancing the learning cause.
We know that we are not alone in this ... we see research scientists [or their grad assistant Ph. D. candidates] developing their own SciOps workflow will rely upon dogfooding some sort of some-established, but still evolving and improving CI/CD process for Sci Ops ... of course that means fully drinking the Kool-Aid on continuous integration, continuous delivery, and continuous deployment architectures ... with some sort of fluid, evolving Sci Ops knowledge graph analytic data engineering lifecycle.
We are not interested in re-inventing any wheels ... this is all about simplifying reuse and dogfooding, accelerated dogfooding and new ways of dogfooding ... a such, it will involve different, new but now firmly-established methods that MANY have been refining and putting to work in the last year or two, such as:
-
Git, GitHub Actions, GitLab CI/CD, Gerrit code review and probably some sort of responsive, light-weight, more usable front-end such as Lit.dev
-
containers, virtual environments, protected spaces like FreeBSD jails or Fedora toolboxes, Podman, Docker, K8s ... it's all about sharability and reproducibility
-
Various package management architectures ... such as Conda package mgmt, FlatPak, RPEL, dnf, yum, apt, apk pip ... reproducibility, reproducibility, reproducibility
-
A Pythonic data exploration infrastructure including the Jupyter architecture and the standard .ipynb JSON notebook schema, SnakeMake workflow mgmt system, as well as others, like Google Colaboratory or Amazon EMR JupyterHub or others which use a similar, nearly standard architecture.
-
Rentable BigCompute, especially in the realm of high performance computing (HPC) ... not just GCP, AWS, Azure which go way beyond the Jupyter Notebook level ... but there are also plenty of Jupyter Notebook alternatives for exploratory analysis including Google Colab, Paperspace Gradient / Core, Lambda GPU Cloud, VAST.ai, GitHub Codespaces, et al. The advent of rental BigCompute really kind of changes everything in the realm of computational science ... and that does not mean we forget about inexpensive, highly usable ubiquitous devices like smartphones, chromebooks, laptops ... and maybe even things like using the GPUs in gaming setups or CAD workstations or bitcoin mining rigs. BigCompute *is everywhere.*edia.org/wiki/Eating_your_own_dog_food) ... as such, it will involve different, new but now firmly-established methods that MANY have been refining and putting to work in the last year or two, such as:
-
Git, GitHub Actions, GitLab CI/CD, Gerrit code review and probably some sort of responsive, light-weight, more usable front-end such as Lit.dev
-
containers, virtual environments, protected spaces like FreeBSD jails or Fedora toolboxes, Podman, Docker, K8s ... it's all about sharability and reproducibility
-
Various package management architectures ... such as Conda package mgmt, FlatPak, RPEL, dnf, yum, apt, apk pip ... reproducibility, reproducibility, reproducibility
-
A Pythonic data exploration infrastructure including the Jupyter architecture and the standard .ipynb JSON notebook schema, SnakeMake workflow mgmt system, as well as others, like Google Colaboratory or Amazon EMR JupyterHub or others which use a similar, nearly standard architecture.
-
Rentable BigCompute, especially in the realm of high performance computing (HPC) ... not just GCP, AWS, Azure which go way beyond the Jupyter Notebook level ... but there are also plenty of Jupyter Notebook alternatives for exploratory analysis including Google Colab, Paperspace Gradient / Core, Lambda GPU Cloud, VAST.ai, GitHub Codespaces, et al. The advent of rental BigCompute really kind of changes everything in the realm of computational science ... and that does not mean we forget about inexpensive, highly usable ubiquitous devices like smartphones, chromebooks, laptops ... and maybe even things like using the GPUs in gaming setups or CAD workstations or bitcoin mining rigs. BigCompute is everywhere.