Skip to content

Latest commit

 

History

History
202 lines (188 loc) · 9 KB

wat.org

File metadata and controls

202 lines (188 loc) · 9 KB

Like SimCity, but for Lockport (or any town).

What should it do?

  • model traffic & roads & signals
  • model neighborhoods & real-estate
  • model police & crime
  • model fire & emergency
  • model schools
  • model taxes & costs
  • model local government & bonds
  • be fun
  • model up and coming events?
  • model skatepark?

But, how?

Implementation

  • Simulations need distributed computing?
    • Or can all be done on client (browser?)
    • Just drawing on client will be expensive…
    • erlang backend + javascript frontend
    • clojure + cljs? (shared code)
    • write the whole thing in javascript
      • nodejs deployments
    • wasm
  • Model vs View
  • It’s a map
  • It’s SimCity
  • Needs timeline / scrubbing
  • It’s almost like the financial/charting projection work
  • It relates to Kman’s map coloring work
  • Erlang, like distributed model trains
  • Queryable
  • Fast

Why Erlang?

I don’t need real-time?

“New hardware is increasing parallel, so new programming languages must support concurrency or they will die.”

Program multi-core. Multi-computer.

OARSYSTEM data

use mkgeo library

./scrape-oars fetch

Traceback (most recent call last): File “./scrape-oars”, line 7, in <module> import requests ImportError: No module named requests Need PIP pip install requests?

OSError: [Errno 2] No such file or directory: ‘data/cityoflockport/oars-2019’

mkdir -p data/cityoflockport/oars-2019

AttributeError: ‘NoneType’ object has no attribute ‘findAll’

$ python bin/scrape-oars ls-streets home/tim.local/lib/python2.7/site-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown. warnings.warn(warning, RequestsDependencyWarning) 2020-04-29 19:38:35 fetching url http://www.cityoflockport.oarsystem.com/assessment/main.asp?swis=140200; 2020-04-29 19:38:36 503 2020-04-29 19:38:36 parsing street list Traceback (most recent call last): File “bin/scrape-oars”, line 293, in <module> crawl(STREET_CRAWL, 1, offset) File “bin/scrape-oars”, line 243, in crawl for i,street in enumerate(get_street_list()[offset::iterator]): File “bin/scrape-oars”, line 235, in get_street_list for option in soup.find(‘select’).findAll(‘option’): AttributeError: ‘NoneType’ object has no attribute ‘findAll’

bail, and write it all in erlang

  • didn’t have a python env setup (pip, packages, filesystem/mkdir)
  • you can call out to python from erlang, but, you need the dependencies?
  • erlang wants to be a running, always on, system of processes and mkgeo is meant to be a script that’s run then completes.

publish an Erlang OARS library?

Webserver

Build tools

  • Makefile?

Rebar?

“Rebar3 is also a self-contained Erlang script. It is easy to distribute or embed directly in a project.”

Install from source:

$ git clone https://github.com/erlang/rebar3.git
$ cd rebar3
$ ./bootstrap

Outputs a ./rebar3 escript file https://rebar3.org/docs/getting-started/

“Installed” a non-escript version with:

$ ./rebar3 local install
===> Extracting rebar3 libs to /home/tim/.cache/rebar3/vsns/3.14.2+build.4880.ref3ef4f08/lib...
===> Writing rebar3 run script /home/tim/.cache/rebar3/bin/rebar3...
===> Add to $PATH for use: export PATH=/home/tim/.cache/rebar3/bin:$PATH
  • and added to $PATH as mentioned
  • Removed the rebar3 escript as mentioned in the getting started guide

rebar expects OTP project structure…

I suppose I should be learning and following OTP guidelines

I wondered about how to install Cowboy under TODO’s Webserver.

  1. Create a test project: rebar3 new umbrella myproj
  2. Read about Basic Usage: https://rebar3.org/docs/basic_usage/
    • We probably want a single application, not an umbrella project
  3. Read about Releases: https://rebar3.org/docs/deployment/releases/
  4. Read about Workflow: https://rebar3.org/docs/workflow/
    • So, ‘lib’ or ‘app’ projects are for libraries to be used within other projects
    • I think we want a ‘release’ project, since we will be deploying it on its own
    • I guess ‘umbrella’ is still an option, although that seems not quite right
  5. Read about Setup: https://adoptingerlang.org/docs/development/setup/
    • Add some Erlang flags to .bashrc for a modern feel
      • Unicode support
      • Shell history
    • [ ] Should check out the Emacs section for Ivy, Hydra, Counsel, and Swiper.
  6. Read about OTP at a High Level: https://adoptingerlang.org/docs/development/otp_high_level/
    • For project setup / structure tips
    • “Erlang has a strong opinion about how your applications should be structured, the level of isolation they should have, and a separation between what Erlang’s VM can do, and what your software can do. It’s not just a programming language, it’s a whole framework for building systems. Understanding its core principles is the key to getting started fast without having to rewrite everything later: it ensures that all applications can fit well together, that updates can be done live, and that your code is easy to instrument and make observable.”
    • “There are basically two variants of OTP applications: Library Applications, which are just collections of modules, and Runnable Applications, which contain a collection of modules, but also specify a stateful process structure stored under a supervision tree.”
  7. Read about Dependencies: https://adoptingerlang.org/docs/development/dependencies/
    • Deps are added to rebar.config, similar to .gemfile or .csproj
    • To get cowboy as a dependency: Add {deps, [{cowboy, “2.8.0”}]} to rebar.config > rebar3 compile (to download dependency) > rebar3 tree (to see list of (transitive) dependencies)
    • Need to add in other places as well?
      • Under the `relx` key in rebar.config?
      • And/or in the .app.src file?
    • After adding Cowboy as a dep and in the `relx` section of rebar.config, running 
      > rebar3 release

      will include the Cowboy and it’s dependency in the release:

      > ls -al _build/default/rel/myrel/lib/
lrwxrwxrwx 1 tim tim   34 Jan  2 21:21 asn1-5.0.9 -> /home/tim/kerl/22.2/lib/asn1-5.0.9
lrwxrwxrwx 1 tim tim   57 Jan  2 21:21 cowboy-2.8.0 -> /home/tim/Code/rebar-test/myrel/_build/default/lib/cowboy
lrwxrwxrwx 1 tim tim   57 Jan  2 21:21 cowlib-2.9.1 -> /home/tim/Code/rebar-test/myrel/_build/default/lib/cowlib
lrwxrwxrwx 1 tim tim   36 Jan  2 21:21 crypto-4.6.3 -> /home/tim/kerl/22.2/lib/crypto-4.6.3
lrwxrwxrwx 1 tim tim   36 Jan  2 21:21 kernel-6.5.1 -> /home/tim/kerl/22.2/lib/kernel-6.5.1
lrwxrwxrwx 1 tim tim   56 Jan  2 21:21 myrel-0.1.0 -> /home/tim/Code/rebar-test/myrel/_build/default/lib/myrel
lrwxrwxrwx 1 tim tim   40 Jan  2 21:21 public_key-1.7.1 -> /home/tim/kerl/22.2/lib/public_key-1.7.1
lrwxrwxrwx 1 tim tim   56 Jan  2 21:21 ranch-1.7.1 -> /home/tim/Code/rebar-test/myrel/_build/default/lib/ranch
lrwxrwxrwx 1 tim tim   34 Jan  2 21:21 sasl-3.4.1 -> /home/tim/kerl/22.2/lib/sasl-3.4.1
lrwxrwxrwx 1 tim tim   31 Jan  2 21:21 ssl-9.5 -> /home/tim/kerl/22.2/lib/ssl-9.5
lrwxrwxrwx 1 tim tim   35 Jan  2 21:21 stdlib-3.11 -> /home/tim/kerl/22.2/lib/stdlib-3.11
      • For a production release: > rebar3 as prod release

Get a landing page up

  • Easy URL
  • Describe the project
  • Way to get in touch (email)
  • Link to source code

Deployment

Terraform / Nomad / Docker / Podman / GitHub / Anisible …

Tech debt

Supervisor for request throttling

In oars.erl, from_url wants to send a message to the one process handling http requests to the oars system, so that we do not send hundreds of requests at once. That one process is started in oars:run_services() and will be restarted if it fails. However, I think this should be turned into a gen_supervisor & worker configuration.

Edit: fixed by converting oars.erl to implement gen_server behavior.