Skip to content

Latest commit

 

History

History
282 lines (213 loc) · 9.14 KB

README.md

File metadata and controls

282 lines (213 loc) · 9.14 KB
Raw

a e o n

: (your '(LISP webserver))

CircleCI Builds

Netlify Builds

Netlify Status CircleCI quay.io codecov

aeon_sfo aeon_aws

What

A webserver in LISP FTW.

aeon

Aeon (Greek: Αἰών) is a Hellenistic deity associated with time, the orb or circle encompassing the universe, and the zodiac. The "time" represented by Aeon is unbounded, in contrast to Chronos as empirical time divided into past, present, and future. He is thus a god of eternity.

dev

lein figwheel

echo 'now in a seperate window...'
open http://localhost:8081/cljs

Alternatively...

lein ring server

TODOs

  • Escape SQL injection (note: no decent sql injection escaping libaries out there); used clojure.string/escape
  • Check if datasets conform to allowed map of datasets. Else, give a 404 instead of throwing an exception
  • Seeing this error: java.lang.OutOfMemoryError: Java heap space
  • Add more unit tests, test each report

Note: formerly valid sql injection: http://localhost:8080/api/equities'select%201'--/latest

API

$ curl [host].duckdns.org/api/equities/latest

{"body":[{"open":148.45,"date":"2017-05-18T07:00:00Z","adj_volume":16101229.00,"adj_close":148.06, \
  "ticker":"FB","adj_low":147.96,"ex_dividend":0.00,"close":148.06,"volume":16101229.00,"high":149.39, \
  "adj_high":149.39,"split_ratio":1.00,"low":147.96,"adj_open":148.45,"dataset":"WIKI"}]}

CORS

CORS (cross origin resource sharing) is way that allows resources from one domain to be requested from another domain. Browers however tended to restrict CORS on api calls, unless:

  • the API passes a pre-flight check, made by an HTTP OPTIONS method call (were it the same origin, only an HTTP GET would be called), and only after OPTIONS is successful (with the server including the Access-Control-Allow-Origin response header including the requester's origin), will the broswer make an HTTP GET; and
  • some headers in the response are sent by the API, specifically
    • Access-Control-Allow-Headers

Because Aeon deploys the static assets on a CDN (netlify), and the server lives on a separate domain, CORS needs to be enabled for the single-page-application on the CDN to work correctly.

Deploy

Make sure the jks is in the prod servers: scp -P [port] java_key_store [user]@[bastion].:~

ssh to the prod servers, start tmux, and fire:

ssh [alias].
tm
deploy/bin/run-docker

environment variables

export ro_jdbc_db_uri='jdbc:postgresql://[host]:[port]/[db_name]?sslmode=require&user=[user]&password=[pass]'
export quandl_api_key=''
# to run athena queries
export jdbc_athena_uri='jdbc:awsathena://athena.[region].amazonaws.com:[port]?s3_staging_dir=s3://aws-athena-query-results-[id]-[region]&query_results_encryption_option=SSE_S3&user=$aws_access_key_id&password=$aws_secret_access_key'

Rotating SSL certificates

https://letsencrypt.org/ certificates are only good for 3 months; as such, this neccessitates a periodic rotation of certificates.

The SSL termination hosted by pfsense is automated automated by the pfsense ACME package's cron job, which runs at 16 3 * * *. If for some reason it hasn't renewed, ssh onto pfsense and run the command: /usr/local/pkg/acme/acme_command.sh "renewall". It may fail because it can't write a txt record to a duckdns subdomain. The fix is confirm the subdomains for the certificate and duckdns are the same. Also confirm the api keys for the various subdomains. After that, HAproxy may restart, and the new certificate will be live: minimal effort!

For externally hosted sites, run the dev-resources/ssl-certs-renew-externally-hosted script. You just need to be on the VPN so pfsense. resolves. The script can be run as a shell script, or each command individually/interactively. After completion, log in to the external server's tmux session and restart the server.

Debugging

  • Did the jks made it onto the host ok? (cat ~/[jks]; ls -la ~/[jks])
  • Did the jks made it into the container ok? (cat /[jks]; ls -la /[jks])
  • Web server Listening on the port? (via another shell in the container)?
  • If in a VM, port forwarding in the VM? (netstat -tulpen or sockstat -l 4)
  • Port forwarding on the host? (VBoxManage controlvm "default" natpf1 "tcp-port8443,tcp,,8443,,8443";
  • Latest docker image! (key!)

Did it work? (locally)

curl localhost:8080
curl --insecure localhost:8443

Did it work? (externally)

curl $(curl v4.ifconfig.co 2>/dev/null)
curl --insecure "https://$(curl v4.ifconfig.co 2>/dev/null)"
curl --insecure https://$(nslookup [host]-aws.duckdns.org | grep Address | tail -n1 | awk '{print $2}')
curl https://[host].duckdns.org

Networking set up on AWS

VPC

Network ACL

Protocol Scheme Port    Source    Allow/Deny
TCP      HTTP   (80)    0.0.0.0/0 ALLOW
TCP      HTTPS  (443)   0.0.0.0/0 ALLOW

Security Group

Protocol Scheme Port    Source    Allow/Deny
TCP      HTTP   (80)    0.0.0.0/0 ALLOW
TCP      HTTPS  (443)   0.0.0.0/0 ALLOW

On the server

sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080
sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8443

Revert

sudo iptables -t nat -D PREROUTING 1

Networking set up on FreeBSD (docker-machine / VirtualBox)

Need to portforward host to guest ports.

VBoxManage controlvm "default" natpf1 "tcp-port8080,tcp,,8080,,8080";
VBoxManage controlvm "default" natpf1 "tcp-port8443,tcp,,8443,,8443";

Optional set up:

docker-machine create --driver "virtualbox" --virtualbox-hostonly-cidr "192.168.99.1/24" default
docker-machine stop default
docker-machine start default
docker-machine regenerate-certs default
echo "192.168.99.100 docker" >>/etc/hosts

Security

Port scanning

sudo nmap -v -sS -A -T4 $(nslookup [host]-aws.duckdns.org | grep Address | tail -n1 | awk '{print $2}')
sudo nmap -sS -O $(nslookup [host]-aws.duckdns.org | grep Address | tail -n1 | awk '{print $2}')
nmap --top-ports 5000 -T4 -sC $(nslookup [host]-aws.duckdns.org | grep Address | tail -n1 | awk '{print $2}')

Third party observations

Git

Git remotes

git remote add pi-vpn ssh://[user]@[bastion].:43/~/deploy/git/aeon.git
git remote add pi-home ssh://[user]@[host]/~/deploy/git/aeon.git

Pre-commit hook to update pom.xml

vim .git/hooks/pre-commit

#!/usr/bin/env bash

lein pom 2>&1 dev-resources/pom.xml

lein

Dependencies

lein deps :tree 2>deps 1>/dev/null && vim deps

cljfmt

lein cljfmt check ; lein cljfmt fix

kibits

lein kibit >tmp ; vim tmp && rm tmp

Custom jdbc/query

(defn- prepare-statement
  [sql params]
  (loop [sql sql
         kvs (map identity params)]
    (if (empty? kvs)
      sql
      (let [[[k v] & others] kvs]
        (recur (string/replace sql (str k) (str (jdbc/sql-value v)))
               others)))))

(defn query
  ([sql]
   (query sql {}))
  ([sql params]
   (with-open [conn (-> :ro-db-jdbc-uri env java.sql.DriverManager/getConnection)]
     (let [sql     (-> sql
                       (string/replace #";" "")
                       (string/replace #"--" "")
                       (string/replace #"\/" "")
                       (string/replace #"\/\*" "")
                       (string/replace #"\*\\" "")
                       (prepare-statement params))
           results (-> conn
                       (.createStatement)
                       (.executeQuery sql))]
       (jdbc/metadata-result results)))))

Resources