Skip to content

Latest commit

 

History

History
 
 

unicorn

README.md

Numenta Unicorn

Cross-platform Desktop Application to demonstrate basic HTM functionality to users using their own data files.

WARNING! CURRENTLY UNDER HEAVY DEVELOPMENT

License

Dual Commercial and AGPLv3 License. See LICENSE.txt and http://numenta.org/licenses/.

Repository

The js/ directory contains Cross-platform Desktop Application GUI code, running Javascript/HTML/CSS/etc. on Node.js, Electron, and Google Chromium. It also contains non-GUI logic to run and manage HTM Models on the Unicorn model runner.

The py/ directory contains Unicorn ModelRunner and supporting code (Python / C++), which drives the main functionality of the app, which it is our goal to demo to the user.

DEPENDENCIES.md     # Module dependency overview file
LICENSE.txt         # Dual: Commercial and AGPLv3
README.md           # This file, a project overview
gulpfile.babel.js   # Babel.js ES6 Config file for the Gulp build tool
npm-debug.log       # NPM/node run output logs, not in source control
package.json        # Node.js `npm` packages, dependencies, and App config
py/                 # Unicorn ModelRunner and support Python/C++ code here!
  README.md         # Overview for HTM/NuPIC part of project
  requirements.txt  # Python pip package dependencies
  setup.py          # Backend Python project initialization tooling
  build/            # Backend build working output
  dist/             # Packaging and distribution output
  unicorn_backend/  # Unicorn Backend code, Model Runner, etc.
js/                 # Frontend+GUI that exposes NuPIC HTM functionality to the User
  README.md         # Overview for Frontend+GUI part of project
  assets/           # App assets
    icons/          # App icons (Electron app packaging, etc)
  browser/          # Javascript, HTML, CSS act as GUI inside electron's "Renderer Process"
    app.js          # Fluxible GUI Browser Web App entry, compiles to bundle.js
    index.html      # App main startup browser window contents
    actions/        # Fluxible Actions JS
    assets/         # Browser assets
      bundle/       # Auto-generated WebPack output compiled Web+Map bundle
      images/       # Browser images for display in UI (logos, etc)
    components/     # React view components JSX
    lib/            # Custom JS libs for inside the browser (engine clients)
      Fluxible/     # Fluxible lib helper code
        Plugins/    # Fluxbile plugins
      MaterialUI/   # Material UI lib helper code
      Unicorn/      # Unicorn lib helper code = Client libraries
    stores/         # Fluxible Stores JS
  config/           # JS Config files loaded by nconf
  database/         # File-based database storage (levelup + leveldown)
    schema/         # Database definition schemas in JSON
  docs/             # Output dir for generated JS docs (not saved in src)
  main/             # JS from electron's "Main Process" (i.e. engine services)
    loader.js       # Electron App entry point loader for main.js ES5 => ES6
    index.js        # ES6 Electron App main entry, creates browser GUI window and model runner engine
  samples/          # Sample .CSV data files to pre-load for user in GUI
logs/               # Logfile output of all kinds should end up here
node_modules/       # Where `npm` installs packages to, not in source control
tests/              # Unicorn project tests
  js/               # Javascript tests
    unit/           # Javascript unit tests
  py/               # Python tests
    integration/    # Python integration tests

Technology

Backend

Stack

See: py/requirements.txt

  • Languages:
  • Machine Intelligence:
    • NuPIC

Description

The Machine Intelligence behind this app is a technology known as Hierarchical Temporal Memory (HTM). NuPIC is Numenta's open source HTM engine. NuPIC runs on streams of data, predicting future values, and detecting pattern anomalies.

Frontend

Stack

See: package.json

Description

The Frontend contains code to manage Models on the Backend. It also contians the Graphical User Interface that allows users to explore HTM Models on their own data.

The GUI for this application is web code. Javascript, HTML, and CSS are loaded into a browser. For Desktop, this browser is a bare-bones Chrome window opened by the Electron framework. Electron also runs Node.js to connect with the host Operating System, allowing for cross-platform native controls. Since web code is notoriously asynchronous, we use js-csp to handle a/synchronous data flow.

In the browser, we run a one-way Uni-directional data flow, an Architecture known as "Flux".

Below is an example of tracing of our way through GUI initialization, GUI first loop run, and GUI loop continuation:

  1. Electron loads js/main.js
  2. .. which (or Browser directly) loads js/browser/index.html
  3. .. which loads js/browser/js/app.js
  4. .. which inits Fluxible
  5. .. and then Fluxible fires off an initial Action
  6. .. which dispatches Events with state data to Stores
  7. .. which then integrate state data into themselves
  8. .. and then View Components tied to updated Stores render
  9. .. and then The User interacts with the app firing off a new Action
  10. .. GOTO #6, RINSE and REPEAT.

Guidelines

  • ECMAScript Styleguide: npm run lint, Rules: .eslintrc
  • HTML5 @TODO
  • CSS3 @TODO

Setup

IMPORTANT: These setup instructions are only about if you care about running the full app with all its components (Electron app with packaged model runner). If you care about the python only - for example - then change to unicorn/py and follow the README instructions there.

Example of setting up development environment on Mac OS/X:

brew install git node
git clone https://github.com/numenta/numenta-apps
cd numenta-apps/unicorn
pip install -r py/requirements.txt
npm install
npm run prepare:python # to freeze model_runner.py so that it can be used in Unicorn.
export APPLICATION_LOG_DIR=logs  # to change to model_runner.py param?

Develop

Desktop App

Important note: when running npm run desktop the application will fail to find dist/model_runner if model_runner.py was never frozen before. Make sure to run npm run prepare:python beforehand, if any of the following applies:

  • The script model_runner.py has never been frozen (i.e. neither npm run build nor npm run prepare:python have been run before).
  • Changes were made to the the code under unicon/py.
  • Changes to NuPIC were made.

Start code via Electron as a Desktop App:

# Freeze the model runner if:
# - It was never frozen before
# - Changes were made to the the code under unicon/py
# - Changes to NuPIC were made.
npm run prepare:python

# desktop dev
npm run desktop
NODE_ENV=development npm run desktop  # same
npm run desktop | `npm bin`/bunyan    # pretty logs

# desktop prod
NODE_ENV=production npm run desktop

Code Documentation

Frontend code documentation can be generated and viewed by following the directions below. Comments are in JSDoc format, with output generated by ESDoc.

# generate documentation
npm run docs

# view documentation (mac os/x)
open js/docs/index.html

Development Problems?

  • Maybe the DB stored bad data? npm run clean:db
  • Sometimes node_modules/ directory can become corrupted, try cleaning and reinstalling: npm run clean ; npm run check
  • Local Git repo can get stuck with out-of-date JS somehow, so back up your current repo, and try again on a fresh repo clone.

Development Notes

  • Do not upgrade the following node/npm packages without syncing with the rest of the team first. The Mac build box is dependent on these versions, and must be upgraded in parallel.
    • npm
    • node
    • electron-packager
    • electron-prebuilt
    • is-electron-renderer
  • Make sure to update packages often, especially after pulling an update into your branch:
    • npm run check
    • pip list --outdated
  • Lint your code before creating pull requests:
    • npm run lint
  • Manual access to LevelDB file database from Lev command-line tool (Mac OS/X):
    • `npm bin`/lev $HOME/Library/Application\ Support/unicorn/database/
  • Remember, this is cross-platform. We need to support all main operating systems! Windows has no environment variables, etc. Paths should not be defined manually, but use the path library helper instead.
  • Awesome Node.js Links: https://github.com/sindresorhus/awesome-nodejs

Test

Write Unit and Integration tests using mocha test framework.

# All tests
npm run test

Unit

# Unit tests only
npm run test:unit

Integration

# Integration tests only
npm run test:integration

Pipeline

To generate reults appropriate for the pipeline mocha needs to run with a different set of options (see mocha.pipeline.opts).

# Pipeline: All tests
npm run test:pipeline

# Pipeline: Unit tests only
npm run test:pipeline:unit

# Pipeline: Integration tests only
npm run test:pipeline:integration

Build

Mac OS/X

Build the electron app with:

npm run build

The resulting .app can be found in unicorn/Unicorn-darwin-x64/

Windows

TODO

Release

Signing

  • You need a certificate type of “Developer ID Application” from Apple. This certificate usually has a common name in the form of Developer ID Application: YourCompany, Inc. (ABCDEFGHIJK).
  • Set the environment variable UNICORN_CERT_DEV_APPLE and sign the app:
export UNICORN_CERT_DEV_APPLE="Developer ID Application: Numenta, Inc. (ABCDEFGHIJK)"

App Store

Debug

NEED npm run blah examples here @TODO

Backend

  • @TODO Python, NuPIC, Models, debugging etc.

Node.js system-level

Command-line

Frontend + Browser

Other

@TODO

  • Spread global config around! nconf work
  • New Electron/Node4 doesn't need babel bundle anymore? (node5 for sure)
  • Add in Flow type checking
  • i18n l10n setup (es6 template strings? react intl? es6/7 solution?)
  • Electron integration test skeleton