Data sources

I got this question by email:

James,

First of all, thank you for sharing information about this project!

I've been looking over your Github project. There is something that I'm curious about ... The JSON files, that are located in the data folder {objects and config}, are these created programmatically or manually?

My response

As you've seen, I have 2 datasets that I've included with the project. I edited the config.json files manually - for example, working on the constraints section to try to help the relationships in the data emerge more clearly. I'll explain a bit about the objects.json files below.

The first dataset (default) represents the objects and data flows for a data analysis and reporting project I worked on at a previous job. I generated this data using a combination of various scripts and procedures specific to that project, and some manual editing as well, I think.

Then I realized that this could be a more generically useful tool for graph visualization, so I stripped the identifying information out of the default dataset so I could publish the code. I can't share the full version, but it is more usable as documentation. For example, the .mkdn files contain some information about what each object is doing and where it can be found.

The second dataset (les-mis) is a standard computer science dataset originally encoded by Donald Knuth. Then I transformed it into the format I needed, I don't remember exactly how but probably again using some kind of moderately clever hack. I don't know what process Knuth used to create the dataset (something exceptionally clever, I'm sure) but here is some more info. https://github.com/mmlc/lesmiserables-character-network