Skip to content

Latest commit

 

History

History
223 lines (178 loc) · 15.2 KB

README.md

File metadata and controls

223 lines (178 loc) · 15.2 KB
Raw

Reinforcement Learning for Algorithmic Trading

Getting Started

Python 3.6.5

I would reccomend creating a virtual enviorment to avoid dependancy issues. You can create a virtual enviorment using Virtualenv if you don't already have it installed in your current python interpreter. The current dependancies are in requirements-cpu.txt or gpu equivalent, and can be installed by the following commands.

pip3 install virtualenv
python3 -m virtualenv env

source env/bin/activate

pip install -r requirements-cpu.txt

The equivalent requirements for gpu support are inside requirements-gpu.txt.

Training

We are currently working on optimizing the distribution of funds between two assets. You run python main.py [source type], where the source type(s) are as follows:

  • markov
    • Markov memory 1 and a fixed asset with return rate 0
  • markov2
    • Markov memory 2 and a fixed asset with return rate 0
  • iid
    • IID uniform random variable and a fixed asset with return rate 0
  • mix
    • Markov memory 1 and IID uniform r.v
  • real
    • Real data

This will populate the contents of a Q table and display the result of following the policy obtained over 100 intervals of testing data. An overview of the code architechure is found below.

  • main.py
    • This is where you can specify episodes and initial investment
  • enviorment.py
    • Here you can manipulate paramaters of the trading enviorment such as action space, observation space, reward function and done flag
  • Q_table.py
    • This class defines learning rate, gamma, epsilon parameters and contains choice action and learning methods
  • utils.py
    • A series of methods used to import, generate and manipulate data for training
  • /Matlab/
    • A bunch of matlab scripts that have been used to determine quantizers, empirical distributions and policies.

Q Learning Results

Below are some observations thus far. Each example is derived with an initial investment of $100 for 10 episodes.

IID Source

When the input source is an iid random variable uniformly distributed on [-1,1] with steps of 0.2, the Q table is populated as follows. Note that there is no observation space, as return rates are independent.

Distribution into Source 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
None 0.013 -0.016 -0.11 -0.038 -0.009 -0.038 0.004 0.001 -0.038 -0.087 -0.123

Which indicates the best option is to invest %0 of capital into the IID stock at any given time.
When the input source is an iid random variable uniformly distributed on [0,0.3], the Q table is populated as follows.

Distribution into Source 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
None 1923.39 2243.24 2064.73 2121.45 2028.12 2122.95 2088.61 2155.73 2096.5 2124.11 4931.78

Which indicates the best option is to invest all capital into the IID stock at any given time.

Markov Memory 1 Source

Here the input source is markovian with the following transition probabilities.

If the possible return rates for the 3 states are -0.2, 0.0, 0.2 the Q table is populated as follows.

Distribution into Source 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Prev Value: 0 218.074 111.159 127.127 105.276 112.809 102.041 98.145 97.792 113.613 95.798 106.162
Prev Value: 0.2 181.137 191.67 261.255 187.992 172.182 179.056 223.266 186.9 182.004 217.171 629.632
Prev Value: -0.2 244.331 58.172 62.42 48.51 56.312 55.855 50.747 63.276 77.343 52.846 55.282

This deduces the policy that, when the previous value is:

  • 0 -> Do not invest
  • -0.2 -> Do not invest
  • 0.2 -> Invest %100 of capital into stock

If the return rates for the 3 possible states are -0.1, 0, 0.4, the Q table below is produced.

Distribution into Source 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Prev Value: -0.1 559.623 62.7048 60.9816 68.2782 61.0308 66.2113 58.6067 56.0653 64.5082 60.6457 56.6935
Prev Value: 0 69.0304 72.4873 64.9642 58.6928 61.7887 561.144 70.8124 69.7554 66.7248 63.1553 64.2676
Prev Value: 0.4 159.824 196.479 200.312 181.142 205.622 200.464 184.247 178.793 196.372 172.393 1281.63

This deduces the policy that, when the previous value is:

  • 0 -> Invest %50 of capital into stock
  • -0.1 -> Do not invest
  • 0.4 -> Invest %100 of capital into stock
    This results are consistend with the optimal investement decision derived in the ./Matlab/ scripts, which yields that the Q-learning is working effectivly.

Using the script ./Matlab/ModelFitting.m the one step transition matrix for IBM computed over the IBM return rates, (discluding the last 1000 samples for testing).

For states x < -.1%, -.1% < x < .1% and x > .1%. We can then enter this transition matrix and states in `utils.py` to generating 5000 samples, and then train the Q learning agent on these samples. Applying the policy obtained from Q-learning, we can then apply it to the testing data for IBM (the last 1000 values).

However, this quantization is generic and randomly chosen. Consider now repeating the experiment, however, we can determine the quantization of our training data using the Lloyd Max algorithm on the real data training set.

Applying this quantiation to `./Matlab/ModelFitting.m` we generate the following transition matrix.

Training a Q agent on the dataset generated according to this empirical distribution and applying the policy obtained to the testing data yeilds the following result.

Repeating this experiment with the Microsoft data, we first obtain a Llloyd Max quantizatizer as shown below.

Afterwards we can obtain an empiratical approximation of the 1 step transition matrix according to this quantizer, with respect to the MSFT data discluding the most recent 1000 entries.

Using this quantizer and transition matrix to generate training data for the Q agent, the policies obtained have the following result on our testing data.

Comparing Q Learning Results for Markov Orders

To better understand what markov order (if any) best represents the conditional dependence of past return rates we experiment with manipulating the obsersvations seen by the Q learning agent. To start, we consider the microsoft data. Below is the populated Q table if this stock is treated as iid. Here, the agent has no access to previous return rates so the observation is always the same.

Distribution into Source 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
None 39.2431 39.5498 39.2918 40.3238 39.5636 48.7792 39.8656 39.8663 39.4475 39.746 39.4284

This policy implies investing %90 of assets into the stock for any given time step. Applying such policy over the testing data yeilds.

Now, if we allow the Q learning agent to observe the previous return rate as `-1, 0, 1` correspoding to being in the negative interval, nuetral interval and positive interval as per the uniform quantization over the training data, we observe the following markov order 1 policy from the Q learning agent.
Distribution into Source 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Prev "Down" 8.69687 8.416 8.76448 8.32668 8.06351 8.44431 8.61975 19.2718 9.00001 7.86183 7.99984
Prev "Nuetral" 21.6007 21.3949 21.1213 21.2595 21.0212 21.3045 20.5782 21.1772 20.8734 28.6595 21.2023
Prev "Positive" 8.626 8.10509 18.6383 8.12764 8.08328 8.12089 7.71774 8.01588 8.39773 8.66198 8.14364

This markov policy yeilds the following performance over the testing data.

Allowing the Q agent to look at the previous two return rates yeilds the following Q table.

Distribution into Source 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Prev Seq: "Down", "Down" 1.46329 1.34375 1.31343 1.51526 1.36004 1.47882 4.51021 1.35624 1.63156 1.34464 1.29957
Prev Seq: "Down", "Nuetral" 4.35364 4.07422 3.88701 4.19443 4.04607 3.83908 4.51836 4.09829 10.6058 3.95363 4.04416
Prev Seq: "Down", "Positive" 1.16335 1.42665 1.21431 1.37276 1.42155 1.41948 1.36941 4.42286 1.15609 1.05923 1.19335
Prev Seq: "Nuetral", "Down" 2.88785 2.68302 3.32496 2.92446 2.70594 2.6983 2.91641 2.71681 7.94342 3.07691 2.84623
Prev Seq: "Nuetral", "Nuetral" 15.2012 15.0627 22.1063 14.9272 14.9772 15.1129 14.9588 14.7254 14.772 15.2999 15.2734
Prev Seq: "Nuetral", "Postive" 7.55047 3.08106 3.07143 3.17324 2.909 3.29652 2.78571 3.24299 3.09982 2.82638 3.24957
Prev Seq: "Positive", "Down" 4.64262 1.45491 1.49116 1.44225 1.37647 1.39101 1.6392 1.17189 1.3111 1.53721 1.38324
Prev Seq: "Positive", "Nuetral" 3.74981 4.25076 4.28741 9.85611 4.12424 4.65483 4.37102 4.29442 3.81727 4.28817 3.76588
Prev Seq: "Positive", "Positive" 1.22974 1.00224 1.16756 1.04358 1.34929 1.13015 1.25551 3.65231 0.896386 1.18419 1.17435

Applying this markov 2 policy to the testing set yeilds

Interestingly engough, the seccond order memory performed worse. This is due to Q learning agent not being able to appropriately explore all states. In order to allow the agent to derive a markov memory 2 policy we must increase the number of training episoides. Conducting the equivilant experiment for the IBM data yeilds the following results. When treated as IID, the following Q table is derived.
Distribution into Source 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
None 43.8113 43.8479 43.6593 44.0324 44.544 44.5033 43.8169 44.1369 43.9382 54.4591 44.3226

And applying such policy to the testing data yeilds.

Moving to markov memory 1, we obtain the following Q table.
Distribution into Source 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Prev "Down" 10.2676 10.2352 11.1674 10.8175 10.6088 11.0861 10.3039 22.9233 9.90889 10.1815 9.52493
Prev "Nuetral" 23.2557 23.2025 23.4107 23.7157 23.1266 23.4601 31.6622 24.0064 22.8477 23.2966 23.6219
Prev "Positive" 7.81495 7.97147 8.75302 8.76693 8.49904 8.13567 8.53647 8.06919 8.06033 8.29538 20.0326

Which yeilds the following testing results.

The difference between the IID test and markov 1 test is that the markov 1 policy seems to be less "risky", it looses less money at the begining of the testing period, however, gains less towards the end. The two policies appear to be very simaler. Moving forward to markov order 2, we obtain the following q table.
Distribution into Source 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Prev Seq: "Down", "Down" 2.01617 2.08723 1.95979 6.864 1.78769 1.70395 1.87764 2.22568 1.64296 1.81577 2.00709
Prev Seq: "Down", "Nuetral" 5.53366 5.52372 5.12495 5.09426 12.9182 5.1679 5.40856 4.4439 4.89715 4.97982 5.47335
Prev Seq: "Down", "Positive" 1.18736 1.34069 1.44999 1.51097 1.20899 1.38332 4.57586 1.24534 1.38879 1.46921 1.37788
Prev Seq: "Nuetral", "Down" 3.6141 3.97473 3.66598 3.66956 3.88415 9.45307 3.82271 3.77226 3.98146 3.84407 3.50747
Prev Seq: "Nuetral", "Nuetral" 16.3345 16.0558 15.8404 16.3348 15.6925 16.4471 16.2041 16.3338 16.1826 16.1871 24.4097
Prev Seq: "Nuetral", "Postive" 3.13525 2.72395 2.83029 3.04262 8.08926 2.84946 3.14385 3.14187 2.75471 2.78054 2.84228
Prev Seq: "Positive", "Down" 1.49701 5.84822 1.38522 1.58129 1.71697 1.74671 1.58833 1.83914 1.59212 1.92212 1.68669
Prev Seq: "Positive", "Nuetral" 4.31702 3.85078 10.0363 3.83889 3.75574 3.99425 3.21814 4.12956 3.47642 3.56835 3.76362
Prev Seq: "Positive", "Positive" 1.30869 1.09477 1.28534 1.12935 1.18675 0.935871 0.9024 0.860647 1.13079 1.10643 4.14827

Which yeilds the following performance over out testing period.