Chapter 1 - Last Example - Agent w GridworldEnv and DiscretePolicy

[ghost]

I typed in the example as given in the book at the end of Chapter 1. I keep getting the 'below' error. Any thoughts?

Full error printout:

ValueError                                Traceback (most recent call last)
<ipython-input-66-99c415f36d0c> in <module>
      2 env = GridworldEnv()
      3 agent = Agent(env.action_space.n, env.observation_space.shape)
----> 4 steps, reward, done, info = evaluate(agent, env)
      5 print(f"steps:{steps} reward:{reward} done:{done} info:{info}")
      6 env.close()

<ipython-input-65-0c150c4aeab3> in evaluate(agent, env, render)
      5     while not done:
      6         print('in while')
----> 7         action = agent.get_action(obs)
      8         print('evaluate action:', action)
      9         obs, reward, done, info = env.step(action)

<ipython-input-44-40d86732302a> in get_action(self, obs)
      5 
      6     def get_action(self, obs):
----> 7         action_logits = self.brain.process(obs)
      8         action = self.policy.get_action(np.squeeze(action_logits, 0))
      9         return action

<ipython-input-52-08a7a86c2d76> in process(self, observations)
     15 
     16     def process(self, observations):
---> 17         action_logits = self.predict_on_batch(observations)
     18         return action_logits

~\anaconda3\envs\tfrl-cookbook\lib\site-packages\tensorflow_core\python\keras\engine\training.py in predict_on_batch(self, x)
   1238     self._check_call_args('predict_on_batch')
   1239     if self._experimental_run_tf_function:
-> 1240       return training_v2_utils.predict_on_batch(self, x, standalone=True)
   1241 
   1242     if (self._distribution_strategy and

~\anaconda3\envs\tfrl-cookbook\lib\site-packages\tensorflow_core\python\keras\engine\training_v2_utils.py in predict_on_batch(model, x, standalone)
    537   # Validate and standardize user data.
    538   inputs, _, _ = model._standardize_user_data(
--> 539       x, extract_tensors_from_dataset=True)
    540 
    541   # If `model._distribution_strategy` is True, then we are in a replica context

~\anaconda3\envs\tfrl-cookbook\lib\site-packages\tensorflow_core\python\keras\engine\training.py in _standardize_user_data(self, x, y, sample_weight, class_weight, batch_size, check_steps, steps_name, steps, validation_split, shuffle, extract_tensors_from_dataset)
   2344     # First, we build the model on the fly if necessary.
   2345     if not self.inputs:
-> 2346       all_inputs, y_input, dict_inputs = self._build_model_with_inputs(x, y)
   2347       is_build_called = True
   2348     else:

~\anaconda3\envs\tfrl-cookbook\lib\site-packages\tensorflow_core\python\keras\engine\training.py in _build_model_with_inputs(self, inputs, targets)
   2570     else:
   2571       cast_inputs = inputs
-> 2572     self._set_inputs(cast_inputs)
   2573     return processed_inputs, targets, is_dict_inputs
   2574 

~\anaconda3\envs\tfrl-cookbook\lib\site-packages\tensorflow_core\python\keras\engine\training.py in _set_inputs(self, inputs, outputs, training)
   2657           kwargs['training'] = training
   2658       try:
-> 2659         outputs = self(inputs, **kwargs)
   2660       except NotImplementedError:
   2661         # This Model or a submodel is dynamic and hasn't overridden

~\anaconda3\envs\tfrl-cookbook\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py in __call__(self, inputs, *args, **kwargs)
    771                     not base_layer_utils.is_in_eager_or_tf_function()):
    772                   with auto_control_deps.AutomaticControlDependencies() as acd:
--> 773                     outputs = call_fn(cast_inputs, *args, **kwargs)
    774                     # Wrap Tensors in `outputs` in `tf.identity` to avoid
    775                     # circular dependencies.

~\anaconda3\envs\tfrl-cookbook\lib\site-packages\tensorflow_core\python\autograph\impl\api.py in wrapper(*args, **kwargs)
    235       except Exception as e:  # pylint:disable=broad-except
    236         if hasattr(e, 'ag_error_metadata'):
--> 237           raise e.ag_error_metadata.to_exception(e)
    238         else:
    239           raise

ValueError: in converted code:

    <ipython-input-52-08a7a86c2d76>:14 call  *
        return self.logits(self.dense1(x))
    C:\Users\billm\anaconda3\envs\tfrl-cookbook\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py:748 __call__
        self._maybe_build(inputs)
    C:\Users\billm\anaconda3\envs\tfrl-cookbook\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py:2116 _maybe_build
        self.build(input_shapes)
    C:\Users\billm\anaconda3\envs\tfrl-cookbook\lib\site-packages\tensorflow_core\python\keras\layers\core.py:1101 build
        raise ValueError('The last dimension of the inputs to `Dense` '

    ValueError: The last dimension of the inputs to `Dense` should be defined. Found `None`.

[praveen-palanisamy]

Hi @BeavFactory ,
[I formatted your post so that it is easier to read]

Thank you for bringing this up for discussion.
I haven't been able to reproduce your issue but this did help me realize that the full, executable recipe was not under the code repository for Chapter 1. I have pushed an update to include the full, executable recipe that you can just run. You can find it here: https://github.com/PacktPublishing/Tensorflow-2-Reinforcement-Learning-Cookbook/blob/master/Chapter01/06_neural_evolutionary_agent.py

It looks like the error you are getting is due to the inputs to the Agent's policy network on your end being None or invalid at one of the Dense layers which is unusual.

Could you please try running the last recipe (now in a single python file that you can execute) using this script and report back if you still face the issue?

[bayareacoder]

hi @praveen-palanisamy , when I run the code of Cookbook/blob/master/Chapter01/06_neural_evolutionary_agent.py (in a Jupyter Notebook) I see what is the in the attached image. Seems not to be converging as the threshold goes to 0 after the 2nd iteration. I also don't understand why 'fire' is used here. The code seems somewhat different than what is in the book.

[praveen-palanisamy]

Hi @bayareacoder ,
It's hard to tell if it converged or not in just 10 episodes. The evolutionary methods usually take lot more samples to learn and converge. Could you try changing the default num_epochs value from 10 to higher number (e.g:1000) and train?

Fire is used to make it easier to run the train function (and other functions) from the command line by supplying the arguments as just command line parameters. I see that you are running it in Jupyter notebook cells. You can also simply launch the training (without using Fire) by executing this in a cell: train() or train(num_epochs=1000).

[bayareacoder]

Ran below for first 100 episodes out of 1000. It doesn't seem right to me.

train(num_epochs=1000)

  0%|          | 1/1000 [00:19<5:29:27, 19.79s/it]
Episode#:1 elite-reward-threshold:-9.40 reward:-9.44 
  0%|          | 2/1000 [00:39<5:28:03, 19.72s/it]
Episode#:2 elite-reward-threshold:-7.10 reward:-7.38 
  0%|          | 3/1000 [00:58<5:23:48, 19.49s/it]
Episode#:3 elite-reward-threshold:-9.50 reward:-9.50 
  0%|          | 4/1000 [01:17<5:22:30, 19.43s/it]
Episode#:4 elite-reward-threshold:-9.50 reward:-9.50 
  0%|          | 5/1000 [01:36<5:20:11, 19.31s/it]
Episode#:5 elite-reward-threshold:-9.50 reward:-9.50 
  1%|          | 6/1000 [01:55<5:19:29, 19.29s/it]
Episode#:6 elite-reward-threshold:-9.50 reward:-9.50 
  1%|          | 7/1000 [02:14<5:16:21, 19.11s/it]
Episode#:7 elite-reward-threshold:-9.50 reward:-9.50 
  1%|          | 8/1000 [02:33<5:14:58, 19.05s/it]
Episode#:8 elite-reward-threshold:-9.50 reward:-9.50 
  1%|          | 9/1000 [02:52<5:15:29, 19.10s/it]
Episode#:9 elite-reward-threshold:-9.50 reward:-9.50 
  1%|          | 10/1000 [03:12<5:20:00, 19.39s/it]
Episode#:10 elite-reward-threshold:-9.50 reward:-9.50 
  1%|          | 11/1000 [03:31<5:18:19, 19.31s/it]
Episode#:11 elite-reward-threshold:-9.50 reward:-9.50 
  1%|          | 12/1000 [03:51<5:21:08, 19.50s/it]
Episode#:12 elite-reward-threshold:-9.50 reward:-9.50 
  1%|▏         | 13/1000 [04:13<5:29:06, 20.01s/it]
Episode#:13 elite-reward-threshold:-9.50 reward:-9.50 
  1%|▏         | 14/1000 [04:31<5:20:41, 19.52s/it]
Episode#:14 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▏         | 15/1000 [04:50<5:18:54, 19.43s/it]
Episode#:15 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▏         | 16/1000 [05:09<5:17:38, 19.37s/it]
Episode#:16 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▏         | 17/1000 [05:28<5:15:45, 19.27s/it]
Episode#:17 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▏         | 18/1000 [05:47<5:12:27, 19.09s/it]
Episode#:18 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▏         | 19/1000 [06:06<5:09:41, 18.94s/it]
Episode#:19 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▏         | 20/1000 [06:24<5:08:24, 18.88s/it]
Episode#:20 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▏         | 21/1000 [06:43<5:05:51, 18.74s/it]
Episode#:21 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▏         | 22/1000 [07:01<5:04:50, 18.70s/it]
Episode#:22 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▏         | 23/1000 [07:20<5:03:01, 18.61s/it]
Episode#:23 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▏         | 24/1000 [07:38<4:59:35, 18.42s/it]
Episode#:24 elite-reward-threshold:-9.50 reward:-9.50 
  2%|▎         | 25/1000 [07:57<5:01:55, 18.58s/it]
Episode#:25 elite-reward-threshold:-9.50 reward:-9.50 
  3%|▎         | 26/1000 [08:15<5:01:02, 18.54s/it]
Episode#:26 elite-reward-threshold:-9.50 reward:-9.50 
  3%|▎         | 27/1000 [08:34<5:00:43, 18.54s/it]
Episode#:27 elite-reward-threshold:-9.50 reward:-9.50 
  3%|▎         | 28/1000 [08:52<4:59:34, 18.49s/it]
Episode#:28 elite-reward-threshold:-9.50 reward:-9.50 
  3%|▎         | 29/1000 [09:10<4:57:44, 18.40s/it]
Episode#:29 elite-reward-threshold:-9.50 reward:-9.50 
  3%|▎         | 30/1000 [09:29<4:58:03, 18.44s/it]
Episode#:30 elite-reward-threshold:-9.50 reward:-9.50 
  3%|▎         | 31/1000 [09:48<5:01:35, 18.67s/it]
Episode#:31 elite-reward-threshold:-9.50 reward:-9.50 
  3%|▎         | 32/1000 [10:07<5:03:13, 18.79s/it]
Episode#:32 elite-reward-threshold:-9.50 reward:-9.50 
  3%|▎         | 33/1000 [10:26<5:02:38, 18.78s/it]
Episode#:33 elite-reward-threshold:-9.50 reward:-9.50 
  3%|▎         | 34/1000 [10:45<5:03:43, 18.87s/it]
Episode#:34 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▎         | 35/1000 [11:04<5:03:46, 18.89s/it]
Episode#:35 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▎         | 36/1000 [11:23<5:03:44, 18.91s/it]
Episode#:36 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▎         | 37/1000 [11:43<5:07:41, 19.17s/it]
Episode#:37 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▍         | 38/1000 [12:02<5:08:50, 19.26s/it]
Episode#:38 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▍         | 39/1000 [12:21<5:09:00, 19.29s/it]
Episode#:39 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▍         | 40/1000 [12:40<5:07:19, 19.21s/it]
Episode#:40 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▍         | 41/1000 [13:01<5:12:53, 19.58s/it]
Episode#:41 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▍         | 42/1000 [13:20<5:08:28, 19.32s/it]
Episode#:42 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▍         | 43/1000 [13:38<5:05:49, 19.17s/it]
Episode#:43 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▍         | 44/1000 [13:57<5:03:12, 19.03s/it]
Episode#:44 elite-reward-threshold:-9.50 reward:-9.50 
  4%|▍         | 45/1000 [14:16<5:00:36, 18.89s/it]
Episode#:45 elite-reward-threshold:-9.50 reward:-9.50 
  5%|▍         | 46/1000 [14:35<4:59:57, 18.87s/it]
Episode#:46 elite-reward-threshold:-9.50 reward:-9.50 
  5%|▍         | 47/1000 [14:54<5:01:50, 19.00s/it]
Episode#:47 elite-reward-threshold:-9.50 reward:-9.50 
  5%|▍         | 48/1000 [15:13<5:02:47, 19.08s/it]
Episode#:48 elite-reward-threshold:-9.50 reward:-9.50 
  5%|▍         | 49/1000 [15:32<5:02:36, 19.09s/it]
Episode#:49 elite-reward-threshold:-9.50 reward:-9.50 
  5%|▌         | 50/1000 [15:51<4:58:33, 18.86s/it]
Episode#:50 elite-reward-threshold:-9.50 reward:-9.50 
  5%|▌         | 51/1000 [16:09<4:56:46, 18.76s/it]
Episode#:51 elite-reward-threshold:-9.50 reward:-9.50 
  5%|▌         | 52/1000 [16:28<4:55:16, 18.69s/it]
Episode#:52 elite-reward-threshold:-9.50 reward:-9.50 
  5%|▌         | 53/1000 [16:46<4:54:34, 18.66s/it]
Episode#:53 elite-reward-threshold:-9.50 reward:-9.50 
  5%|▌         | 54/1000 [17:04<4:52:32, 18.55s/it]
Episode#:54 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▌         | 55/1000 [17:23<4:51:30, 18.51s/it]
Episode#:55 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▌         | 56/1000 [17:42<4:54:09, 18.70s/it]
Episode#:56 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▌         | 57/1000 [18:01<4:54:36, 18.74s/it]
Episode#:57 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▌         | 58/1000 [18:19<4:53:02, 18.66s/it]
Episode#:58 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▌         | 59/1000 [18:38<4:50:59, 18.55s/it]
Episode#:59 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▌         | 60/1000 [18:56<4:50:48, 18.56s/it]
Episode#:60 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▌         | 61/1000 [19:14<4:48:42, 18.45s/it]
Episode#:61 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▌         | 62/1000 [19:33<4:46:53, 18.35s/it]
Episode#:62 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▋         | 63/1000 [19:50<4:44:38, 18.23s/it]
Episode#:63 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▋         | 64/1000 [20:08<4:42:43, 18.12s/it]
Episode#:64 elite-reward-threshold:-9.50 reward:-9.50 
  6%|▋         | 65/1000 [20:26<4:42:17, 18.12s/it]
Episode#:65 elite-reward-threshold:-9.50 reward:-9.50 
  7%|▋         | 66/1000 [20:44<4:40:15, 18.00s/it]
Episode#:66 elite-reward-threshold:-9.50 reward:-9.50 
  7%|▋         | 67/1000 [21:02<4:39:33, 17.98s/it]
Episode#:67 elite-reward-threshold:-9.50 reward:-9.50 
  7%|▋         | 68/1000 [21:20<4:38:55, 17.96s/it]
Episode#:68 elite-reward-threshold:-9.50 reward:-9.50 
  7%|▋         | 69/1000 [21:38<4:38:50, 17.97s/it]
Episode#:69 elite-reward-threshold:-9.50 reward:-9.50 
  7%|▋         | 70/1000 [21:56<4:38:12, 17.95s/it]
Episode#:70 elite-reward-threshold:-9.50 reward:-9.50 
  7%|▋         | 71/1000 [22:14<4:40:35, 18.12s/it]
Episode#:71 elite-reward-threshold:-9.50 reward:-9.50 
  7%|▋         | 72/1000 [22:33<4:42:40, 18.28s/it]
Episode#:72 elite-reward-threshold:-9.50 reward:-9.50 
  7%|▋         | 73/1000 [22:51<4:42:42, 18.30s/it]
Episode#:73 elite-reward-threshold:-9.50 reward:-9.50 
  7%|▋         | 74/1000 [23:10<4:43:53, 18.39s/it]
Episode#:74 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 75/1000 [23:28<4:43:43, 18.40s/it]
Episode#:75 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 76/1000 [23:47<4:43:37, 18.42s/it]
Episode#:76 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 77/1000 [24:06<4:44:06, 18.47s/it]
Episode#:77 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 78/1000 [24:24<4:44:51, 18.54s/it]
Episode#:78 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 79/1000 [24:43<4:45:10, 18.58s/it]
Episode#:79 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 80/1000 [25:02<4:45:54, 18.65s/it]
Episode#:80 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 81/1000 [25:20<4:45:38, 18.65s/it]
Episode#:81 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 82/1000 [25:39<4:43:56, 18.56s/it]
Episode#:82 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 83/1000 [25:57<4:41:37, 18.43s/it]
Episode#:83 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 84/1000 [26:15<4:40:41, 18.39s/it]
Episode#:84 elite-reward-threshold:-9.50 reward:-9.50 
  8%|▊         | 85/1000 [26:33<4:40:01, 18.36s/it]
Episode#:85 elite-reward-threshold:-9.50 reward:-9.50 
  9%|▊         | 86/1000 [26:52<4:41:31, 18.48s/it]
Episode#:86 elite-reward-threshold:-9.50 reward:-9.50 
  9%|▊         | 87/1000 [27:11<4:41:11, 18.48s/it]
Episode#:87 elite-reward-threshold:-9.50 reward:-9.50 
  9%|▉         | 88/1000 [27:29<4:40:09, 18.43s/it]
Episode#:88 elite-reward-threshold:-9.50 reward:-9.50 
  9%|▉         | 89/1000 [27:47<4:37:55, 18.30s/it]
Episode#:89 elite-reward-threshold:-9.50 reward:-9.50 
  9%|▉         | 90/1000 [28:05<4:37:01, 18.27s/it]
Episode#:90 elite-reward-threshold:-9.50 reward:-9.50 
  9%|▉         | 91/1000 [28:23<4:35:35, 18.19s/it]
Episode#:91 elite-reward-threshold:-9.50 reward:-9.50 
  9%|▉         | 92/1000 [28:41<4:35:11, 18.18s/it]
Episode#:92 elite-reward-threshold:-9.50 reward:-9.50 
  9%|▉         | 93/1000 [29:00<4:35:20, 18.21s/it]
Episode#:93 elite-reward-threshold:-9.50 reward:-9.50 
  9%|▉         | 94/1000 [29:18<4:34:38, 18.19s/it]
Episode#:94 elite-reward-threshold:-9.50 reward:-9.50 
 10%|▉         | 95/1000 [29:36<4:34:18, 18.19s/it]
Episode#:95 elite-reward-threshold:-9.50 reward:-9.50 
 10%|▉         | 96/1000 [29:55<4:36:15, 18.34s/it]
Episode#:96 elite-reward-threshold:-9.50 reward:-9.50 
 10%|▉         | 97/1000 [30:13<4:35:34, 18.31s/it]
Episode#:97 elite-reward-threshold:-9.50 reward:-9.50 
 10%|▉         | 98/1000 [30:31<4:35:53, 18.35s/it]
Episode#:98 elite-reward-threshold:-9.50 reward:-9.50 
 10%|▉         | 99/1000 [30:50<4:36:05, 18.39s/it]
Episode#:99 elite-reward-threshold:-9.50 reward:-9.50 
 10%|█         | 100/1000 [31:08<4:35:28, 18.36s/it]
Episode#:100 elite-reward-threshold:-9.50 reward:-9.50 
 10%|█         | 100/1000 [31:22<4:42:20, 18.82s/it]