RuntimeError: The Session graph is empty. Add operations to the graph before calling run()

[fraseralex96]

Hi Team,

I get the following error when trying to initialise the CEM explainer:

`---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Input In [52], in <cell line: 4>()
2 shape = (1,) + X_train.shape[1:]
3 mode = 'PN'
----> 4 cem = CEM(model, mode, shape, kappa=0., beta=.1,
5 feature_range=(X_train.min(), X_train.max()),
6 gamma=100, max_iterations=1000,
7 c_init=1., c_steps=10, learning_rate_init=1e-2,
8 clip=(-1000.,1000.), no_info_val=-1.)

File ~/.local/lib/python3.9/site-packages/alibi/explainers/cem.py:107, in CEM.init(self, predict, mode, shape, kappa, beta, feature_range, gamma, ae_model, learning_rate_init, max_iterations, c_init, c_steps, eps, clip, update_num_grad, no_info_val, write_dir, sess)
105 if is_model: # Keras or TF model
106 self.model = True
--> 107 classes = self.sess.run(self.predict(tf.convert_to_tensor(np.zeros(shape), dtype=tf.float32))).shape[1]
108 else:
109 self.model = False

File ~/.local/lib/python3.9/site-packages/tensorflow/python/client/session.py:969, in BaseSession.run(self, fetches, feed_dict, options, run_metadata)
966 run_metadata_ptr = tf_session.TF_NewBuffer() if run_metadata else None
968 try:
--> 969 result = self._run(None, fetches, feed_dict, options_ptr,
970 run_metadata_ptr)
971 if run_metadata:
972 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

File ~/.local/lib/python3.9/site-packages/tensorflow/python/client/session.py:1119, in BaseSession._run(self, handle, fetches, feed_dict, options, run_metadata)
1117 raise RuntimeError('Attempted to use a closed Session.')
1118 if self.graph.version == 0:
-> 1119 raise RuntimeError('The Session graph is empty. Add operations to the '
1120 'graph before calling run().')
1122 # Create request.
1123 feed_dict_tensor = {}

RuntimeError: The Session graph is empty. Add operations to the graph before calling run().`

My model architecture is as follow:

`## CNN using functional API
def DeepIC50_tester(X_train, learning_rate, momentum, seed):

# set layer weights initialiser
initializer = keras.initializers.GlorotUniform(seed=seed)

# drug-cell line data input
x_input = layers.Input(shape=(X_train.shape[1],1))

# 1st convolution layer
x = layers.Conv1D(filters=16, kernel_size=11, kernel_initializer=initializer, activation='relu')(x_input) 
x = layers.BatchNormalization()(x)
#x = layers.MaxPooling1D()(x)
#x = layers.Dropout(0.1)(x)

# 2nd convolution layer
x = layers.Conv1D(filters=16, kernel_size=11, kernel_initializer=initializer, activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.MaxPooling1D()(x)
#x = layers.Dropout(0.1)(x)

# 3rd convolution layer
x = layers.Conv1D(filters=32, kernel_size=11, kernel_initializer=initializer, activation='relu')(x)
x = layers.BatchNormalization()(x)
#x = layers.MaxPooling1D()(x)
#x = layers.Dropout(0.1)(x)

# 4th convolution layer
x = layers.Conv1D(filters=32, kernel_size=11, kernel_initializer=initializer, activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.MaxPooling1D()(x)
#x = layers.Dropout(0.1)(x)

# 5th convolution layer
x = layers.Conv1D(filters=64, kernel_size=11, kernel_initializer=initializer, activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.MaxPooling1D()(x)
#x = layers.Dropout(0.1)(x)

# 6th convolution layer
x = layers.Conv1D(filters=64, kernel_size=11, kernel_initializer=initializer, activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.MaxPooling1D()(x)
#x = layers.Dropout(0.1)(x)

x = layers.Flatten()(x)

# 5 fully connected layers with batch norm and dropout
x = layers.Dense(128, kernel_initializer=initializer, activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.Dropout(0.1)(x)
x = layers.Dense(256, kernel_initializer=initializer, activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.Dropout(0.1)(x)
x = layers.Dense(512, kernel_initializer=initializer, activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.Dropout(0.1)(x)
x = layers.Dense(256, kernel_initializer=initializer, activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.Dropout(0.1)(x)
x = layers.Dense(128, kernel_initializer=initializer, activation='relu')(x)
x = layers.BatchNormalization()(x)
x = layers.Dropout(0.1)(x)


output = layers.Dense(3, activation = 'softmax',activity_regularizer=keras.regularizers.l2())(x) # actiation layer is 3 neuron for each class

model = keras.Model(x_input, output)

model.compile( loss='sparse_categorical_crossentropy', optimizer=keras.optimizers.Adam(), metrics=['accuracy'])
return model`

I run this code when initialising:

`# initialize CEM explainer

shape = (1,) + X_train.shape[1:]

mode = 'PN'

cem = CEM(model, mode, shape, kappa=0., beta=.1,
feature_range=(X_train.min(), X_train.max()),
gamma=100, max_iterations=1000,
c_init=1., c_steps=10, learning_rate_init=1e-2,
clip=(-1000.,1000.), no_info_val=-1.)`

Please assist! It would be much appreciated!

Alex

[RobertSamoilescu]

Hi @fraseralex96,

Try to add the following lines as in the example from here:

import tensorflow as tf
tf.get_logger().setLevel(40) # suppress deprecation messages
tf.compat.v1.disable_v2_behavior() # disable TF2 behaviour as alibi code still relies on TF1 constructs

[fraseralex96]

Hi Robert,

Thank you for your response.

Unfortunately I now receive the following error message when I build my NN model because of the above code:

WARNING:absl:There is a known slowdown when using v2.11+ Keras optimizers on M1/M2 Macs. Falling back to the legacy Keras optimizer, i.e., tf.keras.optimizers.legacy.Adam`.

NotImplementedError Traceback (most recent call last)
Input In [12], in <cell line: 7>()
4 momentum = 0.5
5 initializer_seed = 42
----> 7 model = DeepIC50(X_train, learning_rate, momentum, initializer_seed)

File ~/ML_notebook/ML_models.py:354, in DeepIC50(X_train, learning_rate, momentum, seed)
350 output = layers.Dense(3, activation = 'softmax',activity_regularizer=keras.regularizers.l2())(x) # actiation layer is 3 neuron for each class
352 model = keras.Model(x_input, output)
--> 354 model.compile( loss='sparse_categorical_crossentropy', optimizer=keras.optimizers.Adam(), metrics=['accuracy'])
355 return model

File ~/.local/lib/python3.9/site-packages/tensorflow/python/trackable/base.py:204, in no_automatic_dependency_tracking.._method_wrapper(self, *args, **kwargs)
202 self._self_setattr_tracking = False # pylint: disable=protected-access
203 try:
--> 204 result = method(self, *args, **kwargs)
205 finally:
206 self._self_setattr_tracking = previous_value # pylint: disable=protected-access

File ~/.local/lib/python3.9/site-packages/keras/src/engine/training_v1.py:321, in Model.compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, weighted_metrics, target_tensors, distribute, **kwargs)
314 if self.run_eagerly:
315 raise ValueError(
316 "Session keyword arguments are not supported "
317 "when run_eagerly=True. You passed the following "
318 "Session arguments: %s" % (self._function_kwargs,)
319 )
--> 321 self._set_optimizer(optimizer)
322 is_any_keras_optimizer_v1 = any(
323 (
324 isinstance(opt, optimizer_v1.Optimizer)
(...)
327 for opt in tf.nest.flatten(self.optimizer)
328 )
330 if (
331 is_any_keras_optimizer_v1
332 and tf.compat.v1.executing_eagerly_outside_functions()
333 ):

File ~/.local/lib/python3.9/site-packages/keras/src/engine/training_v1.py:1473, in Model._set_optimizer(self, optimizer)
1471 self.optimizer = [optimizers.get(opt) for opt in optimizer]
1472 else:
-> 1473 self.optimizer = optimizers.get(optimizer)
1475 if self._dtype_policy.name == "mixed_float16" and not isinstance(
1476 self.optimizer, loss_scale_optimizer.LossScaleOptimizer
1477 ):
1478 if isinstance(self.optimizer, list):

File ~/.local/lib/python3.9/site-packages/keras/src/optimizers/init.py:298, in get(identifier, **kwargs)
291 optimizer_name = identifier.class.name
292 logging.warning(
293 "There is a known slowdown when using v2.11+ Keras optimizers "
294 "on M1/M2 Macs. Falling back to the "
295 "legacy Keras optimizer, i.e., "
296 f"tf.keras.optimizers.legacy.{optimizer_name}."
297 )
--> 298 return convert_to_legacy_optimizer(identifier)
300 # Wrap legacy TF optimizer instances
301 elif isinstance(identifier, tf.compat.v1.train.Optimizer):

File ~/.local/lib/python3.9/site-packages/keras/src/optimizers/init.py:222, in convert_to_legacy_optimizer(optimizer)
216 raise ValueError(
217 "convert_to_legacy_optimizer should only be called "
218 "on instances of tf.keras.optimizers.Optimizer, but "
219 f"received {optimizer} of type {type(optimizer)}."
220 )
221 optimizer_name = optimizer.class.name.lower()
--> 222 config = optimizer.get_config()
223 # Remove fields that only exist in experimental optimizer.
224 keys_to_remove = [
225 "weight_decay",
226 "use_ema",
(...)
230 "is_legacy_optimizer",
231 ]

File ~/.local/lib/python3.9/site-packages/keras/src/optimizers/adam.py:211, in Adam.get_config(self)
206 def get_config(self):
207 config = super().get_config()
209 config.update(
210 {
--> 211 "learning_rate": self._serialize_hyperparameter(
212 self._learning_rate
213 ),
214 "beta_1": self.beta_1,
215 "beta_2": self.beta_2,
216 "epsilon": self.epsilon,
217 "amsgrad": self.amsgrad,
218 }
219 )
220 return config

File ~/.local/lib/python3.9/site-packages/keras/src/optimizers/optimizer.py:736, in _BaseOptimizer._serialize_hyperparameter(self, hyperparameter)
734 return learning_rate_schedule.serialize(hyperparameter)
735 if isinstance(hyperparameter, tf.Variable):
--> 736 return hyperparameter.numpy()
737 if callable(hyperparameter):
738 return hyperparameter()

File ~/.local/lib/python3.9/site-packages/tensorflow/python/ops/resource_variable_ops.py:689, in BaseResourceVariable.numpy(self)
687 if context.executing_eagerly():
688 return self.read_value().numpy()
--> 689 raise NotImplementedError(
690 "numpy() is only available when eager execution is enabled.")

NotImplementedError: numpy() is only available when eager execution is enabled.`

[RobertSamoilescu]

Try to use tensorflow v2.10:

pip install --upgrade tensorflow==2.10

[fraseralex96]

Hi Robert,

I get the following error message now when I try to run CEM (and also when I individually run the model I am trying to explain):

`2023-08-03 11:04:01.763603: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

ValueError Traceback (most recent call last)
Input In [22], in <cell line: 4>()
2 shape = X_train.shape
3 mode = 'PN'
----> 4 cem = CEM(model, mode, shape, kappa=0., beta=.1,
5 feature_range=(X_train.min(), X_train.max()),
6 gamma=100, max_iterations=1000,
7 c_init=1., c_steps=10, learning_rate_init=1e-2,
8 clip=(-1000.,1000.), no_info_val=-1.)

File ~/.local/lib/python3.9/site-packages/alibi/explainers/cem.py:235, in CEM.init(self, predict, mode, shape, kappa, beta, feature_range, gamma, ae_model, learning_rate_init, max_iterations, c_init, c_steps, eps, clip, update_num_grad, no_info_val, write_dir, sess)
233 self.pred_proba_s = self.predict(self.delta_s)
234 elif self.mode == "PN":
--> 235 self.pred_proba = self.predict(self.adv)
236 self.pred_proba_s = self.predict(self.adv_s)
238 # probability of target label prediction

File ~/.local/lib/python3.9/site-packages/keras/utils/traceback_utils.py:70, in filter_traceback..error_handler(*args, **kwargs)
67 filtered_tb = _process_traceback_frames(e.traceback)
68 # To get the full stack trace, call:
69 # tf.debugging.disable_traceback_filtering()
---> 70 raise e.with_traceback(filtered_tb) from None
71 finally:
72 del filtered_tb

File ~/.local/lib/python3.9/site-packages/keras/engine/input_spec.py:250, in assert_input_compatibility(input_spec, inputs, layer_name)
248 ndim = x.shape.rank
249 if ndim is not None and ndim < spec.min_ndim:
--> 250 raise ValueError(
251 f'Input {input_index} of layer "{layer_name}" '
252 "is incompatible with the layer: "
253 f"expected min_ndim={spec.min_ndim}, "
254 f"found ndim={ndim}. "
255 f"Full shape received: {tuple(shape)}"
256 )
257 # Check dtype.
258 if spec.dtype is not None:

ValueError: Input 0 of layer "conv1d_18" is incompatible with the layer: expected min_ndim=3, found ndim=2. Full shape received: (Dimension(10427), Dimension(1378))`

I tested it and it is directly because of this code:

import tensorflow as tf tf.get_logger().setLevel(40) # suppress deprecation messages tf.compat.v1.disable_v2_behavior() # disable TF2 behaviour as alibi code still relies on TF1 constructs

Do you have any suggestion?

Thanks!
Alex

[RobertSamoilescu]

My guess is that you input shape to the Input layer is not properly set. I see you are initialising it as:

x_input = layers.Input(shape=(X_train.shape[1],1))

I think the correct initialisation is:

x_input = layers.Input(shape=X_train.shape[1:])

You should be able to test if everything is ok by performing a forward pass through your model:

model(X_train[:1])

See the documentation here. Also, since you are using conv1d, your training data must have the following shape (B, L, D), where B is the batch size, L is the sequence length, and D is the channels size.

[RobertSamoilescu]

Related to a previous error you had when initialising the optimizer, you can probably try using the legacy optimizers with the newe vesions of tensorflow (see here). There has been a change to the optimizers from tensorflow 2.11 (see release notes here)