Initializing Variables, Variable Scope And Import_graph_def In Tensorflow

I have a number of related questions about tensorflow behavior when attempting to do graph surgery using import_graph_def. 2 different graph surgeries In the image above, I represe

Solution 1:

To begin with, why is the variable reported as foo/z1/foo/c? Why does the scope foo appear twice?

After you've called tf.import_graph_def(...), the graph got duplicated. The first graph is defined in foo score. The second subgraph has been imported under the scope foo/z1 (because name='z1', plus foo is preserved from the scope above). So the graph g1 now contains the following tensors:

foo/x
foo/y
foo/c
...
foo/z1/foo/x
foo/z1/foo/y
foo/z1/foo/c
...

The first foo/c is initialized, but the second foo/z1/foo/c is not (see below).

Why is nothing reported when I print the uninitialized variables? Why is only foo/c reported when I print the GLOBAL_VARIABLES collection under the scope foo?

Since report_uninitialized_variables() scans LOCAL_VARIABLES and GLOBAL_VARIABLES by default, this is basically the same question.

And it probably is a bug: GLOBAL_VARIABLES collection isn't updated after tf.import_graph_def call. I say probably because GLOBAL_VARIABLES was designed as a mere convenience collection. Tensorflow tries to keep it up do date, but probably doesn't guarantee it always has all variables. The fact that tf.add_to_collection exists publicly supports this idea -- one can add any value to any collection if they want it. Bottom line: this behavior may or may not change in future versions, but as of 1.5 the client is responsible to update the global variables after graph import.

In particular, the last print informs that there are no unintialized variables. However, uncommenting the last line, yields the error

To fix this error, you simply need to run the initializer for the z1 subgraph. Like this:

# note that it's defined before `g1.as_graph_def()` to be a part of graph def
init_op = tf.global_variables_initializer()

g1_def = g1.as_graph_def()
z1, = tf.import_graph_def(g1_def, input_map={'foo/x:0': y}, return_elements=["foo/z:0"],
                          name='z1')

# find the init op
z1_init_op = tf.get_default_graph().get_operation_by_name('foo/z1/foo/init')

...

sess.run(z1_init_op)

And voila! You have the duplicated graphs, just like you wanted to.

Solution 2:

I faced a similar issue but simply running the init operation didn't work.

I fixed it by manually running all "Assign" ops of the global variables of the imported graph.

In my scenario I want to run an encoding op 'z' with input 'patch:0' using two different input tensors.

with tf.Session(graph=tf.get_default_graph()).as_default() as sess:

        g = tf.Graph()       
        saved_model = predictor.from_saved_model(args.export_dir, graph=g)
        variables = g.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)]

        fetch_ops = ['z:0','init']
        fetch_ops.extend([v.name.strip(":0") + "/Assign"for v in variables)

        image_graph = tf.graph_util.import_graph_def(
            g.as_graph_def(),
            input_map={'patch:0': image},
            return_elements=fetch_ops,
            name='image')

        warped_graph = tf.graph_util.import_graph_def(
            g.as_graph_def(),
            input_map={'patch:0': warped_image},
            return_elements=fetch_ops,
            name='warp')

        loss = tf.reduce_sum(tf.math.squared_difference(image_graph[0], warped_graph[0]))

        optimizer =  tf.train.GradientDescentOptimizer(learning_rate=0.0001)
        compute_gradients = optimizer.compute_gradients(
            loss,
            var_list=[dest_control_point_locations])

        apply_gradients = optimizer.apply_gradients(compute_gradients, global_step=step)

        sess.run(image_graph[1:])
        sess.run(warped_graph[1:])
        sess.run(tf.global_variables_initializer())

        gradients = sess.run(compute_gradients)

When extracting the operation and running it by feeding my tensors with feed_dict, gradient_computation doesn't work, that's why I used tf.graph_util.import_graph_def(...).

Hope this might help anyone facing the same issue.