Does not work with TensorFlow versions >= 1.3

[WeitaoVan]

Thanks for your inspiring idea and the corresponding code.

I try to run the tensorflow code train_cifar10.py.
But it takes more than 2 hours to construct the computational graph and I'm still stuck here.
The screen does not print any thing.
If I replace the CNNs in the original code with a plain ResNet-32 (without Weight Normalization or other tricks), the whole code goes on well.

Do you know what might be wrong?

Thanks.

[tarvaina]

Hi and thanks for the comment.

Which TF version are you using? The newest version seems to hang up in the way you describe. TF 1.2.1 should work.

I think the issue is in either weight norm or ema model code or maybe their combination. Unfortunately I haven’t had time to hunt it down. PRs welcome if you figure it out.

[WeitaoVan]

I changed to TF 1.2.1. Now it works fine.
Thank you!

[tarvaina]

Nice! Thanks for your comment.

[tarvaina]

See the discussion in pull request #5. This is related to the data-dependent initialization of weight-normalized layers.

Smart variable initialization appears to be broken in TensorFlow since 1.3, and may get fixed in the next release. Here's the currently active bug report in TensorFlow issues: TensorFlow variable initializers broken #13351. As a possible workaround, here's a smarter initialization scheme from OpenAI: smart_initialize.py.

[liuajian]

your python --version? @WeitaoVan

[liuajian]

I can not download the tf1.2.x with python3.x

[tarvaina]

I think I used both Python 3.5 and 3.6 in the development of this, and TF 1.2.x.

[LuJyKa]

On Tensorflow 1.8, I found a workaround:
There is a process to ensure no cycle dependencies, but cost a lot of time.

Comment the line 851&852 around in pyfolder/Lib/site-packages/tensorflow/python/ops/variables.py:

    def has_cycle(op, path):
      """Detect cycles in the dependencies of `initial_value`."""
      if op.name in path:
        return True
      path.add(op.name)
      for op_input in op.inputs:
        if has_cycle(op_input.op, path):
          return True
      for op_control_input in op.control_inputs:
        if has_cycle(op_control_input, path):
          return True
      path.remove(op.name)
      return False
    # if has_cycle(initial_value.op, path=set()):  # <--- comment this
    #   return initial_value  # <--- comment this

    return self._safe_initial_value_from_tensor(initial_value, op_cache={})

[liangzimei]

tf 1.2.0 has a bug using dropout , see here . tf 1.2.1 is ok.

[rajneo]

Tried running svhn.py.
No output shown for atleast 2hrs.
Using tensorflow 1.2.1

Do you know what might be wrong

[Xiaotian0726]

I can not download the tf1.2.x with python3.x

Tensorflow 1.x is not available for python 3.8, use python 3.5 and 3.6 instead. See more at issue #39768 of tensorflow