Text generation yields different outputs despite temperature = 0.0

[ScTof]

Describe the bug
Even though temperature is set to 0.0 in the configs, GPT-NeoX' generate.py (interactive mode) produces different results different outputs for the same input submitted multiple times in a row.

To Reproduce
Steps to reproduce the behavior:

1. Use the attached config files
2. Run python deepy.py generate.py PATH/TO/CONFIGS_DIR/*
3. Submit a query (e.g., "This is a test:")
4. Resubmit the same query
5. Observe the different outputs

Expected behavior
As far as I'm concerned, setting temperature to 0.0 should make GPT-NeoX produce the same output for the same input, no matter how often it is submitted. At the moment, it yields the same output only if the model is reloaded after each query. Setting reload = false or reload = true doesn't make a difference.

Environment (please complete the following information):

• GPUs: 2
• Configs: See attachments (notably, set temperature = 0.0 and recompute = true)
  20B.yml.txt
  local_setup_20B.yml.txt
  text_generation.yml.txt