Cannot set worker.SingleTestResult.passed to False (python3.10)

[rabah-khalek]

Issue Type

Bug

Source

source

Giskard Library Version

1.7.0

Giskard Server Version

1.3.2

OS Platform and Distribution

both mac and linux

Python version

3.10

Installed python packages

tensorlfow==2.9.2

Current Behaviour?

Traceback (most recent call last):
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.10/site-packages/google/protobuf/internal/python_message.py", line 702, in field_setter
    new_value = type_checker.CheckValue(new_value)
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.10/site-packages/google/protobuf/internal/type_checkers.py", line 126, in CheckValue
    raise TypeError(message)
TypeError: False has type <class 'numpy.bool_'>, but expected one of: (<class 'bool'>, <class 'numbers.Integral'>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.10/site-packages/google/protobuf/internal/python_message.py", line 558, in init
    setattr(self, field_name, field_value)
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.10/site-packages/google/protobuf/internal/python_message.py", line 704, in field_setter
    raise TypeError(
TypeError: Cannot set worker.SingleTestResult.passed to False: False has type <class 'numpy.bool_'>, but expected one of: (<class 'bool'>, <class 'numbers.Integral'>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rak/Documents/giskard-master/python-client/giskard/ml_worker/utils/error_interceptor.py", line 48, in wrapper
    res = await loop.run_in_executor(pool, behavior, request, context)
  File "/usr/local/Cellar/[email protected]/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/rak/Documents/giskard-master/python-client/giskard/ml_worker/server/ml_worker_service.py", line 146, in runTest
    exec(request.code, _globals)
  File "<string>", line 26, in <module>
  File "/Users/rak/Documents/giskard-master/python-client/giskard/ml_worker/testing/performance_tests.py", line 428, in test_diff_precision
    return self._test_diff_prediction(
  File "/Users/rak/Documents/giskard-master/python-client/giskard/ml_worker/testing/performance_tests.py", line 301, in _test_diff_prediction
    metric_1 = test_fn(reference_slice, model).metric
  File "/Users/rak/Documents/giskard-master/python-client/giskard/ml_worker/testing/performance_tests.py", line 195, in test_precision
    return self._test_classification_score(precision_score, actual_slice, model, threshold)
  File "/Users/rak/Documents/giskard-master/python-client/giskard/ml_worker/testing/performance_tests.py", line 87, in _test_classification_score
    SingleTestResult(
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.10/site-packages/google/protobuf/internal/python_message.py", line 560, in init
    _ReraiseTypeErrorWithFieldName(message_descriptor.name, field_name)
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.10/site-packages/google/protobuf/internal/python_message.py", line 477, in _ReraiseTypeErrorWithFieldName
    raise exc.with_traceback(sys.exc_info()[2])
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.10/site-packages/google/protobuf/internal/python_message.py", line 558, in init
    setattr(self, field_name, field_value)
  File "/Users/rak/Documents/giskard-master/python-client/.venv/lib/python3.10/site-packages/google/protobuf/internal/python_message.py", line 704, in field_setter
    raise TypeError(
TypeError: Cannot set worker.SingleTestResult.passed to False: False has type <class 'numpy.bool_'>, but expected one of: (<class 'bool'>, <class 'numbers.Integral'>) for field SingleTestResult.passed
×

Standalone code OR list down the steps to reproduce the issue

- upload the Text_classification_Using_Tensorflow_Neural_Network.ipynb model
- run all the automatic test
- all will fail with the error above except: 
  - Right label
  - Classification Probability in Range
  - Custom Test

Relevant log output

No response

[andreybavt]

@rabah-khalek which version of tensorflow were you using? I'm having issues serializing the TF model from this notebook

[rabah-khalek]

Yes, I left a note back then in the manual test plan: tensorflow == 2.9.2 tensorflow 2.11.0 won’t work.

…

On 9 Feb 2023, at 11:17, Andrey Avtomonov ***@***.***> wrote: @rabah-khalek <https://github.com/rabah-khalek> which version of tensorflow were you using? I'm having issues serializing the TF model from this notebook — Reply to this email directly, view it on GitHub <#731 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHZRUPIKATEXWIYHUZHZ4J3WWS73XANCNFSM6AAAAAAUVW43NA>. You are receiving this because you were mentioned.

[andreybavt]

Weird, I didn't manage to reproduce with the following python 3.10.8 conda environment:

env.txt

@rabah-khalek , could you maybe attach your pip freeze too?

[rabah-khalek]

I have just rechecked, I still have the same error. This is my pip freeze output:

absl-py==1.4.0
anyio==3.6.2
appnope==0.1.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
astroid==2.14.1
asttokens==2.2.1
astunparse==1.6.3
async-timeout==4.0.2
attrs==22.1.0
Babel==2.10.3
backcall==0.2.0
bandit==1.7.4
beautifulsoup4==4.11.1
black==22.10.0
bleach==5.0.1
CacheControl==0.12.11
cachetools==5.3.0
cachy==0.3.0
catboost==1.1.1
certifi==2022.9.24
cffi==1.15.1
cfgv==3.3.1
chardet==5.0.0
charset-normalizer==2.1.1
cleo==1.0.0a5
click==8.1.3
cloudpickle==2.2.1
colorama==0.4.6
coverage==6.5.0
crashtest==0.3.1
cycler==0.11.0
darglint==1.8.1
decorator==5.1.1
defusedxml==0.7.1
deptry==0.5.13
dill==0.3.6
distlib==0.3.6
docutils==0.19
dparse==0.6.2
dulwich==0.20.50
eli5==0.13.0
entrypoints==0.4
exceptiongroup==1.0.0
executing==1.2.0
fastjsonschema==2.16.2
filelock==3.8.0
flake8==3.9.2
flatbuffers==1.12
fonttools==4.38.0
frozenlist==1.3.3
fsspec==2023.1.0
gast==0.4.0
-e git+https://github.com/Giskard-AI/giskard.git@e57911a8f8a85c9c2c0441451e93b7838cfec47b#egg=giskard&subdirectory=python-client
gitdb==4.0.9
GitPython==3.1.29
google-auth==2.16.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
googleapis-common-protos==1.56.4
graphviz==0.20.1
grpcio==1.50.0
grpcio-status==1.48.2
grpcio-tools==1.48.2
h5py==3.8.0
html5lib==1.1
httpretty==1.1.4
huggingface-hub==0.12.0
identify==2.5.8
idna==3.4
importlib-metadata==4.13.0
iniconfig==1.1.1
ipykernel==5.5.6
ipython==8.9.0
ipython-genutils==0.2.0
ipywidgets==8.0.2
isort==5.10.1
jaraco.classes==3.2.3
jedi==0.18.1
Jinja2==3.1.2
joblib==1.2.0
json5==0.9.10
jsonschema==4.16.0
jupyter==1.0.0
jupyter-console==6.4.4
jupyter-server==1.21.0
jupyter_client==7.4.4
jupyter_core==4.11.2
jupyterlab==3.5.0
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.3
jupyterlab_server==2.16.1
keras==2.9.0
Keras-Preprocessing==1.1.2
keyring==23.13.1
kiwisolver==1.4.4
lazy-object-proxy==1.8.0
libclang==15.0.6.1
llvmlite==0.39.1
lockfile==0.12.2
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib==3.5.3
matplotlib-inline==0.1.6
mccabe==0.6.1
mistune==2.0.4
mixpanel==4.10.0
more-itertools==9.0.0
msgpack==1.0.4
multidict==6.0.4
multiprocess==0.70.14
mypy==0.982
mypy-extensions==0.4.3
mypy-protobuf==3.3.0
nbclassic==0.4.7
nbclient==0.7.0
nbconvert==7.2.3
nbformat==5.7.0
nest-asyncio==1.5.6
nodeenv==1.7.0
notebook==6.4.12
notebook_shim==0.2.0
numba==0.56.3
numpy==1.21.6
oauthlib==3.2.2
opt-einsum==3.3.0
packaging==21.3
pandas==1.3.5
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.10.1
pbr==5.11.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.2.0
pkginfo==1.9.6
platformdirs==2.5.2
plotly==5.11.0
pluggy==1.0.0
poetry==1.2.2
poetry-core==1.3.2
poetry-plugin-export==1.2.0
pre-commit==2.20.0
prometheus-client==0.15.0
prompt-toolkit==3.0.36
protobuf==3.19.6
psutil==5.9.3
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.7.0
pycparser==2.21
pydantic==1.10.2
pydocstyle==6.3.0
pyflakes==2.3.1
Pygments==2.13.0
pylev==1.4.0
pylint==2.16.0
pyparsing==3.0.9
pyrsistent==0.18.1
pytest==7.2.0
pytest-cov==3.0.0
python-daemon==2.3.2
python-dateutil==2.8.2
pytz==2022.5
pyupgrade==2.38.4
PyYAML==6.0
pyzmq==24.0.1
qtconsole==5.3.2
QtPy==2.2.1
regex==2022.10.31
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
responses==0.18.0
rsa==4.9
safety==1.10.3
scikit-learn==1.0.2
scipy==1.7.3
Send2Trash==1.8.0
shap==0.41.0
shellingham==1.5.0.post1
simplegeneric==0.8.1
six==1.16.0
slicer==0.0.7
smmap==5.0.0
sniffio==1.3.0
snowballstemmer==2.2.0
soupsieve==2.3.2.post1
stack-data==0.6.2
stevedore==3.5.2
tabulate==0.9.0
tenacity==8.1.0
tensorboard==2.9.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.9.2
tensorflow-estimator==2.9.0
tensorflow-io-gcs-filesystem==0.30.0
termcolor==2.2.0
terminado==0.17.0
threadpoolctl==3.1.0
tinycss2==1.2.1
tokenize-rt==4.2.1
tokenizers==0.13.2
toml==0.10.2
tomli==2.0.1
tomlkit==0.11.6
torch==1.13.1
tornado==6.2
tqdm==4.64.1
traitlets==5.5.0
transformers==4.26.0
tweepy==4.12.1
types-protobuf==3.20.4.2
typing_extensions==4.4.0
urllib3==1.26.12
virtualenv==20.16.6
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.4.1
Werkzeug==2.2.2
widgetsnbextension==4.0.3
wrapt==1.14.1
xattr==0.9.9
xxhash==3.2.0
yarl==1.8.2
zipp==3.10.0
zstandard==0.15.2

[rabah-khalek]

this is the same test that you did. I will refresh my installation of my python3.10 env.

[rabah-khalek]

Okay, I cleaned my environment, re-installed python3.10 and tensorflow 2.9.2 and I can't reproduce the error.

This is the diff between two pip freeze:

colordiff new.txt old.text 
8a9
> async-timeout==4.0.2
49a51,52
> frozenlist==1.3.3
> fsspec==2023.1.0
64a68
> huggingface-hub==0.12.0
105a110,111
> multidict==6.0.4
> multiprocess==0.70.14
143a150
> pyarrow==11.0.0
165a173
> regex==2022.10.31
168a177
> responses==0.18.0
197a207
> tokenizers==0.13.2
200a211
> torch==1.13.1
203a215,216
> transformers==4.26.0
> tweepy==4.12.1
214a228,229
> xxhash==3.2.0
> yarl==1.8.2

[rabah-khalek]

I tried installing these libraries one by one and reporfmed the test but I couldn't reproduce the issue:

pip install transformers==4.26.0
pip install tweepy
pip install responses==0.18.0
pip install torch==1.13.1

I suggest closing it for now, until we can reproduce it again.