Update dependencies v0.56

[juhoinkinen]

Dependency updates for Annif v0.56.

Updates to (edited):

• Omikuji 0.4.x
• TensorFlow 2.7.0 & lmbd 1.3.0
• Gensim 4.1.x
• joblib 1.1.0
• Optuna 2.10.x
• SciPy 1.7.x
• Scikit-learn 1.0.2
• NumPy 1.21.x
• Click 8.0.x
• Flask >=1.0.4,<3
• Connexion to use connexion2 PyPI project

Below is a table of Omikuji 0.3.4 and 0.4.1 models comparison. Training was performed on full Finnish Finna dataset (real and user times include preprocessing), and evaluation in parallel.

		v0.3	v0.4
train time
	real	190m	174m
	user	2061m	1789m
	reported by Omikuji	7776s	6783s
eval time
	real	2m59s	2m3s
	user	53m55s	14m33s
	sys	3m18s	1m47s
model size on disk		3.8G	3.4G
eval results
	JYU-theses	0.4716	0.4759
	kirjaesittelyt2021	0.4511	0.4496
	kirjastonhoitaja	0.2759	0.2824
	satakunnan-kansa	0.2808	0.2524
	vapaakappaleet-orig	0.4648	0.4613
	average	0.3888	0.3843

[codecov]

Codecov Report

Merging #549 (f6cf926) into master (586adba) will decrease coverage by 0.03%.
The diff coverage is 84.61%.

@@            Coverage Diff             @@
##           master     #549      +/-   ##
==========================================
- Coverage   99.53%   99.49%   -0.04%     
==========================================
  Files          80       80              
  Lines        5341     5340       -1     
==========================================
- Hits         5316     5313       -3     
- Misses         25       27       +2     

Impacted Files	Coverage Δ
annif/backend/omikuji.py	96.42% <60.00%> (-2.34%)	⬇️
annif/backend/pav.py	98.90% <100.00%> (ø)
annif/backend/tfidf.py	98.80% <100.00%> (-0.05%)	⬇️
annif/eval.py	100.00% <100.00%> (ø)
annif/lexical/mllm.py	100.00% <100.00%> (ø)
annif/lexical/util.py	100.00% <100.00%> (ø)
annif/suggestion.py	99.21% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 586adba...f6cf926. Read the comment docs.

[juhoinkinen]

Other points:

• Omikuji models trained with previous version do not work on the new Omikuji version. Others models stay working, also NN ensemble and TFIDF models despite TensorFlow and Gensim version updates.
• Python 3.6 & 3.7 tests show now a RuntimeWarning raised by test_english_analyzer_normalize_word, but it has been around since Optimize startup time using local & lazy imports (take 2) #544.
• Python 3.6 tests show also many DeprecationWarnings.

[juhoinkinen]

Some dependencies were not updated to newer versions because they do not support Python 3.6; some latest versions do not support even 3.7:

• SciPy 1.7.x and 1.6.x require 3.7+ (to-be-released 1.8.0 will require Python 3.8+)
• NumPy 1.20 requires Python 3.7-3.9; 1.21 requires 3.7-3.10; 1.22 requires 3.8-3.10
• Scikit-learn 1.0 requires Python 3.7-3.9 (stwfsa has scikit-learn pinned to 0.24.x >= 0.24.x (but to ==0.24.2 in Pipfile?))
• TensorFlow 2.7 has wheels and is tested for Python 3.7-3.9

Should we keep Python 3.6 support for Annif?

[osma]

We've tried to support three consecutive versions of Python. But we already currently have support for 3.7, 3.8 and 3.9, so dropping 3.6 wouldn't violate that policy. Python 3.6 has been EOL'd and the final release 3.6.15 was released on 2021-09-04.

Python 3.6 is the default version in Ubuntu 18.04, but it's possible to install newer versions of Python from PPAs.

So I think dropping 3.6 support would be OK. Should that be done in a separate PR or is it easier to just do it in this one? (I guess that would mainly involve reorganizing the CI setup, and updating README.md)

Next steps would then be adding support for 3.10 and then we can consider when to drop 3.7.

[juhoinkinen]

Sounds good, I make a separate PR for dropping Python 3.6 support.

[osma]

Omikuji eval time has improved a lot in 0.4! The other numbers look good as well.

What happens when you try to use an old model with omikuji 0.4? Is there some kind of exception? Should we catch that and display a more user friendly error message?

[juhoinkinen]

Rebased on master after dropping Python 3.6 support (#550) and force-pushed.

[juhoinkinen]

What happens when you try to use an old model with omikuji 0.4? Is there some kind of exception? Should we catch that and display a more user friendly error message?

Omikuji shows first Failed to load model: Unable to deserialize tree: invalid type: map, expected variant identifier, then about one-screen length of traceback and RuntimeError: Failed to load model from data/projects/yso-bonsai-old-fi/omikuji-model.

This already hints something being wrong with the model, so I'm not sure if it is worth processing it within Annif...?

[osma]

[...] about one-screen length of traceback and RuntimeError: Failed to load model from data/projects/yso-bonsai-old-fi/omikuji-model
This already hints something being wrong with the model, so I'm not sure if it is worth processing it within Annif...?

I think that showing tracebacks to the user is not a recipe for good UX... It would be better to show a more informative message such as "Omikuji models trained on Annif versions older than 0.56 cannot be loaded. Please retrain your project."

[osma]

@juhoinkinen Since you've upgraded (again) Click to 8.0.*, we need to take care of also upgrading Connexion and Flask to avoid hitting the version mismatch problem reported in #533 once more (which was fixed by downgrading Click back to 7.1.2 in PR #534).

There's a little problem with Connexion releases though - PyPI currently only has 2.9.0 which doesn't support Flask 2. Version 2.10.0 has been released on GitHub but on the PyPI side it has a different name connexion2. See here for details. So we should probably switch to connexion2 at least for now.

[juhoinkinen]

Yes, I already switched to connexion2 in Annif's requirements, but now I noticed some dependency is still using connexion, so both connexion 2.7.0 and connexion2 2.10.0 end up to be installed.

As we don't actually need connexion2 or Click 8, maybe it's best to just go back to Click 8 (again)...? Edit: to Click 7

[juhoinkinen]

swagger-tester being the dependency requiring connexion.

[osma]

Ah, right. The dependency on connexion seems to come from swagger-tester which is a dev dependency for Annif. Its last release 0.2.12 was made in May 2018 and the project seems pretty dead. Maybe we should try to get rid of this dependency - it's used in test_swagger.py for some basic exercise of the REST API methods.

[juhoinkinen]

I checked the compatibility of models trained with Annif 0.55 after upgrading also SciPy, Numpy and Scikit-learn, and TensorFlow again/further.

The usual warnings like UserWarning: Trying to unpickle estimator TfidfTransformer from version 0.24.2 when using version 1.0.2. [...] are shown but models seem to work (suggestions and scores remain the same for a short text input).

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[juhoinkinen]

Rebased on master after removing swagger-tester dependency (#551) and force-pushed.