cve_lookup installation fails seemingly at random

[jstucke]

cve_lookup installation is inconsistent fails at random points with the trace

13:28:46 [2021-04-21 13:28:46][backend][INFO]: Running ../plugins/analysis/cve_lookup/install.sh
(...)
13:28:52 Setting up database
13:28:52 Traceback (most recent call last):
13:28:52 File "setup_repository.py", line 298, in
13:28:52 main()
13:28:52 File "setup_repository.py", line 287, in main
13:28:52 init_repository(extraction_path, args.target, years=years)
13:28:52 File "setup_repository.py", line 229, in init_repository
13:28:52 import_cpe(cpe_extract_path=extraction_path)
13:28:52 File "setup_repository.py", line 73, in import_cpe
13:28:52 insert_into(query='insert_cpe', table_name='cpe_table', input_data=setup_cpe_table(get_cpe_content(path=cpe_extract_path)))
13:28:52 File "setup_repository.py", line 77, in get_cpe_content
13:28:52 dp.download_cpe(download_path=path)
13:28:52 File "/home/jenkins/workspace/focal/src/plugins/analysis/cve_lookup/internal/data_parsing.py", line 60, in download_cpe
13:28:52 process_url(CPE_URL, download_path)
13:28:52 File "/home/jenkins/workspace/focal/src/plugins/analysis/cve_lookup/internal/data_parsing.py", line 37, in process_url
13:28:52 zipped_data = ZipFile(BytesIO(request.content))
13:28:52 File "/usr/lib/python3.8/zipfile.py", line 1269, in init
13:28:52 self._RealGetContents()
13:28:52 File "/usr/lib/python3.8/zipfile.py", line 1336, in _RealGetContents
13:28:52 raise BadZipFile("File is not a zip file")
13:28:52 zipfile.BadZipFile: File is not a zip file

Some kind of error mitigation and maybe retries for the downloads could fix the issue.

[jstucke]

side note: the error logging could be more helpful (i.e. what file actually failed)

[jstucke]

It seems the links for the cpe dictionaries were updated today and don't work at the moment. Maybe the problem will fix itself in the next days.

[jstucke]

The error did not fix itself. Instead, it was replaced by this:

Setting up database
Traceback (most recent call last):
File "setup_repository.py", line 298, in
main()
File "setup_repository.py", line 287, in main
init_repository(extraction_path, args.target, years=years)
File "setup_repository.py", line 231, in init_repository
import_cve(cve_extract_path=extraction_path, years=years)
File "setup_repository.py", line 164, in import_cve
cve_list, summary_list = get_cve_import_content(cve_extract_path, year_selection)
File "setup_repository.py", line 97, in get_cve_import_content
cve_data, summary_data = dp.extract_cve(file)
File "FACT_core/src/plugins/analysis/cve_lookup/internal/data_parsing.py", line 107, in extract_cve
return extract_data_from_cve(json.loads(Path(cve_file).read_text()))
File "FACT_core/src/plugins/analysis/cve_lookup/internal/data_parsing.py", line 99, in extract_data_from_cve
cpe_entries = list(set(extract_cpe_data_from_cve(feed['configurations']['nodes'])))
File "~/git/github/FACT_core/src/plugins/analysis/cve_lookup/internal/data_parsing.py", line 72, in extract_cpe_data_from_cve
cpe['cpe23Uri'], cpe.get('versionStartIncluding', ''), cpe.get('versionStartExcluding', ''),
KeyError: 'cpe23Uri'