Make the test suite connectivity-agnostic.

[jayaddison]

Feature or Bugfix

• Bugfix

Purpose

• Intended to allow the test suite to run equally well whether the test host is online.

Detail

• Use local test HTTP server instances instead of fetching remote resources.
• Remove a remote image that causes epubcheck to fail when checking an epub file that contains remote references (because the image couldn't be retrieved during the Sphinx build).

Relates

• Resolves tests: network state (online/offline) affects epubcheck unit test outcome. #12094.

[jayaddison]

The Windows test output for this change appears to have uncovered an unrelated problem: handling http:https://localhost:7777/sphinx.png raises a WinError exception somewhere on that platform while it does not on Linux.

WARNING: Could not fetch remote image: http:https://localhost:7777/sphinx.png [[WinError 267] The directory name is invalid: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\...\\images\\_build\\doctrees\\images\\http/localhost:7777']

My best guess is around this area of code (it has quite a large try...catch block):

sphinx/sphinx/transforms/post_transforms/images.py

Lines 57 to 113 in 98d0cc8

	try:
	basename = os.path.basename(node['uri'])
	if '?' in basename:
	basename = basename.split('?')[0]
	if basename == '' or len(basename) > MAX_FILENAME_LEN:
	filename, ext = os.path.splitext(node['uri'])
	basename = sha1(filename.encode(), usedforsecurity=False).hexdigest() + ext
	basename = CRITICAL_PATH_CHAR_RE.sub("_", basename)
	dirname = node['uri'].replace(':https://', '/').translate({ord("?"): "/",
	ord("&"): "/"})
	if len(dirname) > MAX_FILENAME_LEN:
	dirname = sha1(dirname.encode(), usedforsecurity=False).hexdigest()
	ensuredir(os.path.join(self.imagedir, dirname))
	path = os.path.join(self.imagedir, dirname, basename)
	headers = {}
	if os.path.exists(path):
	timestamp: float = ceil(os.stat(path).st_mtime)
	headers['If-Modified-Since'] = epoch_to_rfc1123(timestamp)
	config = self.app.config
	r = requests.get(
	node['uri'], headers=headers,
	_user_agent=config.user_agent,
	_tls_info=(config.tls_verify, config.tls_cacerts),
	)
	if r.status_code >= 400:
	logger.warning(__('Could not fetch remote image: %s [%d]') %
	(node['uri'], r.status_code))
	else:
	self.app.env.original_image_uri[path] = node['uri']
	if r.status_code == 200:
	with open(path, 'wb') as f:
	f.write(r.content)
	last_modified = r.headers.get('last-modified')
	if last_modified:
	timestamp = rfc1123_to_epoch(last_modified)
	os.utime(path, (timestamp, timestamp))
	mimetype = guess_mimetype(path, default='*')
	if mimetype != '*' and os.path.splitext(basename)[1] == '':
	# append a suffix if URI does not contain suffix
	ext = get_image_extension(mimetype)
	newpath = os.path.join(self.imagedir, dirname, basename + ext)
	os.replace(path, newpath)
	self.app.env.original_image_uri.pop(path)
	self.app.env.original_image_uri[newpath] = node['uri']
	path = newpath
	node['candidates'].pop('?')
	node['candidates'][mimetype] = path
	node['uri'] = path
	self.app.env.images.add_file(self.env.docname, path)
	except Exception as exc:
	logger.warning(__('Could not fetch remote image: %s [%s]') % (node['uri'], exc))

These lines initially seem like a possibility:

sphinx/sphinx/transforms/post_transforms/images.py

Lines 70 to 71 in 98d0cc8

	ensuredir(os.path.join(self.imagedir, dirname))
	path = os.path.join(self.imagedir, dirname, basename)

[jayaddison]

The Windows test output for this change appears to have uncovered an unrelated problem: handling http:https://localhost:7777/sphinx.png raises a WinError exception somewhere on that platform while it does not on Linux.

WARNING: Could not fetch remote image: http:https://localhost:7777/sphinx.png [[WinError 267] The directory name is invalid: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\...\\images\\_build\\doctrees\\images\\http/localhost:7777']

See #12100 (comment) for more details about the cause of this. In short: the colon character, as used to indicate the network port number within URIs, is not considered a valid character within the components of directory paths on Windows, and therefore remote image URIs such as this one were caused a problem for the existing ImageDownloader transform.

[jayaddison]

No further changes planned. Please note that this branch depends upon / includes #12253.

[jayaddison]

Thank you!