can't use higher than v1.11.1

[150d]

Hi,

I'm trying to run SVN with Apache v2.4.4 (x64, VC15, ApacheHaus build) on Windows Server 2016. If I'm using the libraries from this project (VC15, x64) in version v1.11.1 or below, the service works fine.

But if I'm trying to load the modules of a current version (v1.12.x or v1.13.x), starting the service fails with the following error:

Cannot load [path]/mod_authz_svn.so into server: Die angegebene Prozedur wurde nicht gefunden.

Did something change between v11 and v12 that needs adjustment of the setup?

Regards

[nono303]

Hi,
As I know, nothing change between v11 and v12 that needs adjustment of the setup.
Do you have more trace about "Cannot load [path]/mod_authz_svn.so" like in windows events ?

[150d]

I'm afraid I don't. There is just this message in the event log, nothing else:

(I'm afraid the GitHub system won't allow me to paste XML code here. But there really isn't any more than this:)

httpd.exe: Syntax error on line 223 of C:/Program Files/Apache/conf/httpd.conf: Cannot load C:/Program Files/Apache.supplemental/win-svn-1.13.0/vc15/x64/mod_authz_svn.so into server: Die angegebene Prozedur wurde nicht gefunden.

If I just change the path to use a v10 or v11 version (all downloads from this project), there is no error and the service runs fine.

Regards

PS: Only mod_authz_svn fails to load, not mod_dav_svn. Might that be some clue?

[Jan-E]

The libaprutil-1.dll by @nono303 depends on expat.dll. The Apachehaus builds import xml.lib in libaprutil-1.dll. Moreover, the Apachehaus builds are vc15 and the releases in this repo are vs16 since release version 1.12.0. Just too many differences between the two. There is a tiny chance you may get things running by replacing the Apachehaus DLL's with the one from this repo.

Start by replacing AH's libaprutil-1.dll by NONO's libaprutil-1.dll + expat.dll. Afterwards do the same with libapr-1.dll and libhttpd.dll. If Apache still runs fine after these replacements tou are lucky, but Apache might be unstable.

[150d]

Would the easier solution be to change my Apache build to something else? I noted that ApacheLounge is offering builds on VS16 - would you recommend that, or another build?

[nono303]

Thanks @Jan-E to note the dependencies point as specified in https://github.com/nono303/win-svn/blob/master/README.md (and see https://www.apachelounge.com/viewtopic.php?p=38610#38610)
@150d I also provide vc15 builds https://github.com/nono303/win-svn/tree/master/vc15. I let you try following readme instructions with vc15 build and let you give us a feedback

[Jan-E]

Would the easier solution be to change my Apache build to something else? I noted that ApacheLounge is offering builds on VS16 - would you recommend that, or another build?

The Apachelounge builds also use a libaprutil-1.dll without an external expat.dll, so that would not make a difference. @nono303's way of building apparently differs from that of both Apachehaus and Apachelounge. You could try it with the x64/vc15 builds by @nono303

[150d]

I'm afraid I didn't understand your last:

Currently, I'm using the ApacheHaus build, which is VC15.

In that, I'm trying to load mod_dav_svn from @nono303 , also from its VC15 directory (@Jan-E your link.)

This combination generates the error on versions v12 or v13, not on v11.

Which Apache build should I use instead? Could you give a reference/link to that maybe?

[nono303]

Hi @150d
Can you have a try with freshly released 1.14.0 and give me your feedback?

[150d]

Hi,

my apologies for the delay - this little pandemic thing messed up a lot of schedules... ;-)

I'm afraid v14 gave me the same loading error the previous versions did:

httpd.exe: Syntax error on line 228 of C:/Program Files/Apache/conf/httpd.conf: Cannot load C:/Program Files/Apache.supplemental/win-svn-master-1.14.0/vc15/x64/mod_authz_svn.so into server: Die angegebene Prozedur wurde nicht gefunden.

Regards

[nono303]

It may be a dependency (vc runtime, dll..) or an incompatibility with your httpd build... Look at event log if you have more detailed information

[150d]

The message I reported already comes from the event log. It's the only relevant message there during the time frame. :-(

[Jan-E]

Could you try if the x64 build from https://ci.appveyor.com/project/Jan-E/svn-windows/builds/33232754/job/juthm7sefil8tx52/artifacts runs and does not have the error?

[Jan-E]

Here is the x64 VC15 version https://ci.appveyor.com/project/Jan-E/svn-windows/builds/34506133/job/xle25lc3whecmdqg/artifacts

It was built with and for the x64 VC15 Apache httpd 2.4.46 of Apachelounge. Thanks, @SteffenAL

It ran the tests quite well. A lot of XFAILS are eXpected fails. Only two unexpected fails:
https://ci.appveyor.com/project/Jan-E/svn-windows/builds/34506133/job/k693fd7uva7cw7ny#L4494

[nono303]

Could you check with https://github.com/lucasg/Dependencies if they are some missing dll on your installation (with my build) for mod_authz_svn.so please?
just post the screenshot like mine:

[150d]

The plot thickens (that tip about "Dependencies" was gold!)

I seem to have a problem with the first few DLLs:
libapr-1.dll
libaprutil-1.dll
libhttpd.dll
mod_dav.so (for mod_dav_svn)

As it turns out, up until now they were loaded from some third-party directory (VisualSVN/bin) within the system path, not from Apache/bin or [svn-modules]/dep.

From where should those DLLs load? I noticed they exist in apache/bin as well as mod_svn/dep.

Regards

[nono303]

@150d

From where should those DLLs load? I noticed they exist in apache/bin as well as mod_svn/dep.

1. same path as mod_*.so or apache/bin if there is no conflict
2. anywhere in a folder referenced in Windows Path

[150d]

Well, something is fishy here. I went back to v10 (working so far) to sort things out:

PATH: VisualSVN/bin, NOT to Apache/bin
-> httpd starts fine

PATH: remove VisualSVN/bin
-> error "mod_dav_svn.so: module not found"

PATH: add Apache/bin
-> error "mod_dav_svn.so: module not found"

My Apache directory is unmodified from distribution, so the files in there should be ok. Why do I still get this failures?

[Jan-E]

My Apache directory is unmodified from distribution, so the files in there should be ok. Why do I still get this failures?

Probably because the libapr*.dll's in this repo by @nono303 are not compatible with the libapr*.dll's in the httpd distributions. They depend on expat.dll, which is not there in the httpd distributions. Please try the mod_dav_svn.so in the zipfile on https://ci.appveyor.com/project/Jan-E/svn-windows/builds/34506133/job/xle25lc3whecmdqg/artifacts (in the modules map).

[150d]

Thanks @Jan-E , but I'm still not there. I'm testing with three versions now:

v10:
Loads when the "foreign" DLLs are in the path, doesn't load if not.
The following DLLs are loaded (as seen by Sysinternal's ProcessExplorer):
libsvn_delta-1.dll
libsvn_fs_fs-1.dll
libsvn_fs_util-1.dll
libsvn_fs-1,dll
libsvn_repos-1.dll
libsvn_subr-1.dll
msvcr120.dll (yes, I DO have the MS Runtime installed, v14.27.x in both x86 and x64 variants.)

v14 (this repo):
mod_dav_svn: loads with the "foreign" DLLs, same list. Without, error "module not found"
mod_authz_svn.so: doesn't load, even with "foreign" DLLs. Error: "procedure not found"

v14 (Jan-E):
same as v14 (this repo)

[Jan-E]

Put the libsvn_*.dll's from https://ci.appveyor.com/project/Jan-E/svn-windows/builds/34506133/job/xle25lc3whecmdqg/artifacts in apache/bin

[150d]

Thanks @Jan-E , but I don't want to do "drop-ins" of DLLs into the Apache directory if I can help it. With any future updates, I'd never hear the end of it.

But I followed your idea the other way round:

I removed apache/bin and similar from my PATH, so mod_authz_svn.so only includes DLLs from its own /bin and /bin/dep directories (as verified with "Dependencies"). With one exception: mod_authz_svn.so requires libhttpd.dll, which is not included in its /dep. Above, you already suggested using @nono303 's build of libhttpd.dll - but where can I find this?

I also noted that mod_authz_svn.so requires libhttpd.dll, but mod_dav_svn.so does not. This could be the difference why the former doesn't load in my system, while the latter does.

Regards

[nono303]

Hi @150d
You're right, libhttpd.dll need to be in /deps if /httpd/bin is not in your Windows PATH.
I'll push them for all built versions in... few hours ;)

[Jan-E]

IMHO, it is a really bad idea to load mod_authz_svn.so into Apache (which has its own libhttpd.dll) and then use another libhttpd.dll in the deps of mod_authz_svn.so. You cannot even be sure which one of those two mod_authz_svn.so will use when it uses a function in libhttps.dll.

[nono303]

You’re fully right @Jan-E; the better is to have all released build with same version, compiler and deps without duplicate binaries on PATH...
But as far as subversion need libhttpd to be build, It might be difficult to assume a backward compatibility with all libhttpd.dll runned with mod_authz_svn.so IF they are not the same than I (2.4.46) or You used to compile it. Does it make sense?

[150d]

@nono303 That's great, thank you! I'm not sure it will really solve my problem, but it shouldn't hurt to include the DLL either, should it?

(Or maybe put the DLL in a seperate, "optional" directory for now instead of including it in the release package? Who knows, maybe the duplicate DLL will cause problems for other users that are running fine now?)

@Jan-E You're right of course, but I wouldn't be better off if I replaced the DLLs Apache ships with (e.g. libapr*.dll) either. And, to be frank, I don't know what else I can do.

Just out of curiosity: What build of Apache are you all running? You must have one that doesn't have the conflicting DLLs. Earlier it was mentioned that ApacheLounge or ApacheHouse wouldn't make a difference, but I couldn't find any other "semi-official" builds. Are you all compiling Apache yourself?

Regards

[nono303]

Are you all compiling Apache yourself?

For my part yes, with all deps

[Jan-E]

Issues like this are exactly the reason why I am using the Apachelounge distribution while building https://ci.appveyor.com/project/Jan-E/svn-windows/builds/34506133/job/xle25lc3whecmdqg/artifacts

[nono303]

Hi @150d
Could you give me a feedback with libhttpd pushed yesterday?
Also added details concerning runtime dependencies on https://github.com/nono303/win-svn/blob/master/README.md : let me know if it seems more clear and understandable for you please.

[150d]

Could you give me a feedback with libhttpd pushed yesterday?

No joy, I'm afraid: Even with the new libhttpd.dll, mod_authz_dav.so will not load.

I've set my path in a way that all includes are fulfilled from the module's directories, not from Apache. Still nothing (while mod_dav_svn.so still loads even in v1.14.0, as before).

I will try a different build of Apache next. Maybe this will clear it up.

Also added details concerning runtime dependencies on https://github.com/nono303/win-svn/blob/master/README.md : let me know if it seems more clear and understandable for you please.

Yes, I do indeed: I had no idea that you even could compile Apache in different ways with effects like that. This hint certainly helps to understand the issue.

MfG

[nono303]

@150d could you give me a zip link of your Apache v2.4.4 x64, VC15 ApacheHaus build to test on my side please?

[Jan-E]

I do not even know if a 2.4.4 ApacheHaus build was ever made, especially in a VC15 version. Back then Gregg was building with VC9 and VC11: https://forum.apachehaus.com/index.php?topic=1211.0

Current ApacheHaus buillds are here: https://www.apachehaus.com/cgi-bin/download.plx

[Jan-E]

See https://forum.apachehaus.com/index.php?topic=965.0 as well.

[150d]

@150d could you give me a zip link of your Apache v2.4.4 x64, VC15 ApacheHaus build to test on my side please?

Of course: I got the file httpd-2.4.41-o11c-x64-vc15-r2.zip from the page at

https://www.apachehaus.com/cgi-bin/download.plx

Just now, I've upgraded this to httpd-2.4.46-o11g-x64-vc15.zip with no changes in behaviour.

(I'm not set on this build for any reason. I've tried to "play it safe" with the VC15 version, but I can try a VS16 version next.)

Additionally, I'm running

mod_authn_ntlm v1.0.7
PHP v7.4.3 (thread-safe)

Regards

[nono303]

Here comes a part of the mystery….
Nothing to deal with dependencies PATH, runtime version or those mentionned above but with an old subject raised at the end of 2017:

OPENSSL_Applink Error for secure connections in dlls

• OPENSSL_Applink Error for secure connections in dlls eclipse-paho/paho.mqtt.c#457 (comment)
• https://www.apachelounge.com/viewtopic.php?t=7861

Here is a (simplified) call stack of mod_dav_svn.so loading

Load Image	c:\subversion\vc15\x64\mod_dav_svn.so		SUCCESS	Image Base: 0x7fff0c980000, Image Size: 0x30000
Load Image	c:\subversion\vc15\x64\libsvn_delta-1.dll	SUCCESS	Image Base: 0x7ffeff8b0000, Image Size: 0x22000
Load Image	c:\subversion\vc15\x64\libsvn_fs-1.dll		SUCCESS	Image Base: 0x7fff19b10000, Image Size: 0x12000
Load Image	c:\subversion\vc15\x64\libsvn_repos-1.dll	SUCCESS	Image Base: 0x7ffecf4d0000, Image Size: 0x40000
Load Image	c:\subversion\vc15\x64\libsvn_subr-1.dll	SUCCESS	Image Base: 0x7ffecf320000, Image Size: 0x1a6000
Load Image	c:\subversion\vc15\x64\libsvn_subr-1.dll	SUCCESS	Image Base: 0x226ca3d0000, Image Size: 0x1a6000
Load Image	c:\subversion\vc15\x64\libsvn_fs_util-1.dll	SUCCESS	Image Base: 0x7fff145a0000, Image Size: 0xb000
Load Image	C:\Windows\System32\ole32.dll			SUCCESS	Image Base: 0x7fff2e190000, Image Size: 0x129000
Load Image	C:\Windows\System32\crypt32.dll			SUCCESS	Image Base: 0x7fff2c9a0000, Image Size: 0x15d000
Load Image	C:\Windows\System32\kernel.appcore.dll		SUCCESS	Image Base: 0x7fff2a530000, Image Size: 0x13000
ReadFile	c:\subversion\vc15\x64\libcrypto-1_1-x64.dll	SUCCESS	Offset: 1 668 096, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
Thread Exit	SUCCESS	Thread ID: 15264, User Time: 0.0000000, Kernel Time: 0.0156250

the kernel.appcore.dll execution stack @thread exit

...
14	KernelBase.dll	LoadLibraryExW + 0x162		0x7fff2cb58982	C:\Windows\System32\KernelBase.dll
15	ucrtbase.dll	_stdio_common_vfwprintf + 0x4f0	0x7fff2c59ba60	C:\Windows\System32\ucrtbase.dll
16	ucrtbase.dll	crt_atexit + 0x87		0x7fff2c5a1357	C:\Windows\System32\ucrtbase.dll
17	ucrtbase.dll	exit + 0x1c2			0x7fff2c59ffc2	C:\Windows\System32\ucrtbase.dll
18	ucrtbase.dll	exit + 0x7f			0x7fff2c59fe7f	C:\Windows\System32\ucrtbase.dll
19	httpd.exe	OPENSSL_Applink + 0x29f		0x7ff6c4e2129f	C:\apachehaus\httpd-2.4.46-o111g-x64-vc15\Apache24\bin\httpd.exe
20	httpd.exe	OPENSSL_Applink + 0x6e2		0x7ff6c4e216e2	C:\apachehaus\httpd-2.4.46-o111g-x64-vc15\Apache24\bin\httpd.exe
21	httpd.exe	OPENSSL_Applink + 0x1fe4	0x7ff6c4e22fe4	C:\apachehaus\httpd-2.4.46-o111g-x64-vc15\Apache24\bin\httpd.exe
22	kernel32.dll	BaseThreadInitThunk + 0x14	0x7fff2dca6fd4	C:\Windows\System32\kernel32.dll
23	ntdll.dll	RtlUserThreadStart + 0x21	0x7fff2ee7cec1	C:\Windows\System32\ntdll.dll

I guess apachehaus doesn't patch httpd.exe with #include "openssl \ applink.c" unlike my homemade version, causing the error loading svn modules (requiring openssl in they own runtime context)

@Jan-E : how do you manage this in apachelounge distribution and in your subversion build?

[Jan-E]

Good catch about the applink error.

AFAIK, @SteffenAL still uses #include "openssl\applink.c in main.c . It is also one of my standard patches when I build Apache myself.

In my Appveyor project, I am just using all the binaries of the ApacheLounge distro. Read the complete build log: https://ci.appveyor.com/project/Jan-E/svn-windows/builds/34506133/job/x1yxesv9q46hkdkd?fullLog=true

Or clone my repo and run it yourself. Appveyor is free for Open Source projects. Just login with your github account.

[nono303]

After digging into my call stack a bite more – in -vvv mode :( , I found that OPENSSL_Applink was not the root cause of the thread exiting but just... forgot to enable mod_dav in my test config ^^
Now to be confident with this, I’ve done this simple test on 2 standard computer (Win7 and Win10) directly on root drive E: (to accord change with your test path…)

1. Install https://aka.ms/vs/16/release/vc_redist.x64.exe
2. Download & extract https://de.apachehaus.com/downloads/httpd-2.4.46-o111g-x64-vc15.zip > E:
3. Change httpd.conf

-Define SRVROOT "/Apache24"
+Define SRVROOT "E:/httpd-2.4.46-o111g-x64-vc15/Apache24"
 ServerRoot "${SRVROOT}"
 
 #
@@ -110,7 +110,7 @@ LoadModule autoindex_module modules/mod_autoindex.so
 LoadModule cgi_module modules/mod_cgi.so
 #LoadModule charset_lite_module modules/mod_charset_lite.so
 #LoadModule data_module modules/mod_data.so
-#LoadModule dav_module modules/mod_dav.so
+LoadModule dav_module modules/mod_dav.so
 #LoadModule dav_fs_module modules/mod_dav_fs.so
 #LoadModule dav_lock_module modules/mod_dav_lock.so
 #LoadModule dbd_module modules/mod_dbd.so
@@ -189,6 +189,9 @@ LoadModule status_module modules/mod_status.so
 #LoadModule watchdog_module modules/mod_watchdog.so
 #LoadModule xml2enc_module modules/mod_xml2enc.so
 
+LoadModule dav_svn_module E:/win-svn/vc15/x64/mod_dav_svn.so
+LoadModule authz_svn_module E:/win-svn/vc15/x64/mod_authz_svn.so
git clone https://github.com/nono303/win-svn.git & git checkout 1.14.0 > E:

4. mklink /h E:\win-svn\vc15\x64\libexpat.dll E:\win-svn\vc15\x64\deps\libexpat.dll and ONLY this one
5. NO change in Windows PATH env
6. launch E:\httpd-2.4.46-o111g-x64-vc15\Apache24\bin>E:\httpd-2.4.46-o111g-x64-vc15\Apache24\bin\httpd -e debug

...
AH01575: loaded module dav_module from E:/httpd-2.4.46-o111g-x64-vc15/Apache24/modules/mod_dav.so
...
AH01575: loaded module dav_svn_module from E:/win-svn/vc15/x64/mod_dav_svn.so
AH01575: loaded module authz_svn_module from E:/win-svn/vc15/x64/mod_authz_svn.so

6. loading succes : https://127.0.0.1/server-info#mod_authz_svn.c
   

@150d Could you do the same with your runtime context please and give me your feedback?

[150d]

It's working.

I still don't quite understand it myself. The problem seems to depend on my particular system though. While working through your (@nono303 ) action list I did the following:

• Made the changes to the config, then tried to run httpd with debug switch. Noted that it did load both modules, but was very unresponsive.

• Ran service again, did not load. My conclusion at the time: Debug loads, without debug it doesn't. Fine, whatever.

• Tinkered with mod_status and mod_info for a while (never used them before.) Checked output, same as your example.

• Became curious: Why does debug load while non-debug doesn't? Tried to run (on cmd) as user, as administrator and as system (used by service). Result: Loads as user, loads as admin, doesn't load as system (!)

• checked for "access denied" with ProcessMonitor (Sysinternals): no errors (with system account).

• created standard user, set service to use it: Loads!

• Access SVN repository (from client): Non-responsive, endless retries, finally timeout. Noted that Apache crashes (and restarts) on every request.

• Started over: Clean Apache, clean mod-directory. Now even mod_.dav_svn.so doesn't load (not just mod_authz_svn as before). Did not begin to drop-in files, but extended PATH step-by-step to include apache/bin, apache/modules, mod/bin, mod/bin/dep. Loads, but crashes again with every request.

• Changed to ApacheLounge build of Apache. Loads.

• Access SVN from client: No crashes!

That's what I did and where I'm now. I need a break to sort all this out, repeat the steps to make sure I didn't leave anything out.

A big apology for having both of you run around in circles all the time when the problem must have been my particular system after all! My best guess right now is that security software running on the system (that I can't disable or remove) must have messed with the system account. The rest were incompatibilities between different builds of DLLs after all, but the root cause must have been my system.

Thank you for sticking with it! I really do appreciate it!

Regards

[nono303]

Wonderful if you were able to catch the root cause of the problem! Windows Rights management & security software rarely works well with httpd ;)
This allowed me to provide clarifications on dependency management and basic configuration so, mutual benefits.
Don’t hesitate to write me a brief summary of your final resolution once you’ll have a clear idea of what happen on your system; I’ll add a note on the readme for those who encounter the same issue pattern.

[Jan-E]

Changed to ApacheLounge build of Apache. Loads.

Interesting. There must be an essential difference between the Apachelounge distro and the ApacheHaus distro.

[nono303]

Interesting. There must be an essential difference between the Apachelounge distro and the ApacheHaus distro.

Surely ;)

After doing my test #6 (comment) for the main httpd distributions (Apache Lounge, Apache Haus, WampServer, XAMPP, BitNami WAMP), all where successful with or without expat binary provided in package...