Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support native windows HTTP client #671

Closed
dtretyakov opened this issue Oct 24, 2018 · 22 comments · Fixed by #3243
Closed

Support native windows HTTP client #671

dtretyakov opened this issue Oct 24, 2018 · 22 comments · Fixed by #3243

Comments

@dtretyakov
Copy link
Contributor

To create native windows applications could be useful to have native HTTP client which supports using Windows certificates store for SSL connections, proxy settings configuration on the system level, etc.

Microsoft recommends using WinHTTP for server-side applications and services. WinHTTP is used in .NET windows http client so it's a good candidate to use it on Windows. WinHTTP could work in sync and async mode, which is recommended to use and async calls could be wrapped by using Kotlin coroutines.

@dtretyakov
Copy link
Contributor Author

The sample kotlin native application which provides WinHTTP-based client is available here: https://github.com/dtretyakov/kotlin-winhttp

Currently it has the following problems:

  • To use suspend functions in kotlin native coroutines should be supported multi-threading working mode, since WinHTTP uses pool of worker threads and completion callback could be executed on any of them: (Support multi-threaded coroutines on Kotlin/Native Kotlin/kotlinx.coroutines#462)
  • WinHTTP headers are not included as Windows platform library in kotlin native, so it requires having cintorop library. I'm not sure where is a better place to have winhttp interop in the windows platform or in this repository.
  • In WinAPI most of calls returns string as UTF16 and I it looks like kotlin native library is missing helper functions to create Kotlin strings from ShortArrays and it requires constructing string like that:
val buffer = allocArray<ShortVar>(size)
// Call to fill in the buffer
String(CharArray(size) {
    buffer[it].toChar()
})

@cy6erGn0m
Copy link
Contributor

Kotlinx I/O supports string constructor String(ByteArray, Charset), perhaps I can support construction from any CPointer as well

@cy6erGn0m
Copy link
Contributor

I believe it's ok to have winhttp cinterop in the client

@cy6erGn0m
Copy link
Contributor

To get it work with coroutines a set of hacks need to be applied: currently coroutines only work in the main thread so you need to send signals from a callback thread. The other issue is that there is no main loop implemented on Windows so it's not clear who can listen to such signals.

@e5l e5l self-assigned this Oct 24, 2018
@e5l e5l added the feature label Oct 24, 2018
@cy6erGn0m
Copy link
Contributor

@dtretyakov Does winhttp support streaming for request and response body?

@dtretyakov
Copy link
Contributor Author

@cy6erGn0m, WinHTTP async read/write operations are based on request -> callback response style of invocation: https://msdn.microsoft.com/en-us/library/windows/desktop/aa383138%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396#art_winhttpheaders

So it looks like e.g. the response body could be consumed as a stream while invoking series of:

WinHttpQueryDataAvailable(...)
// -> WINHTTP_CALLBACK_STATUS_DATA_AVAILABLE callback
// allocate buffer
WinHttpReadData(...)
// -> WINHTTP_CALLBACK_STATUS_READ_COMPLETE callback
// read from the buffer

until WINHTTP_CALLBACK_STATUS_DATA_AVAILABLE returns 0 for statusInfo

@cy6erGn0m
Copy link
Contributor

Note that we are going to introduce curl client, see #777

@e5l
Copy link
Member

e5l commented Jan 11, 2019

Merged in master

@e5l e5l closed this as completed Jan 11, 2019
@msink
Copy link

msink commented Jan 12, 2019

Why was closed?
On Windows libcurl from msys2-mingw is not well integrated and hard to redistribute - good enought for prototyping, but not so good for production.
So client implementation based on winhttp or wininet still required.

@e5l e5l reopened this Jan 12, 2019
dtretyakov added a commit to dtretyakov/ktor that referenced this issue Feb 17, 2019
dtretyakov added a commit to dtretyakov/ktor that referenced this issue Feb 18, 2019
It allows to make streaming while writing request and response body.

Issue ktorio#671
@dtretyakov
Copy link
Contributor Author

@cy6erGn0m, @e5l I've added preliminary PR to support WinHTTP-based client: #957

dtretyakov added a commit to dtretyakov/ktor that referenced this issue Feb 22, 2019
dtretyakov added a commit to dtretyakov/ktor that referenced this issue Feb 22, 2019
It allows to make streaming while writing request and response body.

Issue ktorio#671
dtretyakov added a commit to dtretyakov/ktor that referenced this issue Feb 22, 2019
dtretyakov added a commit to dtretyakov/ktor that referenced this issue Feb 23, 2019
dtretyakov added a commit to dtretyakov/ktor that referenced this issue Feb 23, 2019
dtretyakov added a commit to dtretyakov/ktor that referenced this issue Feb 27, 2019
e5l pushed a commit that referenced this issue Mar 1, 2019
* Add windows platform for gradle modules

* Add WinHTTP support for HTTP client

Issue #671

* Use futures in WinHTTP-based client

It allows to make streaming while writing request and response body.

Issue #671

* Add synchronous working mode for WinHTTP-based client

Issue #671

* Extract response header processing

* Remove debug logging in WinHTTP-based asynchronous callbacks

* Allows to use HTTP/2.0 protocol in WinHTTP-based client

Issue #671

* Allows to specify security protocols in WinHTTP-based client

Issue #671

* Improve error handling for WinHTTP function calls

Issue #671
e5l pushed a commit that referenced this issue Mar 4, 2019
* Add windows platform for gradle modules

* Add WinHTTP support for HTTP client

Issue #671

* Use futures in WinHTTP-based client

It allows to make streaming while writing request and response body.

Issue #671

* Add synchronous working mode for WinHTTP-based client

Issue #671

* Extract response header processing

* Remove debug logging in WinHTTP-based asynchronous callbacks

* Allows to use HTTP/2.0 protocol in WinHTTP-based client

Issue #671

* Allows to specify security protocols in WinHTTP-based client

Issue #671

* Improve error handling for WinHTTP function calls

Issue #671
e5l pushed a commit that referenced this issue Mar 4, 2019
* Add windows platform for gradle modules

* Add WinHTTP support for HTTP client

Issue #671

* Use futures in WinHTTP-based client

It allows to make streaming while writing request and response body.

Issue #671

* Add synchronous working mode for WinHTTP-based client

Issue #671

* Extract response header processing

* Remove debug logging in WinHTTP-based asynchronous callbacks

* Allows to use HTTP/2.0 protocol in WinHTTP-based client

Issue #671

* Allows to specify security protocols in WinHTTP-based client

Issue #671

* Improve error handling for WinHTTP function calls

Issue #671
e5l pushed a commit that referenced this issue Mar 13, 2019
* Add windows platform for gradle modules

* Add WinHTTP support for HTTP client

Issue #671

* Use futures in WinHTTP-based client

It allows to make streaming while writing request and response body.

Issue #671

* Add synchronous working mode for WinHTTP-based client

Issue #671

* Extract response header processing

* Remove debug logging in WinHTTP-based asynchronous callbacks

* Allows to use HTTP/2.0 protocol in WinHTTP-based client

Issue #671

* Allows to specify security protocols in WinHTTP-based client

Issue #671

* Improve error handling for WinHTTP function calls

Issue #671
@Dominaezzz
Copy link

Is the WinHTTP client going to be released soon or has it been indefinitely put aside?

@PatrickZGW
Copy link

Just to clarify: There is currently no ktor client available for windows, correct?

@e5l
Copy link
Member

e5l commented Apr 20, 2020

CURL client engine supports Windows.

@PatrickZGW
Copy link

That's interesting, since if I include both required imports:

mingwX64().compilations["main"].defaultSourceSet { dependencies { implementation("io.ktor:ktor-client-core:$ktorVersion") implementation("io.ktor:ktor-client-curl:$ktorVersion") } }

I can still not import io.ktor.* Is there anything else I need to take into consideration?

@msink
Copy link

msink commented Apr 20, 2020

CURL client engine supports Windows

Does not support https:, only http: works.

@PatrickZGW
Copy link

@msink https works for me

@msink
Copy link

msink commented Apr 27, 2020

@PatrickLemke - it works in debug mode, but I failed to make distribution package.
Do you have any example how to distibute it?

@PatrickZGW
Copy link

@msink I haven't tried that yet. Are you referring to the difficulty of installing msys on the client system? I was thinking to use an installer like NSIS to install it manually.

@msink
Copy link

msink commented Apr 27, 2020

If I remember correctly - it works only if .exe was copied to msys2 bin/ directory, and runned from here. Something like as Git for Windows is distributed.

@PatrickZGW
Copy link

Well I guess you could also copy all the libraries you need into your own folder. The only thing that should matter is that mysys has access to all the libraries it needs.

@msink
Copy link

msink commented Apr 27, 2020

Not exactly - it looks for ssl certificates in directory relative to .exe - ../ssl , so .exe should be in subdirectory, for example bin/

@oleg-larshin
Copy link

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.

dtretyakov added a commit to dtretyakov/ktor that referenced this issue Nov 12, 2020
* Add windows platform for gradle modules

* Add WinHTTP support for HTTP client

Issue ktorio#671

* Use futures in WinHTTP-based client

It allows to make streaming while writing request and response body.

Issue ktorio#671

* Add synchronous working mode for WinHTTP-based client

Issue ktorio#671

* Extract response header processing

* Remove debug logging in WinHTTP-based asynchronous callbacks

* Allows to use HTTP/2.0 protocol in WinHTTP-based client

Issue ktorio#671

* Allows to specify security protocols in WinHTTP-based client

Issue ktorio#671

* Improve error handling for WinHTTP function calls

Issue ktorio#671
dtretyakov added a commit to dtretyakov/ktor that referenced this issue Nov 20, 2020
* Add windows platform for gradle modules

* Add WinHTTP support for HTTP client

Issue ktorio#671

* Use futures in WinHTTP-based client

It allows to make streaming while writing request and response body.

Issue ktorio#671

* Add synchronous working mode for WinHTTP-based client

Issue ktorio#671

* Extract response header processing

* Remove debug logging in WinHTTP-based asynchronous callbacks

* Allows to use HTTP/2.0 protocol in WinHTTP-based client

Issue ktorio#671

* Allows to specify security protocols in WinHTTP-based client

Issue ktorio#671

* Improve error handling for WinHTTP function calls

Issue ktorio#671
@e5l e5l closed this as completed in #3243 Nov 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants