Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove incorrect usage of curls Multi API #185

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

flokaiser
Copy link

There are several issues introduced by #29

When running an application that is rarely restarted, that links to this sdk I encountered hangs inside of the select() call inside of http.c multiple times, that required a restart of the application.

Looking into the man pages of curl, it is quite obvious, that the usage of select is a bad idea here.

On one hand, the requests.fdset() call may return no filedescriptors, if there is currently nothing to wait for. (https://curl.se/libcurl/c/curl_multi_fdset.html)

when fdset() returns no filedescriptors, maxfd is set to -1. this leads to a select call, that has nfds set to 0 and a disabled timeout. -> this leads to an infinite select() until either the program is terminated or a signal is received.

On the other hand, if your application is running for some time, new sockets may have filedescriptors larger than FD_SETSIZE (1024), according to the docs (https://curl.se/libcurl/c/curl_multi_fdset.html), fdset does not return any filedescriptors in that case, which leads to the infinite select() again.

There are two possible solutions to this:

  1. waiting for this PR to be merged: 1.add curlpp::Multi::poll/wait 2.add option:TimeoutMs jpbarrette/curlpp#173
    • Use the poll() or wait() call from there
  2. revert to the Easy API again.
    • If I understand it correctly the only reason for using the Multi API was to be able to abort transfers inside of the ResponseCallback, but that was implemented incorrectly anyway. Currently the request is removed by requests.remove(request) inside of the ResponseCallback, which is forbidden by: 'It is fine to remove a handle at any time during a transfer, just not from within any libcurl callback function.' (https://curl.se/libcurl/c/curl_multi_remove_handle.html)

Due to these reasons I propose to revert back to the Easy API. To keep the possibility to abort a running request, CURL_WRITEFUNC_ERROR is returned at the corresponding locations inside of the ResponseCallback() function.

There are several issues introduced by minio#29

When running an application that is rarely restarted, that links to this
sdk I encountered hangs inside of the select() call inside of http.c
multiple times, that required a restart of the application.

Looking into the man pages of curl, it is quite obvious, that the usage of select
is a bad idea here.

On one hand, the requests.fdset() call may return no
filedescriptors, if there is currently nothing to wait for.
(https://curl.se/libcurl/c/curl_multi_fdset.html)

when fdset() returns no filedescriptors, maxfd is set to -1.
this leads to a select call, that has nfds set to 0 and a disabled
timeout. -> this leads to an infinite select() until either the program
is terminated or a signal is received.

On the other hand, if your application is running for some time, new
sockets may have filedescriptors larger than FD_SETSIZE (1024),
according to the docs (https://curl.se/libcurl/c/curl_multi_fdset.html),
fdset does not return any filedescriptors in that case, which leads to
the infinite select() again.

There are two possible solutions to this:
1. waiting for this PR to be merged: jpbarrette/curlpp#173
   - Use the poll() or wait() call from there
2. revert to the Easy API again.
   - If I understand it correctly the only reason for using the Multi
     API was to be able to abort transfers inside of the ResponseCallback,
     but that was implemented incorrectly anyway.
     Currently the request is removed by requests.remove(request) inside
     of the ResponseCallback, which is forbidden by:
       'It is fine to remove a handle at any time during a transfer, just
        not from within any libcurl callback function.'
        (https://curl.se/libcurl/c/curl_multi_remove_handle.html)

Due to these reasons I propose to revert back to the Easy API.
To keep the possibility to abort a running request, CURL_WRITEFUNC_ERROR
is returned at the corresponding locations inside of the
ResponseCallback() function.
Copy link
Member

@balamurugana balamurugana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flokaiser The requirement is described https://github.com/minio/minio-cpp/pull/29/files#r892559012. That's the reason curlpp::Multi is used. If it is curlpp issue, there is no issue using low level libcurl C APIs directly. Feel free to update the PR accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants