HTTPChunkedInputStream does not raise an error when the stream ends prematurely #5032
Description
Describe the bug
When receiving data with chunked transfer encoding, if the connection is closed by the peer without sending the terminating 0\r\n\r\n, or if a chunk is not sent completely, the response stream still returns the data as if it were delivered correctly. Furthermore, there is no API to detect whether the data was incomplete.
For example, suppose the server returns (\r\n are omitted for readability):
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Type: application/octet-stream
Transfer-Encoding: chunked
7
Foo
Suppose then the server then closes the connection. The stream is incomplete because the final zero-sized chunk is missing, and the first chunk does not have 7 bytes of data, only 3.
In this small example, the stream returns 0 bytes. For larger payloads, it may return some data. In both cases, however, there is no indication that the data is incomplete.
For reference, curl detects this situation and exits with code 18 (Partial file):
curl: (18) transfer closed with outstanding read data remaining
To Reproduce
This is a simple client I used to reproduce the issue:
#include <sstream>
#include <Poco/Net/HTTPClientSession.h>
#include <Poco/Net/HTTPRequest.h>
#include <Poco/Net/HTTPResponse.h>
#include <Poco/StreamCopier.h>
int main()
{
try
{
Poco::Net::HTTPClientSession session("127.0.0.1", 8123);
session.setKeepAlive(true);
Poco::Net::HTTPRequest req(Poco::Net::HTTPRequest::HTTP_GET, "/");
req.setVersion(Poco::Net::HTTPMessage::HTTP_1_1);
req.set("Connection", "Keep-Alive");
std::ostream & request_stream = session.sendRequest(req);
Poco::Net::HTTPResponse res;
std::istream & response_stream = session.receiveResponse(res);
std::stringstream buffer{};
Poco::StreamCopier::copyStream(response_stream, buffer);
std::string str = buffer.str();
std::cout << "Read " << str.size() << " bytes, EOF set to " << response_stream.eof() << ", ";
if (auto * exception = session.networkException())
std::cout << "session exception: " << exception->what() << "\n";
else
std::cout << "no exceptions\n";
}
catch (const std::exception & ex)
{
std::cerr << "Exception: " << ex.what() << "\n";
}
return EXIT_SUCCESS;
}
Running with incomplete data shows that some bytes are read from the stream without any errors, and EOF is set to 1.
A simple server example is included at the end of this report to make reproducing the issue easier.
Important note
This issue is reproducible only when the server closes the connection gracefully (with FIN, not RST).
If the server resets the connection with RST, the session.networkException() function raises an exception, which is sufficient to indicate that the data is incomplete.
However, many servers close connections with FIN during a restart, and some proxies even convert RST to FIN. As a result, it is not uncommon for a server to return an incomplete stream while still performing a graceful (FIN) socket close.
Expected behavior
session.networkException() should always rerun an exception when connection was closed before reading all the data including the terminating 0\r\n\r\n part.
Please add relevant environment information:
- Linux Fedora 41 and 42
- POCO Version 1.14.2
Additional context
This is the server I use to reproduce the scenario. It writes an incomplete stream of data and then waits. Stopping the server with Ctrl-C causes the OS to close the socket gracefully (with FIN), which results in the following message from the example in the To Reproduce section:
Read 10240 bytes, EOF set to 1, no exceptions
The server:
#include <errno.h>
#include <limits.h>
#include <signal.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/socket.h>
#define PORT 8123
#define B93 "============================================================================================="
#define KB "400\r\n" B93 B93 B93 B93 B93 B93 B93 B93 B93 B93 B93 "\n\r\n"
const char example[] = "HTTP/1.1 200 OK\r\n"
"Connection: Keep-Alive\r\n"
"Content-Type: application/octet-stream\r\n"
"Transfer-Encoding: chunked\r\n\r\n" KB KB KB KB KB KB KB KB KB KB
// "0\r\n\r\n" // <- end of stream marker is commented out
;
static ssize_t skip_bytes(int fd, size_t size)
{
char input_buffer[1024];
size_t total_bytes_read = 0;
while (total_bytes_read < size) {
size_t remaining_size = size - total_bytes_read;
size_t max_size = remaining_size > sizeof(input_buffer) ? sizeof(input_buffer) : remaining_size;
ssize_t bytes_read = recv(fd, input_buffer, max_size, MSG_WAITALL);
if (bytes_read == 0)
return bytes_read; // EOF
else if (bytes_read < 0) {
printf("recv: %s\n", strerror(errno));
return bytes_read;
}
total_bytes_read += bytes_read;
}
return total_bytes_read;
}
static ssize_t write_bytes(int fd, const void * buf, size_t size)
{
size_t total_bytes_written = 0;
while (total_bytes_written < size) {
const void * send_buf = (const char *)buf + total_bytes_written;
const size_t send_size = size - total_bytes_written;
int bytes_written = send(fd, send_buf, send_size, 0);
if (bytes_written < 0) {
printf("send: %s\n", strerror(errno));
return bytes_written;
}
total_bytes_written += bytes_written;
}
return total_bytes_written;
}
int main(void)
{
signal(SIGPIPE, SIG_IGN);
int server_fd = socket(AF_INET, SOCK_STREAM, 0);
if (server_fd == -1) {
printf("socket: %s\n", strerror(errno));
return EXIT_FAILURE;
}
int reuse_address = 1;
if (setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &reuse_address, sizeof(reuse_address)) == -1) {
printf("setsockopt: %s\n", strerror(errno));
return EXIT_FAILURE;
}
struct sockaddr_in server_addr = {0};
server_addr.sin_family = AF_INET;
server_addr.sin_addr.s_addr = INADDR_ANY;
server_addr.sin_port = htons(PORT);
if (bind(server_fd, (struct sockaddr *)&server_addr, sizeof(server_addr)) == -1) {
printf("bind: %s\n", strerror(errno));
return EXIT_FAILURE;
}
if (listen(server_fd, 1) == -1) {
printf("listen: %s\n", strerror(errno));
return EXIT_FAILURE;
}
printf("waiting connections on port %d\n", PORT);
while (1)
{
int client_socket = accept(server_fd, NULL, NULL);
if (client_socket == -1) {
printf("accept: %s\n", strerror(errno));
break;
}
// Read a bit of data to make sure that the client has sent anything
skip_bytes(client_socket, 10);
write_bytes(client_socket, example, sizeof(example) - 1);
printf("wrote the data, now press Ctrl-C to close\n");
skip_bytes(client_socket, INT_MAX);
}
}