connection: Do not busy-loop if a message exceeds the buffer size

If the length of a message exceeds the maximum length of the buffer, the
buffer size will reach its maximum value and stay there forever, with no
message ever being successfully processed.  Since libwayland uses
level-triggered epoll, this will cause the compositor to loop forever
and consume CPU time.  In libwayland 1.22 and below, there was an
explicit check that caused messages exceeding 4096 bytes to result in an
EOVERFLOW error, preventing the loop.  However, this check was removed
between d074d5290263 ("connection: Dynamically resize connection buffers").

To prevent this problem, always limit the size of messages to 4096 bytes.
Since the default and minimum buffer size is 4096 bytes, this ensures
that a single message will always fit in the buffer.  It would be
possible to allow larger messages if the buffer size was larger, but the
maximum size of a message should not depend on the buffer size chosen by
the compositor.

Rejecting messages that exceed 4092 bytes seems to have the advantage of
reserving 4 bits, not 3, in the size field for future use.  However,
message sizes in the range [0x0, 0x7] are invalid, so one can obtain a
fourth bit by negating the meaning of bit 12 if bits 0 through 11
(inclusive) are 0.  Allowing 4096-byte messages provides the far more
important advantage that regressions compared to 1.22 are impossible
and regressions compared to 1.23 are extremely unlikely.  The only case
where a regression is possible is:

- The receiving side is using libwayland 1.23.
- The sending side is either using libwayland 1.23 or is not using
  libwayland.
- The sender sends a message exceeding 4096 bytes.
- If the sender of the large message is the client, the server has
  increased the buffer size from the default value.

This combination is considered extremely unlikely, as libwayland 1.22
and below would disconnect upon receiving such a large message.
4096-byte messages, however, have always worked, so there was no reason
to avoid sending them.

Fixes: d074d5290263 ("connection: Dynamically resize connection buffers").
Fixes: #494
Signed-off-by: Demi Marie Obenour <demi@invisiblethingslab.com>
(cherry picked from commit adf84614ca6189fa4efc522408ffbbc4b27ae497)

1 files changed, 8 insertions, 0 deletions

diff --git a/src/wayland-util.h b/src/wayland-util.h
index 4540f04..98c72fd 100644
--- a/src/wayland-util.h
+++ b/src/wayland-util.h
@@ -91,6 +91,14 @@ extern "C" {
 struct wl_object;
 
 /**
+ * The maximum size of a protocol message.
+ *
+ * If a message size exceeds this value, the connection will be dropped.
+ * Servers will send an invalid_method error before disconnecting.
+ */
+#define WL_MAX_MESSAGE_SIZE 4096
+
+/**
  * Protocol message signature
  *
  * A wl_message describes the signature of an actual protocol message, such as a