fix(imessage): gate split-send coalescing on imsg balloon metadata with back-compat (#90858)

Gate iMessage same-sender DM split-send coalescing on imsg's structural
`balloon_bundle_id` URL-balloon marker (openclaw/imsg#137) instead of timing/
text-shape inference, with a session capability latch and a back-compat path:

- URL-balloon marker present -> merge (precise split-send).
- Build known to emit balloon metadata (session latch) -> keep non-marker
  buckets separate (the precision win).
- Build that never emits balloon metadata (older imsg) -> preserve the legacy
  unconditional merge, so split-send users do not regress to two turns.

Never merges more than shipped main already did. Verified live end-to-end: the
patched gateway, watching a real chat.db via an imsg #137 build, merged a real
iPhone-sent `Dump <url>` split-send into one turn. Client-side removal once imsg
coalesces upstream is tracked in #91243 (openclaw/imsg#141).

Closes #90795
This commit is contained in:
Omar Shahine
2026-06-07 19:14:13 -07:00
committed by GitHub
parent f2530de832
commit 9caff5f873
8 changed files with 327 additions and 25 deletions

View File

@@ -654,14 +654,14 @@ When a user types a command and a URL together — e.g. `Dump https://example.co
The two rows arrive at OpenClaw ~0.8-2.0 s apart on most setups. Without coalescing, the agent receives the command alone on turn 1, replies (often "send me the URL"), and only sees the URL on turn 2 — at which point the command context is already lost. This is Apple's send pipeline, not anything OpenClaw or `imsg` introduces.
`channels.imessage.coalesceSameSenderDms` opts a DM into merging consecutive same-sender rows into a single agent turn. Group chats continue to dispatch per-message so multi-user turn structure is preserved.
`channels.imessage.coalesceSameSenderDms` opts a DM into buffering consecutive same-sender rows. When `imsg` exposes the structural URL-preview marker `balloon_bundle_id: "com.apple.messages.URLBalloonProvider"` on one of the source rows, OpenClaw merges only that real split-send and keeps any other buffered rows as separate turns. On older `imsg` builds that emit no balloon metadata at all, OpenClaw cannot tell a split-send from separate sends, so it falls back to merging the bucket. That preserves the pre-metadata behavior rather than regressing `Dump <url>` split-sends into two turns. Group chats continue to dispatch per-message so multi-user turn structure is preserved.
<Tabs>
<Tab title="When to enable">
Enable when:
- You ship skills that expect `command + payload` in one message (dump, paste, save, queue, etc.).
- Your users paste URLs, images, or long content alongside commands.
- Your users paste URLs alongside commands.
- You can accept the added DM turn latency (see below).
Leave disabled when:
@@ -702,7 +702,8 @@ The two rows arrive at OpenClaw ~0.8-2.0 s apart on most setups. Without coalesc
</Tab>
<Tab title="Trade-offs">
- **Added latency for DM messages.** With the flag on, every DM (including standalone control commands and single-text follow-ups) waits up to the debounce window before dispatching, in case a payload row is coming. Group-chat messages keep instant dispatch.
- **Precise merging needs current `imsg` payload metadata.** When the URL row includes `balloon_bundle_id`, only that real split-send merges and other buffered rows stay separate. On older `imsg` builds that expose no balloon metadata, OpenClaw falls back to merging the buffered bucket so `Dump <url>` split-sends are not regressed into two turns (interim back-compat, removed once `imsg` coalesces split-sends upstream).
- **Added latency for DM messages.** With the flag on, every DM (including standalone control commands and single-text follow-ups) waits up to the debounce window before dispatching, in case a URL-preview row is coming. Group-chat messages keep instant dispatch.
- **Merged output is bounded.** Merged text caps at 4000 chars with an explicit `…[truncated]` marker; attachments cap at 20; source entries cap at 10 (first-plus-latest retained beyond that). Every source GUID is tracked in `coalescedMessageGuids` for downstream telemetry.
- **DM-only.** Group chats fall through to per-message dispatch so the bot stays responsive when multiple people are typing.
- **Opt-in, per-channel.** Other channels (Telegram, WhatsApp, Slack, …) are unaffected. Legacy BlueBubbles configs that set `channels.bluebubbles.coalesceSameSenderDms` should migrate that value to `channels.imessage.coalesceSameSenderDms`.
@@ -712,15 +713,17 @@ The two rows arrive at OpenClaw ~0.8-2.0 s apart on most setups. Without coalesc
### Scenarios and what the agent sees
| User composes | `chat.db` produces | Flag off (default) | Flag on + 2500 ms window |
| ------------------------------------------------------------------ | --------------------- | --------------------------------------- | ----------------------------------------------------------------------- |
| `Dump https://example.com` (one send) | 2 rows ~1 s apart | Two agent turns: "Dump" alone, then URL | One turn: merged text `Dump https://example.com` |
| `Save this 📎image.jpg caption` (attachment + text) | 2 rows | Two turns (attachment dropped on merge) | One turn: text + image preserved |
| `/status` (standalone command) | 1 row | Instant dispatch | **Wait up to window, then dispatch** |
| URL pasted alone | 1 row | Instant dispatch | Instant dispatch (only one entry in bucket) |
| Text + URL sent as two deliberate separate messages, minutes apart | 2 rows outside window | Two turns | Two turns (window expires between them) |
| Rapid flood (>10 small DMs inside window) | N rows | N turns | One turn, bounded output (first + latest, text/attachment caps applied) |
| Two people typing in a group chat | N rows from M senders | M+ turns (one per sender bucket) | M+ turns — group chats are not coalesced |
The "Flag on" column shows behavior on an `imsg` build that emits `balloon_bundle_id`. On older `imsg` builds that emit no balloon metadata at all, the rows below marked "Two turns" / "N turns" instead fall back to a legacy merge (one turn): OpenClaw cannot structurally tell a split-send from separate sends, so it preserves the pre-metadata merge. Precise separation activates once the build emits balloon metadata.
| User composes | `chat.db` produces | Flag off (default) | Flag on + window (imsg emits balloon metadata) |
| ------------------------------------------------------------------ | ----------------------------------- | --------------------------------------- | ------------------------------------------------ |
| `Dump https://example.com` (one send) | 2 rows ~1 s apart | Two agent turns: "Dump" alone, then URL | One turn: merged text `Dump https://example.com` |
| `Save this 📎image.jpg caption` (attachment + text) | 2 rows without URL balloon metadata | Two turns | Two turns (legacy merge on metadata-less builds) |
| `/status` (standalone command) | 1 row | Instant dispatch | **Wait up to window, then dispatch** |
| URL pasted alone | 1 row | Instant dispatch | Wait up to window, then dispatch |
| Text + URL sent as two deliberate separate messages, minutes apart | 2 rows outside window | Two turns | Two turns (window expires between them) |
| Rapid flood (>10 small DMs inside window) | N rows without URL balloon metadata | N turns | N turns (legacy merge on metadata-less builds) |
| Two people typing in a group chat | N rows from M senders | M+ turns (one per sender bucket) | M+ turns — group chats are not coalesced |
## Catching up after gateway downtime

View File

@@ -1053,6 +1053,7 @@ describe("iMessage monitor last-route updates", () => {
id: 78,
guid: "LIVE-GUID-78",
text: "https://example.com",
balloon_bundle_id: "com.apple.messages.URLBalloonProvider",
created_at: "2026-05-22T15:30:01.000Z",
},
]) {
@@ -1105,4 +1106,154 @@ describe("iMessage monitor last-route updates", () => {
expect((await loadIMessageCatchupCursor("default"))?.lastSeenRowid).toBe(78);
});
});
it("legacy-merges coalesce buckets when imsg emits no balloon metadata (older builds)", async () => {
// Back-compat: older imsg builds emit no balloon_bundle_id, so a Dump + URL
// split-send arrives as two fieldless rows. We cannot structurally tell that
// apart from separate sends, so we preserve the pre-metadata merge rather
// than regress split-send users to two turns. Removed once imsg coalesces
// upstream (openclaw/imsg#141, tracked by #91243).
debouncerControl.holdEntries = true;
let onNotification: ((message: { method: string; params: unknown }) => void) | undefined;
const client = {
request: vi.fn(async (method: string) => {
if (method === "watch.subscribe") {
return { subscription: 1 };
}
throw new Error(`unexpected imsg method ${method}`);
}),
waitForClose: vi.fn(async () => {
for (const row of [
{ id: 91, guid: "LIVE-GUID-91", text: "Dump", created_at: "2026-05-22T15:30:00.000Z" },
{
id: 92,
guid: "LIVE-GUID-92",
text: "https://example.com",
created_at: "2026-05-22T15:30:01.000Z",
},
]) {
onNotification?.({
method: "message",
params: {
message: {
...row,
chat_id: 123,
sender: "+15550001111",
is_from_me: false,
is_group: false,
},
},
});
}
await vi.waitFor(() => {
expect(debouncerControl.flush).toBeDefined();
});
await debouncerControl.flush?.();
await Promise.resolve();
}),
stop: vi.fn(async () => {}),
};
createIMessageRpcClientMock.mockImplementation(async (params) => {
if (!params?.onNotification) {
throw new Error("expected iMessage notification handler");
}
onNotification = params.onNotification;
return client as never;
});
await monitorIMessageProvider({
config: {
channels: {
imessage: {
coalesceSameSenderDms: true,
dmPolicy: "allowlist",
allowFrom: ["+15550001111"],
sendReadReceipts: false,
},
},
messages: { inbound: { debounceMs: 2500 } },
session: { mainKey: "main" },
} as never,
runtime: { error: vi.fn(), exit: vi.fn(), log: vi.fn() },
});
expect(dispatchInboundMessageMock).toHaveBeenCalledTimes(1);
const mergedBody = dispatchInboundMessageMock.mock.calls[0]?.[0].ctx.Body ?? "";
expect(mergedBody).toContain("Dump");
expect(mergedBody).toContain("https://example.com");
});
it("merges coalesce buckets when imsg marks the URL balloon row structurally", async () => {
debouncerControl.holdEntries = true;
let onNotification: ((message: { method: string; params: unknown }) => void) | undefined;
const client = {
request: vi.fn(async (method: string) => {
if (method === "watch.subscribe") {
return { subscription: 1 };
}
throw new Error(`unexpected imsg method ${method}`);
}),
waitForClose: vi.fn(async () => {
for (const row of [
{ id: 93, guid: "LIVE-GUID-93", text: "Dump", created_at: "2026-05-22T15:30:00.000Z" },
{
id: 94,
guid: "LIVE-GUID-94",
text: "https://example.com",
balloon_bundle_id: "com.apple.messages.URLBalloonProvider",
created_at: "2026-05-22T15:30:01.000Z",
},
]) {
onNotification?.({
method: "message",
params: {
message: {
...row,
chat_id: 123,
sender: "+15550001111",
is_from_me: false,
is_group: false,
},
},
});
}
await vi.waitFor(() => {
expect(debouncerControl.flush).toBeDefined();
});
await debouncerControl.flush?.();
await Promise.resolve();
}),
stop: vi.fn(async () => {}),
};
createIMessageRpcClientMock.mockImplementation(async (params) => {
if (!params?.onNotification) {
throw new Error("expected iMessage notification handler");
}
onNotification = params.onNotification;
return client as never;
});
await monitorIMessageProvider({
config: {
channels: {
imessage: {
coalesceSameSenderDms: true,
dmPolicy: "allowlist",
allowFrom: ["+15550001111"],
sendReadReceipts: false,
},
},
messages: { inbound: { debounceMs: 2500 } },
session: { mainKey: "main" },
} as never,
runtime: { error: vi.fn(), exit: vi.fn(), log: vi.fn() },
});
expect(dispatchInboundMessageMock).toHaveBeenCalledTimes(1);
expect(dispatchInboundMessageMock.mock.calls[0]?.[0].ctx.Body).toContain(
"Dump https://example.com",
);
});
});

View File

@@ -2,9 +2,12 @@
import { describe, expect, it } from "vitest";
import {
combineIMessagePayloads,
hasIMessageUrlBalloonBundleID,
IMESSAGE_URL_BALLOON_BUNDLE_ID,
MAX_COALESCED_ATTACHMENTS,
MAX_COALESCED_ENTRIES,
MAX_COALESCED_TEXT_CHARS,
shouldCombineIMessagePayloadBucket,
} from "./coalesce.js";
import type { IMessagePayload } from "./types.js";
@@ -21,6 +24,52 @@ const makePayload = (overrides: Partial<IMessagePayload> = {}): IMessagePayload
});
describe("combineIMessagePayloads", () => {
it("recognizes URL balloon rows from imsg structural metadata", () => {
const text = makePayload({ text: "Dump" });
const balloon = makePayload({
text: "https://example.com/article",
balloon_bundle_id: IMESSAGE_URL_BALLOON_BUNDLE_ID,
});
expect(hasIMessageUrlBalloonBundleID(text)).toBe(false);
expect(hasIMessageUrlBalloonBundleID(balloon)).toBe(true);
// A real URL split-send merges regardless of the session capability latch.
expect(shouldCombineIMessagePayloadBucket([text, balloon], false)).toBe(true);
expect(shouldCombineIMessagePayloadBucket([text, balloon], true)).toBe(true);
});
it("falls back to a legacy merge when the build has never emitted balloon metadata (older imsg)", () => {
// Older imsg builds emit no balloon_bundle_id at all. We cannot tell a URL
// split-send from separate sends, so we preserve the pre-metadata merge
// rather than regress split-send users to two turns. Back-compat path,
// removed once imsg coalesces upstream (openclaw/imsg#141, tracked by #91243).
const text = makePayload({ text: "Dump" });
const url = makePayload({ text: "https://example.com/article" });
expect(shouldCombineIMessagePayloadBucket([text, url], false)).toBe(true);
});
it("keeps a plain bucket separate once the build is known to emit balloon metadata", () => {
// Capability latch is true (a prior row this session carried metadata), so a
// plain bucket with no URL marker is genuinely not a split-send. imsg omits
// the field for plain rows, so this case is indistinguishable per-bucket and
// depends on the session-level signal.
const a = makePayload({ text: "first" });
const b = makePayload({ text: "second" });
expect(shouldCombineIMessagePayloadBucket([a, b], true)).toBe(false);
});
it("keeps a bucket separate when imsg exposes balloon metadata in the bucket but no URL marker", () => {
// New imsg surfaced balloon metadata in this very bucket, proving this build
// emits the field, but the bucket is not a URL split-send. Keep separate even
// if the latch had not flipped yet.
const text = makePayload({ text: "hi" });
const nonUrlBalloon = makePayload({
text: "tap to vote",
balloon_bundle_id: "com.apple.messages.MSMessageExtensionBalloonPlugin",
});
expect(shouldCombineIMessagePayloadBucket([text, nonUrlBalloon], false)).toBe(false);
});
it("throws on empty input", () => {
expect(() => combineIMessagePayloads([])).toThrow(
"combineIMessagePayloads: cannot combine empty payloads",
@@ -44,6 +93,7 @@ describe("combineIMessagePayloads", () => {
const balloon = makePayload({
id: 42,
text: "https://example.com/article",
balloon_bundle_id: IMESSAGE_URL_BALLOON_BUNDLE_ID,
guid: "row-2",
created_at: "2025-01-01T00:00:01.500Z",
});

View File

@@ -16,6 +16,60 @@ import type { IMessagePayload } from "./types.js";
export const MAX_COALESCED_TEXT_CHARS = 4000;
export const MAX_COALESCED_ATTACHMENTS = 20;
export const MAX_COALESCED_ENTRIES = 10;
export const IMESSAGE_URL_BALLOON_BUNDLE_ID = "com.apple.messages.URLBalloonProvider";
export function hasIMessageUrlBalloonBundleID(payload: IMessagePayload): boolean {
return payload.balloon_bundle_id === IMESSAGE_URL_BALLOON_BUNDLE_ID;
}
// imsg only emits `balloon_bundle_id` for rows that actually carry a balloon
// (the nil case is omitted on the wire), so a present, non-empty value is the
// signal that this build exposes balloon metadata at all.
export function hasIMessageBalloonMetadata(payload: IMessagePayload): boolean {
return typeof payload.balloon_bundle_id === "string" && payload.balloon_bundle_id.length > 0;
}
/**
* Decide whether a debounced same-sender bucket should merge into one turn.
*
* `buildEmitsBalloonMetadata` is a session-level capability latch: once any
* inbound row from this imsg build has carried balloon metadata, absence of a
* URL marker is meaningful (the row genuinely is not a URL split-send), so we
* can keep ordinary buffered DMs separate. It must be session-scoped, not
* per-bucket: imsg omits `balloon_bundle_id` on the wire for non-balloon rows,
* so a bucket of plain text rows looks identical on old and new builds.
*/
export function shouldCombineIMessagePayloadBucket(
payloads: readonly IMessagePayload[],
buildEmitsBalloonMetadata: boolean,
): boolean {
// Precise path: a real Apple URL-preview split-send carries the URL-balloon
// marker on the preview row — merge it into one turn.
if (payloads.some(hasIMessageUrlBalloonBundleID)) {
return true;
}
// Metadata-capable build (observed earlier this session or in this bucket):
// the missing URL marker is trustworthy, so keep ordinary buffered DMs as
// separate turns. This is the precision the structural gate exists for.
if (buildEmitsBalloonMetadata || payloads.some(hasIMessageBalloonMetadata)) {
return false;
}
// Back-compat (remove once imsg coalesces split-sends upstream — see
// openclaw/imsg#141, tracked by #91243): a build that has never emitted any
// balloon metadata cannot structurally tell a `Dump <url>` split-send from
// separate sends. Preserve the pre-metadata merge so split-send users do not
// regress to two turns on a released imsg that lacks the field.
//
// This never merges more than the shipped behavior already did: with
// `coalesceSameSenderDms` enabled, `main` debounces every same-sender DM and
// merges each multi-entry bucket unconditionally. So an unlatched session
// (old build, or a metadata-capable build before its first balloon row) is
// identical to today, not a new regression. Flushing these buckets instead
// would re-break old-imsg split-sends — the very case this guards. Fully
// closing the pre-latch window needs an imsg-advertised capability flag, which
// is part of the upstream #141 work.
return true;
}
export type CoalescedIMessagePayload = IMessagePayload & {
/**

View File

@@ -66,7 +66,11 @@ import { normalizeIMessageHandle } from "../targets.js";
import { attachIMessageMonitorAbortHandler } from "./abort-handler.js";
import { runIMessageCatchup } from "./catchup-bridge.js";
import { advanceIMessageCatchupCursor, resolveCatchupConfig } from "./catchup.js";
import { combineIMessagePayloads } from "./coalesce.js";
import {
combineIMessagePayloads,
hasIMessageBalloonMetadata,
shouldCombineIMessagePayloadBucket,
} from "./coalesce.js";
import { repairIMessageConversationAnchor } from "./conversation-repair.js";
import { createIMessageEchoCachingSend, deliverReplies } from "./deliver.js";
import { resolveIMessageDmHistoryContext, resolveIMessageDmHistoryLimit } from "./dm-history.js";
@@ -359,6 +363,12 @@ export async function monitorIMessageProvider(opts: MonitorIMessageOpts = {}): P
const debounceMsOverride =
coalesceSameSenderDms && !hasExplicitInboundDebounce ? 2500 : undefined;
// Session capability latch: flips true once any inbound row from this imsg
// build carries balloon metadata. The coalesce flush gate needs a build-level
// (not per-bucket) signal because imsg omits `balloon_bundle_id` for plain
// rows, so a bucket of plain text looks identical on old and new builds.
let imsgEmitsBalloonMetadata = false;
const { debouncer: inboundDebouncer } = createChannelInboundDebouncer<{
message: IMessagePayload;
}>({
@@ -376,12 +386,10 @@ export async function monitorIMessageProvider(opts: MonitorIMessageOpts = {}): P
? `chat:${msg.chat_id}`
: (msg.chat_guid ?? msg.chat_identifier ?? "unknown");
// With coalesceSameSenderDms enabled, DMs key on chat:sender so two
// distinct user sends — `Dump` followed by a pasted URL that Apple
// delivers as a separate row — fall into the same bucket and merge
// into one agent turn. Group chats fall through to the legacy key so
// shouldDebounce can route them to the instant-dispatch path and
// preserve multi-user turn structure.
// With coalesceSameSenderDms enabled, DMs key on chat:sender so Apple's
// split text row and URL-balloon row land in the same bucket. The flush
// path still requires imsg's structural balloon metadata before merging.
// Group chats keep the legacy key to preserve multi-user turn structure.
if (coalesceSameSenderDms && msg.is_group !== true) {
return `imessage:${accountInfo.accountId}:dm:${conversationId}:${sender}`;
}
@@ -398,11 +406,10 @@ export async function monitorIMessageProvider(opts: MonitorIMessageOpts = {}): P
return false;
}
// With coalesceSameSenderDms enabled, debounce DM messages aggressively
// (text, media, control commands) so split-sends — `Dump <URL>`,
// `Save 📎image caption`, and rapid floods — merge into one agent
// turn. Group chats keep instant dispatch so the bot stays responsive
// when multiple people are typing.
// Hold opt-in DMs long enough for a following URL-balloon row to arrive.
// The flush gate (shouldCombineIMessagePayloadBucket) decides merge vs.
// separate: it merges precisely on imsg's balloon marker, and falls back
// to a legacy merge only when the build emits no balloon metadata at all.
if (coalesceSameSenderDms) {
return msg.is_group !== true;
}
@@ -425,7 +432,15 @@ export async function monitorIMessageProvider(opts: MonitorIMessageOpts = {}): P
return;
}
const combined = combineIMessagePayloads(entries.map((e) => e.message));
const messages = entries.map((e) => e.message);
if (!shouldCombineIMessagePayloadBucket(messages, imsgEmitsBalloonMetadata)) {
for (const message of messages) {
await handleMessageNow(message);
}
return;
}
const combined = combineIMessagePayloads(messages);
if (shouldLogVerbose()) {
const text = combined.text ?? "";
const preview = text.slice(0, 50);
@@ -1038,6 +1053,11 @@ export async function monitorIMessageProvider(opts: MonitorIMessageOpts = {}): P
runtime.error?.(`imessage: dropping malformed RPC message payload (keys=${shape})`);
return;
}
// Latch build capability from any row that carries balloon metadata so the
// coalesce flush gate can trust a missing URL marker on later plain buckets.
if (!imsgEmitsBalloonMetadata && hasIMessageBalloonMetadata(message)) {
imsgEmitsBalloonMetadata = true;
}
if (
watchStartupRowidWatermark !== null &&
typeof message.id === "number" &&

View File

@@ -68,6 +68,28 @@ describe("parseIMessageNotification", () => {
expect(parsed?.reacted_to_guid).toBe("target-guid");
});
it("preserves imsg balloon bundle metadata when present", () => {
const parsed = parseIMessageNotification({
message: {
id: 1,
guid: "link-preview-guid",
chat_id: 2,
sender: "+10000000000",
is_from_me: false,
text: "https://example.com/article",
balloon_bundle_id: "com.apple.messages.URLBalloonProvider",
attachments: null,
chat_identifier: null,
chat_guid: null,
chat_name: null,
participants: null,
is_group: false,
},
});
expect(parsed?.balloon_bundle_id).toBe("com.apple.messages.URLBalloonProvider");
});
it("accepts iMessage attachment transfer_name and uti metadata", () => {
const parsed = parseIMessageNotification({
message: {

View File

@@ -66,6 +66,7 @@ export function parseIMessageNotification(raw: unknown): IMessagePayload | null
!isOptionalNumber(message.chat_id) ||
!isOptionalString(message.sender) ||
!isOptionalString(message.destination_caller_id) ||
!isOptionalString(message.balloon_bundle_id) ||
!isOptionalBoolean(message.is_from_me) ||
!isOptionalString(message.text) ||
!isOptionalStringOrNumber(message.reply_to_id) ||

View File

@@ -17,6 +17,7 @@ export type IMessagePayload = {
chat_id?: number | null;
sender?: string | null;
destination_caller_id?: string | null;
balloon_bundle_id?: string | null;
is_from_me?: boolean | null;
text?: string | null;
reply_to_id?: number | string | null;