[josephjclark]

This PR builds on top of #1161.

It adds batch processing to the new event processor. It basically works like this:

• Events can be arbitrarily queued. Just pass an option to the processor
• When an event is queued, we do the following:
  • Set a batch flag to listen for all incoming events
  • Save any batched events to an array
  • Look in the queue to see if there are any other events already queued. If so, add them straight into the batch
  • After 10ms, send the array of batched messages to the listener. A batch listener must be designed to handle an array of events, not just one event
  • If a different event comes through while batching, cancel the timeout, instantly send the batch, and start processing the new event

Fixes #790

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[josephjclark]

It doesn't look like the interval stuff is actually working. I'll either disable that option (do we really need it?) or try and get it fixed tomorrow

[josephjclark]

TODO need to add more integration level tests around batching. I've made the log handler robust to batch and non batch, which basically means that most of the tests are in non batch mode.

I need to either add a few explicit batch tests, or make batching default in all tests. For the sake of an easier life I think I'll just add a few new tests

[josephjclark]

from speaking with Stu: not to worried about the size of log payloads on the wire

raise an issue to take size into account later. Maybe track batch size or break batch on a large payload

Let's be a bit daring and set the default batch to 50. Easy to modify

[josephjclark]

When enabling batch logs by default on the worker, we should make that a breaking change (because not all lightning versions will be compatible)

Alternatively, negotiate support with lightning when joining the run channel, so that we ensure compatiblity

[josephjclark]

Persistent integration test fails in CI. Having a hard time figuring this out - seems OK locally. I am finding a strange exception getting thrown so looking into that...

[josephjclark]

Phew. Finally got this passing. A bit of docs then I can merge