[astkwcz]

User defined system prompt for summary task

Related work:

This PR is stacked on #437 :

• Feat/custom system prompts #437 - configurable system prompts for summary and translation (admin defined)
• this PR - additional configurable system prompt for summary (user defined), it was removed from Feat/custom system prompts #437

Feel free to check #437, it describes the whole work for system prompts for different task types. It also provide a description of User defined system prompt for summary and presents UI look. Hierarchy is as follows:

• user defined system prompt (Complexity and Format options are ignored)
• admin defined system prompt (Complexity and Format options are appended)
• default system prompt

@edward-ly asked to move the user-defined system prompt part out of #437 and suggested to add user-defined system prompt part in separate PR with justification.

Justification for user defined system prompts

I agree with @edward-ly that it adds complexity for the user. The user has to know a little about how system prompts work for LLM and how to write instructions. On the other hand, this functionality is available after clicking Show advanced options so we can split users into two group:

• base options - for everyday users, nothing to configure (by the user) here
• advanced options - for a little bit more advanced users, who are able to write their own instructions for LLM

Use cases and why admin prompt is not enough

Admin defined system prompt for summary is a good option but it has to be general, because it has to work for everyone.

User-defined system prompt gives a user more control. This control might be required for different type of text to summarize. For example:

• meeting notes - someone gives a transcript as an input and asks to summarize a meeting and extract decisions, action items etc
• code - user might want to summarize some log file and call out errors/warnings
• contracts - user might tell LLM to summarize it and extract parties, duties and deadlines
• academic papers - user defined system prompt might include instruction to preserve citations, authors and key-findings

So those are field-specific summary tasks types. In general it is a summary, but based on the domain, the user might need different kind of information to be extracted.

What is more, with user defined system prompt for summary a user can set output format themselves for their needs:

• tabular data
• markdown format with ## headings
• company specific structure of a file
  • specific section names
  • specific language
  • some sections bullet points only
  • different sections with paragraph format
  • section that only includes people and their responsibilities
  • section with deadlines

Output formats can be mixed in summary if user defines that.

It all can be achieved with setting a good instruction for a model, but it is field/user specific instruction. Well written admin system prompt can be a good fit for most cases, but for specific kind of field/work user-defined system prompt might give better results if written correctly.

I would really appreciate your thoughts on these use-cases. If you prefer a different approach I am happy to rework this.

[edward-ly]

Hello, the way I see it, all of these use cases have existing workarounds or can be implemented in simpler ways.

advanced options - for a little bit more advanced users, who are able to write their own instructions for LLM

Why not just use the Assistant chat or the Generate text task instead, providing the "system prompt" and the user prompt in the same field? Separating them wouldn't provide much extra benefit and only confuses all users even more. Plus, overriding the system prompt means users could theoretically provide non-summary prompts in a summary task, which is clearly not the intention.

So those are field-specific summary tasks types. In general it is a summary, but based on the domain, the user might need different kind of information to be extracted.

An LLM might be able to extract these important parts already without the need for additional prompts. But even if it doesn't, you can still use the same workaround as before.

a user can set output format themselves for their needs

I would recommend adding these additional options to the Format input parameter instead. If users require any custom formatting, I would again point them to the same workaround as before.

[astkwcz]

Why not just use the Assistant chat or the Generate text task instead, providing the "system prompt" and the user prompt in the same field?

Considering workaround: it works as long as the input fits a single request. SummaryProvider chunks long input and sends multiple requests if needed, so each chunk stays within max_input_tokens_per_request limit. Chat with AI does not do that.

overriding the system prompt means users could theoretically provide non-summary prompts in a summary task, which is clearly not the intention.

I agree with that. It can be decided by the admin - I can add an admin toggle to enable/disable user-defined prompts if needed.

I would recommend adding these additional options to the Format input parameter instead.

Considering format: I think that adding tables, headings etc format is a good idea. My proposal was about formats that do not fit a fixed enum.

[github-actions]

Hello there,
Thank you so much for taking the time and effort to create a pull request to our Nextcloud project.

We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process.

Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6

Thank you for contributing to Nextcloud and we hope to hear from you soon!

(If you believe you should not receive this message, you can add yourself to the blocklist.)

[edward-ly]

Considering workaround: it works as long as the input fits a single request. SummaryProvider chunks long input and sends multiple requests if needed, so each chunk stays within max_input_tokens_per_request limit. Chat with AI does not do that.

Sure, chunking might not make the most sense for chat, but other tasks such as Improve text do already have chunking built in. You can run a separate task there to instruct the LLM to reformat text as well. What do you think?

I agree with that. It can be decided by the admin - I can add an admin toggle to enable/disable user-defined prompts if needed.

Considering format: I think that adding tables, headings etc format is a good idea. My proposal was about formats that do not fit a fixed enum.

Hm, I think you can use the Improve text task for these cases instead as well. That might be the easiest place to input your custom prompts or formatting. In the meantime, if you have ideas for additional formatting options to add to the enum, feel free to do so.

---

Also, now that #437 is merged, please rebase the branch, remove the extra commits from that PR, and resolve any additional conflicts too.

[astkwcz]

Improve text do already have chunking built in.
That might be the easiest place to input your custom prompts or formatting.

I wasn't aware of that. In that case I agree that it is valid workaround and we don't need user-system-prompt option.

In the meantime, if you have ideas for additional formatting options to add to the enum, feel free to do so.

Nice to hear that. For now I would not add formatting options just in case. If I run into a different formatting style that is general and I actually use I will open a follow-up PR for it.

Also, now that #437 is merged, please rebase the branch, remove the extra commits from that PR, and resolve any additional conflicts too.

Done ✅

Thanks for collaboration 😊

[edward-ly]

I wasn't aware of that. In that case I agree that it is valid workaround and we don't need user-system-prompt option.

Nice to hear that. For now I would not add formatting options just in case. If I run into a different formatting style that is general and I actually use I will open a follow-up PR for it.

Appreciate it, but going back to the original question, then: what other use cases does this PR provide that isn't already covered? I'm not sure if I see a need to merge this at all otherwise.

[astkwcz]

Appreciate it, but going back to the original question, then: what other use cases does this PR provide that isn't already covered? I'm not sure if I see a need to merge this at all otherwise.

Ok, I looked at the code of TextToTextImproveProvider.

Both providers send user's instructions together with a chunk
There is a slightly difference between Improve text and introduced user-system-prompt in SummaryProvider. In summary provider there is a reduction step, whereas Improvde text does not cover that.

Improve text:

• the input is split into chunks
• there is a single foreach ($chunks as $chunk) loop
• the result is the per-chunk LLM outputs simply combined

Summarize:

• the input is split into chunks
• there is a do while ($oldNumChunks > $newNumChunks) loop that combines the per-chunk results into one summary
• this summary is treated as the new input. The input (or summary) is again split into chunks.
• the mechanism is repeated until loop condition fails

So there is a difference in generated final output:

• Improve text returns chunked pieces joined together
• SummaryProvider reduces everything into one coherent summary because of do while loop. It might summarize better for long documents, transcripts etc

The concrete example that I came up with is the case that we have a long transcript of a meeting and want a summarized version of it with fixed sections, for example: 1. Meeting purpose, 2 Attendants. 3. Deadlines.
So those section names are a part of user instructions for LLM.

If the input is split into 3 chunks, the output will be

• Improve text: the sections repeated 3 times (once per chunk)
• Summary: one coherent summary with the sections appearing once

That's the case where a user-system-prompt for summary adds something over the Improve text workaround.
I know that I said we don't need user-system-prompt option, but looking at the reduction step I think that there's a still some advantage.
Nevertheless, if you think that is worth the extra option, keep it. If you decide it is not worth it, I am happy to close this PR. Your call.