UN-3332 [FIX] Use default profile for export validation when single-pass is enabled#1991
Open
pk-zipstack wants to merge 4 commits into
Open
UN-3332 [FIX] Use default profile for export validation when single-pass is enabled#1991pk-zipstack wants to merge 4 commits into
pk-zipstack wants to merge 4 commits into
Conversation
The Prompt Studio export validation filtered prompt outputs by prompt.profile_manager, but single-pass execution stores outputs against the tool's default profile (see OutputManagerHelper.handle_prompt_output_update) with is_single_pass_extract=True. When the default profile differed from the prompt-level profile FK, the validation lookup missed the rows produced by the single-pass run and incorrectly rejected the export with "Prompt Studio project without prompts cannot be exported." Match the validation lookup to the mode the prompts were actually run in: the tool default profile + is_single_pass_extract=True when single-pass is enabled, the prompt's profile_manager + False otherwise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Regression test for frame_export_json: - single-pass on: filter uses default profile + is_single_pass_extract=True - single-pass off: filter uses prompt.profile_manager + is_single_pass_extract=False - force_export=True bypasses the PromptStudioOutputManager lookup Follows the sys.modules-stub pattern from test_build_index_payload.py to avoid pulling in Django app loading; skips cleanly if imports drift. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Summary by CodeRabbit
WalkthroughExport validation in frame_export_json now selects an output_profile per prompt based on tool.single_pass_extraction_mode (single-pass => default LLM profile; otherwise the prompt's profile_manager), passes that profile and is_single_pass_extract to PromptStudioOutputManager filter when not force_export, and sources profile-dependent payload fields from the selected output_profile. Regression tests cover single-pass, non-single-pass, and force-export paths. ChangesProfile-aware output validation for single-pass extraction
sequenceDiagram
participant Client
participant PromptStudioRegistryHelper
participant ProfileManager
participant PromptStudioOutputManager
Client->>PromptStudioRegistryHelper: frame_export_json(...)
alt tool.single_pass_extraction_mode == True
PromptStudioRegistryHelper->>ProfileManager: get_default_llm_profile()
else
Note right of PromptStudioRegistryHelper: use prompt.profile_manager
end
PromptStudioRegistryHelper->>PromptStudioOutputManager: objects.filter(tool_id, prompt_id, output_profile, is_single_pass_extract)
PromptStudioOutputManager-->>PromptStudioRegistryHelper: filter result
PromptStudioRegistryHelper-->>Client: exported JSON payloads (populated from output_profile)
🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
| Filename | Overview |
|---|---|
| backend/prompt_studio/prompt_studio_registry_v2/prompt_studio_registry_helper.py | Introduces output_profile to correctly resolve the active profile per prompt based on single-pass mode; fixes both the validation filter and all per-prompt export fields. Non-single-pass path is unchanged: is_single_pass_extract defaults to False in the model so the new explicit filter matches existing rows. |
| backend/prompt_studio/prompt_studio_registry_v2/tests/test_frame_export_json.py | New regression tests covering single-pass, non-single-pass, and force_export branches; asserts filter kwargs and per-prompt export fields match the correct profile. Follows the established sys.modules-stub pattern used elsewhere in the test suite. |
| backend/prompt_studio/prompt_studio_registry_v2/tests/init.py | Empty init file to make the new tests/ directory a Python package. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[frame_export_json called] --> B{prompt active & not NOTES?}
B -- No --> SKIP[skip prompt]
B -- Yes --> C{prompt.profile_manager set?}
C -- No --> D[fallback: prompt.profile_manager = default_llm_profile]
C -- Yes --> E{single_pass_extraction_mode?}
D --> E
E -- Yes --> F[output_profile = default_llm_profile]
E -- No --> G[output_profile = prompt.profile_manager]
F --> H{force_export?}
G --> H
H -- Yes --> J[skip validation]
H -- No --> I[PromptStudioOutputManager.filter
profile_manager=output_profile
is_single_pass_extract=single_pass_mode]
I --> K{rows found?}
K -- No --> L[add to invalidated_outputs]
K -- Yes --> J
J --> M[Build per-prompt JSON
from output_profile
vector_db, llm, embedding,
x2text, chunk_size, etc.]
M --> N[outputs.append]
Reviews (3): Last reviewed commit: "UN-3332 [MISC] Tighten output_profile in..." | Re-trigger Greptile
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@backend/prompt_studio/prompt_studio_registry_v2/tests/test_frame_export_json.py`:
- Around line 142-153: The current module-level try/except sets _IMPORT_ERROR
and skips the whole test suite on any Exception; change the except clause to
only catch ImportError and ModuleNotFoundError (or ImportError subclass) and set
_IMPORT_ERROR in that case, but re-raise any other Exception so unexpected
errors in prompt_studio_registry_helper surface; adjust the handling of
PromptStudioRegistryHelper and _psrh_mod the same way and keep pytestmark using
_IMPORT_ERROR as the skip reason.
- Around line 67-134: The test mutates sys.modules via the helper functions
_install and _install_package (which assign directly into sys.modules), causing
cross-test pollution; update the tests to contain these overrides by saving and
restoring originals or using pytest's monkeypatch: replace direct assignments in
_install/_install_package with monkeypatch.setitem(sys.modules, name, mod) (or
wrap the calls in a fixture that records original sys.modules entries and
restores them in teardown), and ensure any inserted module keys are
removed/restored after the test so other tests do not see the stubbed modules.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4b5fc4f0-0fc1-4771-9bc4-d281a840e861
📒 Files selected for processing (3)
backend/prompt_studio/prompt_studio_registry_v2/prompt_studio_registry_helper.pybackend/prompt_studio/prompt_studio_registry_v2/tests/__init__.pybackend/prompt_studio/prompt_studio_registry_v2/tests/test_frame_export_json.py
…e-pass mode Addresses two review-bot findings on top of the initial validation fix: 1. Greptile: per-prompt export entries (vector_db / embedding / llm / x2text / chunk_size / chunk_overlap / retrieval_strategy / similarity_top_k / section / reindex) still read from prompt.profile_manager even when single-pass mode was on. Same root cause as the validation bug — if the user added a second profile and set it as default before running, the exported tool embedded the stale prompt-level profile's settings instead of the default profile that was actually executed. Lifts the output_profile resolution out of the validation block so it drives both the lookup and the per-prompt payload, eliminating the divergence. 2. CodeRabbit: the test module's import-time `except Exception` masked genuine helper bugs (NameError, TypeError at module load, etc.) behind a "skipped suite" — defeating regression coverage. Narrow to ImportError / ModuleNotFoundError / AttributeError so only true environment/dependency failures skip; everything else surfaces. Tests extended to assert per-prompt JSON content (llm, vector_db, embedding, etc.) matches the expected profile for each mode. Verified the new assertions correctly fail on pre-fix per-prompt logic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
@greptile-apps — addressed in 4ff2b3c. The per-prompt JSON divergence is the same root cause as UN-3332, so I folded it into this PR: lifted Tests extended in the same commit — Re. CodeRabbit's |
Contributor
|
Looks good — the fix is clean and complete. The A few observations on the final state:
The PR addresses the original validation bug, the follow-up per-prompt JSON divergence, and the test coverage gap. LGTM. Tip: You can customize Greptile's behavior for this repo with |
chandrasekharan-zipstack
approved these changes
May 27, 2026
Contributor
chandrasekharan-zipstack
left a comment
chandrasekharan-zipstack
left a comment
There was a problem hiding this comment.
LGTM, we might have to revisit the tests later
| if not prompt.profile_manager: | ||
| prompt.profile_manager = default_llm_profile | ||
|
|
||
| # Single-pass execution always runs prompts via the tool's |
Contributor
There was a problem hiding this comment.
@pk-zipstack make the comment more concise and generic
Contributor
Author
There was a problem hiding this comment.
Done in 356f795 — collapsed the 8-line comment down to two lines stating just the rule:
# Single-pass runs every prompt via the tool's default profile;
# otherwise each prompt uses its own profile_manager.
Contributor
Author
There was a problem hiding this comment.
Done in 356f795 — collapsed the 8-line comment down to two lines stating just the rule:
# Single-pass runs every prompt via the tool's default profile;
# otherwise each prompt uses its own profile_manager.Per review feedback — drop the implementation-history reasoning and keep just the factual rule: single-pass uses the default profile, everything else uses the prompt-level profile_manager. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Contributor
Test ResultsSummary
Runner Tests - Full Report
SDK1 Tests - Full Report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Why
PromptStudioOutputManagerrows against that default profile withis_single_pass_extract=True(seeOutputManagerHelper.handle_prompt_output_update).frame_export_json, however, filtered the validation lookup byprompt.profile_manager(the prompt-card FK, frozen at creation time) and built the per-prompt export JSON from the same FK.How
backend/prompt_studio/prompt_studio_registry_v2/prompt_studio_registry_helper.py: resolveoutput_profileonce per prompt — the tool default profile whensingle_pass_extraction_modeis on, otherwiseprompt.profile_manager. This single profile now drives both:PromptStudioOutputManagervalidation lookup (with a matchingis_single_pass_extractflag), andCan this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)
output_profileresolves toprompt.profile_manager, exactly as before (the only addition is an explicitis_single_pass_extract=Falseon the lookup, which matches how non-single-pass rows are stored).force_export=Truestill bypasses validation entirely.Database Migrations
Env Config
Relevant Docs
Related Issues or PRs
Dependencies Versions
Notes on Testing
backend/prompt_studio/prompt_studio_registry_v2/tests/test_frame_export_json.py(3 cases — single-pass, non-single-pass,force_export=True). They assert both the validation filter args and the per-prompt export JSON content per mode. Samesys.modules-stub pattern astest_build_index_payload.py(nopytest-djangodependency).is_single_pass_extract) and pass on the fix; all 7prompt_studiobackend tests pass:pytest prompt_studio/.Screenshots
N/A — backend validation/export logic change only.
Checklist
I have read and understood the Contribution Guidelines.
🤖 Generated with Claude Code