Skip to content

Commit 04c9057

Browse files
waleedlatif1claude
waleedlatif1
and
claude
authored
fix(kb): disable connectors after repeated sync failures (#4046)
* fix(kb): improve error logging when connector token resolution fails The generic "Failed to obtain access token" error hid the actual root cause. Now logs credentialId, userId, authMode, and provider to help diagnose token refresh failures in trigger.dev. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(kb): disable connectors after 10 consecutive sync failures Connectors that fail 10 times in a row are set to 'disabled' status, stopping the cron from scheduling further syncs. The UI shows an alert triangle with a reconnect banner. Users can re-enable via the play button or by reconnecting their account, which resets failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(kb): disable sync button for disabled connectors, use amber badge variant Sync button should be disabled when connector is in disabled state to guide users toward reconnecting first. Badge variant changed from red to amber to match the warning banner styling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(kb): address PR review comments for disabled connector feature - Use `=== undefined` instead of falsy check for nextSyncAt to preserve explicit null (manual sync only) when syncIntervalMinutes is 0 - Gate Reconnect button on serviceId/providerId so it only renders for OAuth connectors; show appropriate copy for API key connectors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(kb): move resolveAccessToken inside try/catch for circuit-breaker coverage Token resolution failures (e.g. revoked OAuth tokens) were thrown before the try/catch block, bypassing consecutiveFailures tracking entirely. Also removes dead `if (refreshed)` guards at mid-sync refresh sites since resolveAccessToken now always returns a string or throws. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(kb): remove dead interval branch when re-enabling connector When `updates.nextSyncAt === undefined`, syncIntervalMinutes was not in the request, so `parsed.data.syncIntervalMinutes` is always undefined. Simplify to just schedule an immediate sync — the sync engine sets the proper nextSyncAt based on the connector's DB interval after completion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 650487c commit 04c9057

File tree

4 files changed

+105
-26
lines changed

4 files changed

+105
-26
lines changed

‎apps/sim/app/api/knowledge/[id]/connectors/[connectorId]/route.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -222,6 +222,13 @@ export async function PATCH(request: NextRequest, { params }: RouteParams) {
222222
}
223223
if (parsed.data.status !== undefined) {
224224
updates.status = parsed.data.status
225+
if (parsed.data.status === 'active') {
226+
updates.consecutiveFailures = 0
227+
updates.lastSyncError = null
228+
if (updates.nextSyncAt === undefined) {
229+
updates.nextSyncAt = new Date()
230+
}
231+
}
225232
}
226233

227234
await db

‎apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/connectors-section/connectors-section.tsx‎

Lines changed: 62 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { createLogger } from '@sim/logger'
55
import { format, formatDistanceToNow, isPast } from 'date-fns'
66
import {
77
AlertCircle,
8+
AlertTriangle,
89
CheckCircle2,
910
ChevronDown,
1011
Loader2,
@@ -66,6 +67,7 @@ const STATUS_CONFIG = {
6667
syncing: { label: 'Syncing', variant: 'amber' as const },
6768
error: { label: 'Error', variant: 'red' as const },
6869
paused: { label: 'Paused', variant: 'gray' as const },
70+
disabled: { label: 'Disabled', variant: 'amber' as const },
6971
} as const
7072

7173
export function ConnectorsSection({
@@ -159,7 +161,10 @@ export function ConnectorsSection({
159161
knowledgeBaseId,
160162
connectorId: connector.id,
161163
updates: {
162-
status: connector.status === 'paused' ? 'active' : 'paused',
164+
status:
165+
connector.status === 'paused' || connector.status === 'disabled'
166+
? 'active'
167+
: 'paused',
163168
},
164169
},
165170
{
@@ -352,7 +357,12 @@ function ConnectorCard({
352357
<div className='rounded-lg border border-[var(--border-1)]'>
353358
<div className='flex items-center justify-between px-3 py-2.5'>
354359
<div className='flex items-center gap-2.5'>
355-
{Icon && <Icon className='h-5 w-5 flex-shrink-0' />}
360+
<div className='relative flex-shrink-0'>
361+
{Icon && <Icon className='h-5 w-5' />}
362+
{connector.status === 'disabled' && (
363+
<AlertTriangle className='-right-1 -top-1 absolute h-3 w-3 text-amber-500' />
364+
)}
365+
</div>
356366
<div className='flex flex-col gap-0.5'>
357367
<div className='flex items-center gap-2'>
358368
<span className='flex items-center gap-1.5 font-medium text-[var(--text-primary)] text-small'>
@@ -407,7 +417,12 @@ function ConnectorCard({
407417
variant='ghost'
408418
className='h-7 w-7 p-0'
409419
onClick={onSync}
410-
disabled={connector.status === 'syncing' || isSyncPending || syncCooldown}
420+
disabled={
421+
connector.status === 'syncing' ||
422+
connector.status === 'disabled' ||
423+
isSyncPending ||
424+
syncCooldown
425+
}
411426
>
412427
<RefreshCw
413428
className={cn(
@@ -441,15 +456,17 @@ function ConnectorCard({
441456
>
442457
{isUpdating ? (
443458
<Loader2 className='h-3.5 w-3.5 animate-spin' />
444-
) : connector.status === 'paused' ? (
459+
) : connector.status === 'paused' || connector.status === 'disabled' ? (
445460
<Play className='h-3.5 w-3.5' />
446461
) : (
447462
<Pause className='h-3.5 w-3.5' />
448463
)}
449464
</Button>
450465
</Tooltip.Trigger>
451466
<Tooltip.Content>
452-
{connector.status === 'paused' ? 'Resume' : 'Pause'}
467+
{connector.status === 'paused' || connector.status === 'disabled'
468+
? 'Resume'
469+
: 'Pause'}
453470
</Tooltip.Content>
454471
</Tooltip.Root>
455472

@@ -481,7 +498,46 @@ function ConnectorCard({
481498
</div>
482499
</div>
483500

484-
{missingScopes.length > 0 && (
501+
{connector.status === 'disabled' && (
502+
<div className='border-[var(--border-1)] border-t px-3 py-2'>
503+
<div className='flex flex-col gap-1 rounded-sm border border-amber-200 bg-amber-50 px-2 py-1.5 dark:border-amber-900 dark:bg-amber-950'>
504+
<div className='flex items-center gap-1.5 font-medium text-amber-800 text-caption dark:text-amber-200'>
505+
<AlertTriangle className='h-3 w-3 flex-shrink-0' />
506+
Connector disabled after repeated sync failures
507+
</div>
508+
<p className='text-amber-700 text-micro dark:text-amber-300'>
509+
Syncing has been paused due to {connector.consecutiveFailures} consecutive failures.
510+
{serviceId
511+
? ' Reconnect your account to resume syncing.'
512+
: ' Use the resume button to re-enable syncing.'}
513+
</p>
514+
{canEdit && serviceId && providerId && (
515+
<Button
516+
variant='active'
517+
onClick={() => {
518+
if (connector.credentialId) {
519+
writeOAuthReturnContext({
520+
origin: 'kb-connectors',
521+
knowledgeBaseId,
522+
displayName: connectorDef?.name ?? connector.connectorType,
523+
providerId: providerId!,
524+
preCount: credentials?.length ?? 0,
525+
workspaceId,
526+
requestedAt: Date.now(),
527+
})
528+
}
529+
setShowOAuthModal(true)
530+
}}
531+
className='w-full px-2 py-1 font-medium text-caption'
532+
>
533+
Reconnect
534+
</Button>
535+
)}
536+
</div>
537+
</div>
538+
)}
539+
540+
{missingScopes.length > 0 && connector.status !== 'disabled' && (
485541
<div className='border-[var(--border-1)] border-t px-3 py-2'>
486542
<div className='flex flex-col gap-1 rounded-sm border bg-[var(--surface-2)] px-2 py-1.5'>
487543
<div className='flex items-center font-medium text-caption'>

‎apps/sim/hooks/queries/kb/connectors.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ export interface ConnectorData {
1212
sourceConfig: Record<string, unknown>
1313
syncMode: string
1414
syncIntervalMinutes: number
15-
status: 'active' | 'paused' | 'syncing' | 'error'
15+
status: 'active' | 'paused' | 'syncing' | 'error' | 'disabled'
1616
lastSyncAt: string | null
1717
lastSyncError: string | null
1818
lastSyncDocCount: number | null

‎apps/sim/lib/knowledge/connectors/sync-engine.ts‎

Lines changed: 35 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ const MAX_PAGES = 500
4646
const MAX_SAFE_TITLE_LENGTH = 200
4747
const STALE_PROCESSING_MINUTES = 45
4848
const RETRY_WINDOW_DAYS = 7
49+
const MAX_CONSECUTIVE_FAILURES = 10
4950

5051
/** Sanitizes a document title for use in S3 storage keys. */
5152
function sanitizeStorageTitle(title: string): string {
@@ -230,7 +231,7 @@ async function resolveAccessToken(
230231
connector: { credentialId: string | null; encryptedApiKey: string | null },
231232
connectorConfig: { auth: ConnectorAuthConfig },
232233
userId: string
233-
): Promise<string | null> {
234+
): Promise<string> {
234235
if (connectorConfig.auth.mode === 'apiKey') {
235236
if (!connector.encryptedApiKey) {
236237
throw new Error('API key connector is missing encrypted API key')
@@ -243,11 +244,22 @@ async function resolveAccessToken(
243244
throw new Error('OAuth connector is missing credential ID')
244245
}
245246

246-
return refreshAccessTokenIfNeeded(
247-
connector.credentialId,
248-
userId,
249-
`sync-${connector.credentialId}`
250-
)
247+
const requestId = `sync-${connector.credentialId}`
248+
const token = await refreshAccessTokenIfNeeded(connector.credentialId, userId, requestId)
249+
250+
if (!token) {
251+
logger.error(`[${requestId}] refreshAccessTokenIfNeeded returned null`, {
252+
credentialId: connector.credentialId,
253+
userId,
254+
authMode: connectorConfig.auth.mode,
255+
authProvider: connectorConfig.auth.provider,
256+
})
257+
throw new Error(
258+
`Failed to obtain access token for credential ${connector.credentialId} (provider: ${connectorConfig.auth.provider})`
259+
)
260+
}
261+
262+
return token
251263
}
252264

253265
/**
@@ -305,12 +317,6 @@ export async function executeSync(
305317
const userId = kbRows[0].userId
306318
const sourceConfig = connector.sourceConfig as Record<string, unknown>
307319

308-
let accessToken = await resolveAccessToken(connector, connectorConfig, userId)
309-
310-
if (!accessToken) {
311-
throw new Error('Failed to obtain access token')
312-
}
313-
314320
const lockResult = await db
315321
.update(knowledgeConnector)
316322
.set({ status: 'syncing', updatedAt: new Date() })
@@ -341,6 +347,8 @@ export async function executeSync(
341347
let syncExitedCleanly = false
342348

343349
try {
350+
let accessToken = await resolveAccessToken(connector, connectorConfig, userId)
351+
344352
const externalDocs: ExternalDocument[] = []
345353
let cursor: string | undefined
346354
let hasMore = true
@@ -357,8 +365,7 @@ export async function executeSync(
357365

358366
for (let pageNum = 0; hasMore && pageNum < MAX_PAGES; pageNum++) {
359367
if (pageNum > 0 && connectorConfig.auth.mode === 'oauth') {
360-
const refreshed = await resolveAccessToken(connector, connectorConfig, userId)
361-
if (refreshed) accessToken = refreshed
368+
accessToken = await resolveAccessToken(connector, connectorConfig, userId)
362369
}
363370

364371
const page = await connectorConfig.listDocuments(
@@ -496,8 +503,7 @@ export async function executeSync(
496503

497504
if (deferredOps.length > 0) {
498505
if (connectorConfig.auth.mode === 'oauth') {
499-
const refreshed = await resolveAccessToken(connector, connectorConfig, userId)
500-
if (refreshed) accessToken = refreshed
506+
accessToken = await resolveAccessToken(connector, connectorConfig, userId)
501507
}
502508

503509
const hydrated = await Promise.allSettled(
@@ -789,15 +795,25 @@ export async function executeSync(
789795

790796
const now = new Date()
791797
const failures = (connector.consecutiveFailures ?? 0) + 1
798+
const disabled = failures >= MAX_CONSECUTIVE_FAILURES
792799
const backoffMinutes = Math.min(failures * 30, 1440)
793-
const nextSync = new Date(now.getTime() + backoffMinutes * 60 * 1000)
800+
const nextSync = disabled ? null : new Date(now.getTime() + backoffMinutes * 60 * 1000)
801+
802+
if (disabled) {
803+
logger.warn('Connector disabled after repeated failures', {
804+
connectorId,
805+
consecutiveFailures: failures,
806+
})
807+
}
794808

795809
await db
796810
.update(knowledgeConnector)
797811
.set({
798-
status: 'error',
812+
status: disabled ? 'disabled' : 'error',
799813
lastSyncAt: now,
800-
lastSyncError: errorMessage,
814+
lastSyncError: disabled
815+
? 'Connector disabled after repeated sync failures. Please reconnect.'
816+
: errorMessage,
801817
nextSyncAt: nextSync,
802818
consecutiveFailures: failures,
803819
updatedAt: now,

0 commit comments

Comments
 (0)