Skip to content

Fix a mismatch between AVX2 and scalar FDCT paths - #40

Open
Shnatsel wants to merge 1 commit into
vstroebel:mainfrom
Shnatsel:fix-avx2-fdct-mismatch
Open

Shnatsel wants to merge 1 commit into
vstroebel:mainfrom
Shnatsel:fix-avx2-fdct-mismatch

Conversation

@Shnatsel

Copy link
Copy Markdown
Contributor

The AVX2 code was using _mm256_set1_epi32 but then interpreted the result as a vector of i16 instead of i32. This caused the values to become [2, 0, 2, 0, ...] instead of [2, 2, 2, 2, ...] as intended and done in the scalar path.

This PR adds a test demonstrating the issue, then fixes it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant