feat: activation stable abi#1007
Conversation
Signed-off-by: David Holtz <david.richard.holtz@gmail.com>
|
/kernel-bot build activation |
|
Build request processed. Command: Dispatched (3):
|
|
/kernel-bot build activation |
|
Build request processed. Command: Dispatched (3):
|
| #define VLLM_DISPATCH_FLOATING_TYPES(TYPE, NAME, ...) \ | ||
| [&] { \ | ||
| switch (TYPE) { \ | ||
| VLLM_DISPATCH_CASE_FLOATING_TYPE(Float, float, __VA_ARGS__) \ | ||
| VLLM_DISPATCH_CASE_FLOATING_TYPE(Half, torch::headeronly::Half, \ | ||
| __VA_ARGS__) \ | ||
| VLLM_DISPATCH_CASE_FLOATING_TYPE(BFloat16, torch::headeronly::BFloat16, \ | ||
| __VA_ARGS__) \ | ||
| default: \ | ||
| STD_TORCH_CHECK(false, NAME " not implemented for the given dtype"); \ | ||
| } \ | ||
| }() |
There was a problem hiding this comment.
I think this removes float8?
We should sync this file against: https://github.com/vllm-project/vllm/blob/main/csrc/libtorch_stable/dispatch_utils.h
There was a problem hiding this comment.
ah yea good point it is much better to sync wth that upstream file, I originally slimmed it down because it wasn't clear to me that fp8 was used.
updated to sync with vllm in latest!
| ] | ||
|
|
||
| [torch.stable-abi] | ||
| cuda = "2.11" |
There was a problem hiding this comment.
ROCm/HIP?
Also, is there a reason that 2.9 cannot be targeted?
There was a problem hiding this comment.
rocm was originally left out because this kernel was only targeting cuda and metal, updated to also target rocm in latest.
and I think that the vllm dispatch header requires us to use 2.10 and later because of torch/headeronly/core/Dispatch.h
it seems like that files only exists in 2.10 and later
https://github.com/pytorch/pytorch/blob/v2.10.0/torch/headeronly/core/Dispatch.h
https://github.com/pytorch/pytorch/blob/v2.9.0/torch/headeronly/core/Dispatch.h
I think we can either unsync the dispatch header and target 2.9 or sync with vllm and target 2.10
|
/kernel-bot build activation |
|
Build request processed. Command: Dispatched (3):
|
|
/kernel-bot build activation |
|
Build request processed. Command: Dispatched (3):
|
2 participants
this PR adds the stable torch ABI to the activation kernel and also exercises the backend scoped stable abi feature that is implemented in this branch huggingface/kernels#682