Repository navigation
Capture compiled artifacts #1
Description
The typical onnxruntime flow is the following via llm-kvc.py (very simplified):
- Find the onnx model in the huggingface cache as presented via the CLI flags
- Run a compilation process, although it is more like transpilation+compilation: input is onnx model, output is a
.bundlewith an ELF in it. Uses et-glow, gcc, llvm, neuralizer, etc. - Run inference. Uses InferenceServer.
If you run the same one a second time, the process recognizes that the model already has been converted from onnx to .bundle, and tuns just the inference step. This takes about 1/10 the time.
The current nekko CLI does not recognize that there are intermediate artifacts to capture. We need to do this via the following:
- decide on a compiled artifacts cache directory default location
- have an option to the CLI to override that location
- mount the location into the container
- configure the
llm-kvc.pyscript to support controlling that location
cc @jerenkrantz
Activity
jerenkrantz commentedon Aug 22, 2025
LGTM!
1 & 2 is conceptually correct!
BTW, on 3, there is no separate inference server running. The Python ETGlow backend used by
llm-kvc.pyhandles the transpilation+compilation, loading to the card, and then executing the model. (This doesn't meaningfully change anything.)rvs commentedon Aug 23, 2025
Sounds reasonable + what @jerenkrantz said. Also, as he pointed out InferenceServer is a separate binary. Confusingly, InferenceServer family of CLI/daemons are more specialized than ONNX runtime, however, the same GenAI model (let's say llama3) can be run via both.
deitch commentedon Aug 24, 2025
There are two parts of this that confused me.
First, @jerenkrantz are you saying that the actual inference is handled by the same python functions? Or do you mean that the same library handles both compilation to bundle (1, 2) and inference, but does so using different functions in the library, called by
llm-kvc.py? Do you mind linking to the specific lines for each stage in llm-kvc.py?First and a half, how does
llm-kvc.pydo the compilation stage? Don't we need gcc+llvm for that? Or does it just call out to separate binaries?Second, @rvs if InferenceServer is not executing the compiled bundle, what is it for?
jerenkrantz commentedon Aug 24, 2025
The onnxruntime session is created here:
https://github.com/nekkoai/hacks/blob/04c6ddd5668c8638f1a4bb1775f7fddcfdf327ea/onnx/llm-kvc.py#L532
The Python code that drives the inference is invoked here:
https://github.com/nekkoai/hacks/blob/04c6ddd5668c8638f1a4bb1775f7fddcfdf327ea/onnx/llm-kvc.py#L543
There are two different code paths as to how the inference is driven:
llm_kvc_inferenceandllm_kvc_inference_iobindings.Two public blog posts from ET team:
https://www.esperanto.ai/blog/compiling-and-executing-small-language-models-with-onnxruntime-2/
https://www.esperanto.ai/blog/optimizing-small-language-models-with-onnxruntime-iobindings/
Lastly,
InferenceServeris not used with onnxruntime, but different inference workflows. I think for the purposes of cli, I think we will focus on onnxruntime and llama.cpp workflows as those are two leading approaches used by the external community.deitch commentedon Aug 24, 2025
Huh, those are barely 50-60 lines into
main(). If that is inference, where is compilation called?deitch commentedon Aug 24, 2025
That has become clear 😆
So what is
InferenceServerused for?deitch commentedon Sep 19, 2025
Solved via #14