Any guide on implementing and building custom operator

[Kwangyeon Gill (davidgill97)]

I'm trying to implement and build custom operator for onnx to import .dll or .so in onnxruntime session, but the documentation is not that helpful. Is there any practical guide on it?

[Advait Kawale (Advait251206)]

Hi Kwangyeon,

Yes. ONNX Runtime has a supported custom-op API, and custom operators can be compiled into a shared library (.dll on Windows or .so on Linux) and loaded when creating a session. You generally do not need to rebuild the entire ONNX Runtime just to develop and load a custom operator.

I’d recommend starting with the official guide and its shared-library example:

• [ONNX Runtime: Custom operators](https://onnxruntime.ai/docs/reference/operators/add-custom-op.html)
• [ONNX Runtime Extensions: Add operators](https://onnxruntime.ai/docs/extensions/add-op.html) — includes an option to generate a C++ template from a model containing a custom operator.

A practical workflow is:

1. Define the operator contract. Decide its ONNX domain, operator name, input/output types, shapes, and attributes. The model must use the same domain and operator name that your custom-op library registers.
2. Implement the kernel in C++. For a first CPU operator, use the C++ custom-op helper API shown in the official examples. This is simpler than implementing the lower-level OrtCustomOp callbacks yourself.
3. Build a shared library. Compile the operator against the ONNX Runtime headers and export the required RegisterCustomOps entry point. The registration function adds your custom-op domain to the supplied session options.
4. Load the library when creating the session. In C++, use SessionOptions::RegisterCustomOpsLibrary(...) (or the corresponding versioned API available in your ORT release) before constructing the session.
5. Test the ONNX model and library together. Confirm that the model’s custom node uses the exact domain/name registered by the library, and that the library matches your target OS, architecture, and ONNX Runtime API version.

For a quick proof of concept, implement a simple CPU operator first, such as element-wise addition, and run a tiny ONNX model that invokes it. Once that works, add your real operation and any required attributes or dynamic-shape handling.

One important distinction: if you want the operator to become a built-in ONNX Runtime operator rather than a separately loaded custom op, that is a different contribution path involving the runtime’s operator schema, kernel implementation, and tests. For application-specific functionality, a custom-op shared library is usually the more direct approach.

If you share whether you are targeting Windows or Linux, and whether your operator should run on CPU or GPU, the build and registration steps can be narrowed down for your setup.

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206

[jinwoo (dev-jinwoohong)]

For a concrete DLL/SO example, look at ORT's own custom_op_library test fixture. It includes the kernels, RegisterCustomOps, export definitions, and a matching custom_op_test.onnx.

The CMake target builds a shared library. Use matching ORT releases for the headers/build and the Python runtime. After configuring an ORT source build with unit-test targets enabled, you can build that target directly:

cmake --build <configured-ORT-build-dir> --config Release --target custom_op_library

Then load the generated library before creating the session:

import onnxruntime as ort

opts = ort.SessionOptions()
opts.register_custom_ops_library("/absolute/path/libcustom_op_library.so")
session = ort.InferenceSession(
    "custom_op_test.onnx", opts, providers=["CPUExecutionProvider"]
)

Use custom_op_library.dll on Windows. The registration implementation also shows two details that are easy to miss when adapting it: initializing the C++ API from the host's OrtApiBase, and keeping the custom-op domains alive after RegisterCustomOps returns. This example uses the ORT source build.