wasi-nn: Add a new target for llama.cpp as a wasi-nn backend (#3709)

Minimum support:
- [x] accept (WasmEdge) customized model parameters. metadata.
- [x] Target [wasmedge-ggml examples](https://github.com/second-state/WasmEdge-WASINN-examples/tree/master/wasmedge-ggml)
  - [x] basic
  - [x] chatml
  - [x] gemma
  - [x] llama
  - [x] qwen

---

In the future, to support if required:
- [ ] Target [wasmedge-ggml examples](https://github.com/second-state/WasmEdge-WASINN-examples/tree/master/wasmedge-ggml)
  - [ ] command-r. (>70G memory requirement)
  - [ ] embedding. (embedding mode)
  - [ ] grammar. (use the grammar option to constrain the model to generate the JSON output)
  - [ ] llama-stream. (new APIS `compute_single`, `get_output_single`, `fini_single`)
  - [ ] llava. (image representation)
  - [ ] llava-base64-stream. (image representation)
  - [ ] multimodel. (image representation)
- [ ] Target [llamaedge](https://github.com/LlamaEdge/LlamaEdge)

Author: lum1n0us

build-scripts/config_common.cmake

@@ -442,7 +442,9 @@ if (WAMR_BUILD_WASI_NN EQUAL 1)
   message ("     WASI-NN enabled")
   add_definitions (-DWASM_ENABLE_WASI_NN=1)
   # Variant backends
-  if (NOT WAMR_BUILD_WASI_NN_TFLITE EQUAL 1 AND NOT WAMR_BUILD_WASI_NN_OPENVINO EQUAL 1)
+  if (NOT WAMR_BUILD_WASI_NN_TFLITE EQUAL 1 AND
+      NOT WAMR_BUILD_WASI_NN_OPENVINO EQUAL 1 AND
+      NOT WAMR_BUILD_WASI_NN_LLAMACPP EQUAL 1)
     message (FATAL_ERROR "   Need to select a backend for WASI-NN")
   endif ()
 
@@ -454,6 +456,10 @@ if (WAMR_BUILD_WASI_NN EQUAL 1)
     message ("     WASI-NN: backend openvino enabled")
     add_definitions (-DWASM_ENABLE_WASI_NN_OPENVINO)
   endif ()
+  if (WAMR_BUILD_WASI_NN_LLAMACPP EQUAL 1)
+    message ("     WASI-NN: backend llamacpp enabled")
+    add_definitions (-DWASM_ENABLE_WASI_NN_LLAMACPP)
+  endif ()
   # Variant devices
   if (WAMR_BUILD_WASI_NN_ENABLE_GPU EQUAL 1)
       message ("     WASI-NN: GPU enabled")

core/iwasm/libraries/wasi-nn/README.md

@@ -4,7 +4,7 @@
 
 ### Host
 
-Enable WASI-NN in the WAMR by spefiying it in the cmake building configuration as follows,
+Enable WASI-NN in the WAMR by specifying it in the cmake building configuration as follows,
 
 ```cmake
 set (WAMR_BUILD_WASI_NN  1)
@@ -17,14 +17,15 @@ $ cmake -DWAMR_BUILD_WASI_NN=1 <other options> ...
 ```
 
 > ![Caution]
-> If enable `WAMR_BUID_WASI_NN`, iwasm will link a shared WAMR library instead of a static one. Wasi-nn backends will be loaded dynamically at runtime. Users shall specify the path of the backend library and register it to the iwasm runtime with `--native-lib=<path of backend library>`. All shared libraries should be placed in the `LD_LIBRARY_PATH`.
+> Enabling WAMR_BUILD_WASI_NN will cause the IWASM to link to a shared WAMR library instead of a static one. The WASI-NN backends will then be loaded dynamically when the program is run. You must ensure that all shared libraries are included in the `LD_LIBRARY_PATH`.
 
 #### Compilation options
 
-- `WAMR_BUILD_WASI_NN`. enable wasi-nn support. can't work alone. need to identify a backend. Match legacy wasi-nn spec naming convention. use `wasi_nn` as import module names.
-- `WAMR_BUILD_WASI_EPHEMERAL_NN`. Match latest wasi-nn spec naming convention. use `wasi_ephemeral_nn` as import module names.
-- `WAMR_BUILD_WASI_NN_TFLITE`. identify the backend as TensorFlow Lite.
-- `WAMR_BUILD_WASI_NN_OPENVINO`. identify the backend as OpenVINO.
+- `WAMR_BUILD_WASI_NN`. This option enables support for WASI-NN. It cannot function independently and requires specifying a backend. It follows the original WASI-NN specification for naming conventions and uses wasi_nn for import module names.
+- `WAMR_BUILD_WASI_EPHEMERAL_NN`. This option adheres to the most recent WASI-NN specification for naming conventions and uses wasi_ephemeral_nn for import module names.
+- `WAMR_BUILD_WASI_NN_TFLITE`. This option designates TensorFlow Lite as the backend.
+- `WAMR_BUILD_WASI_NN_OPENVINO`. This option designates OpenVINO as the backend.
+- `WAMR_BUILD_WASI_NN_LLAMACPP`. This option designates Llama.cpp as the backend.
 
 ### Wasm
 
@@ -44,7 +45,7 @@ typedef enum { fp16 = 0, fp32, up8, ip32 } tensor_type;
 
 It is required to recompile the Wasm application if you want to switch between the two sets of functions.
 
-#### Openvino
+#### Openvino installation
 
 If you're planning to use OpenVINO backends, the first step is to install OpenVINO on your computer. To do this correctly, please follow the official installation guide which you can find at this link: https://docs.openvino.ai/2024/get-started/install-openvino/install-openvino-archive-linux.html.
 
@@ -162,17 +163,9 @@ Supported:
 
 ### Testing with WasmEdge-WASINN Examples
 
-To ensure everything is set up correctly, use the examples from [WasmEdge-WASINN-examples](https://github.com/second-state/WasmEdge-WASINN-examples/tree/master). These examples help verify that WASI-NN support in WAMR is functioning as expected.
+To make sure everything is configured properly, refer to the examples provided at [WasmEdge-WASINN-examples](https://github.com/second-state/WasmEdge-WASINN-examples/tree/master). These examples are useful for confirming that the WASI-NN support in WAMR is working correctly.
 
-> Note: The repository contains two types of examples. Some use the [standard wasi-nn](https://github.com/WebAssembly/wasi-nn), while others use [WasmEdge's version of wasi-nn](https://github.com/second-state/wasmedge-wasi-nn), which is enhanced to meet specific customer needs.
-
-The examples test the following machine learning backends:
-
-- OpenVINO
-- PyTorch
-- TensorFlow Lite
-
-Due to the different requirements of each backend, we'll use a Docker container for a hassle-free testing environment.
+Because each backend has its own set of requirements, we recommend using a Docker container to create a straightforward testing environment without complications.
 
 #### Prepare the execution environment
 
@@ -186,9 +179,20 @@ $ docker build -t wasi-nn-smoke:v1.0 -f ./core/iwasm/libraries/wasi-nn/test/Dock
 #### Execute
 
 ```bash
+$ pwd
+/workspaces/wasm-micro-runtime/
 $ docker run --rm wasi-nn-smoke:v1.0
 ```
 
-### Testing with bytecodealliance wasi-nn
+It should be noted that the qwen example is selected as the default one about the Llama.cpp backend because it uses a small model and is easy to run.
+
+```bash
+- openvino_mobile_image. PASS
+- openvino_mobile_raw. PASS
+- openvino_road_segmentation_adas. PASS
+- wasmedge_ggml_qwen. PASS
+```
+
+### Testing with bytecodealliance WASI-NN
 
 For another example, check out [classification-example](https://github.com/bytecodealliance/wasi-nn/tree/main/rust/examples/classification-example), which focuses on OpenVINO. You can run it using the same Docker container mentioned above.

core/iwasm/libraries/wasi-nn/cmake/Findcjson.cmake

@@ -0,0 +1,17 @@
+# Copyright (C) 2019 Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+include(FetchContent)
+
+set(CJSON_SOURCE_DIR "${WAMR_ROOT_DIR}/core/deps/cjson")
+
+FetchContent_Declare(
+  cjson
+  GIT_REPOSITORY https://github.com/DaveGamble/cJSON.git
+  GIT_TAG        v1.7.18
+  SOURCE_DIR     ${CJSON_SOURCE_DIR}
+)
+
+set(ENABLE_CJSON_TEST OFF CACHE INTERNAL "Turn off tests")
+set(ENABLE_CJSON_UNINSTALL OFF CACHE INTERNAL "Turn off uninstall to avoid targets conflict")
+FetchContent_MakeAvailable(cjson)

core/iwasm/libraries/wasi-nn/cmake/Findllamacpp.cmake

@@ -0,0 +1,18 @@
+# Copyright (C) 2019 Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+include(FetchContent)
+
+set(LLAMA_SOURCE_DIR "${WAMR_ROOT_DIR}/core/deps/llama.cpp")
+
+FetchContent_Declare(
+  llamacpp
+  GIT_REPOSITORY https://github.com/ggerganov/llama.cpp.git
+  GIT_TAG        b3573
+  SOURCE_DIR     ${LLAMA_SOURCE_DIR}
+)
+
+set(LLAMA_BUILD_TESTS OFF)
+set(LLAMA_BUILD_EXAMPLES OFF)
+set(LLAMA_BUILD_SERVER OFF)
+FetchContent_MakeAvailable(llamacpp)

core/iwasm/libraries/wasi-nn/cmake/Findtensorflow_lite.cmake

@@ -1,47 +1,25 @@
 # Copyright (C) 2019 Intel Corporation. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
-find_library(TENSORFLOW_LITE
-  NAMES tensorflow-lite
-  HINTS ${CMAKE_CURRENT_BINARY_DIR}/tensorflow-lite
-  NO_DEFAULT_PATHS
+include(FetchContent)
+
+set(TFLITE_SOURCE_DIR "${WAMR_ROOT_DIR}/core/deps/tensorflow-src")
+
+FetchContent_Declare(
+  tensorflow_lite
+  GIT_REPOSITORY https://github.com/tensorflow/tensorflow.git 
+  GIT_TAG        v2.12.0 
+  GIT_SHALLOW    ON
+  GIT_PROGRESS   ON
+  SOURCE_DIR     ${TFLITE_SOURCE_DIR}
+  SOURCE_SUBDIR  tensorflow/lite
 )
 
-if(NOT TENSORFLOW_LITE)
-  if(NOT EXISTS "${WAMR_ROOT_DIR}/core/deps/tensorflow-src")
-    execute_process(
-      COMMAND "${WAMR_ROOT_DIR}/core/deps/install_tensorflow.sh"
-      RESULT_VARIABLE TENSORFLOW_RESULT
-    )
-  else()
-    message("Tensorflow is already downloaded.")
-  endif()
-
-  set(TENSORFLOW_SOURCE_DIR "${WAMR_ROOT_DIR}/core/deps/tensorflow-src")
-
-  if(WAMR_BUILD_WASI_NN_ENABLE_GPU EQUAL 1)
-    # Tensorflow specific:
-    # * https://www.tensorflow.org/lite/guide/build_cmake#available_options_to_build_tensorflow_lite
-    set (TFLITE_ENABLE_GPU ON)
-  endif()
-
-  if (CMAKE_SIZEOF_VOID_P EQUAL 4)
-    set (TFLITE_ENABLE_XNNPACK OFF)
-  endif()
-
-  add_subdirectory(
-    "${TENSORFLOW_SOURCE_DIR}/tensorflow/lite"
-    "${CMAKE_CURRENT_BINARY_DIR}/tensorflow-lite"
-    EXCLUDE_FROM_ALL
-  )
-else ()
-  message(STATUS "TensorFlow Lite library found: ${TENSORFLOW_LITE}")
-  set(TENSORFLOW_SOURCE_DIR "${WAMR_ROOT_DIR}/core/deps/tensorflow-src")
+if(WAMR_BUILD_WASI_NN_ENABLE_GPU EQUAL 1)
+  set(TFLITE_ENABLE_GPU ON)
+endif()
+if (CMAKE_SIZEOF_VOID_P EQUAL 4)
+  set(TFLITE_ENABLE_XNNPACK OFF)
 endif()
 
-set(TENSORFLOW_LITE_INCLUDE_DIR "${TENSORFLOW_SOURCE_DIR}/tensorflow/lite")
-set(FLATBUFFER_INCLUDE_DIR "${CMAKE_CURRENT_BINARY_DIR}/flatbuffers/include")
-
-include_directories(${TENSORFLOW_SOURCE_DIR})
-include_directories(${FLATBUFFER_INCLUDE_DIR})
-link_directories(${CMAKE_CURRENT_BINARY_DIR}/tensorflow-lite)
+FetchContent_MakeAvailable(tensorflow_lite)

core/iwasm/libraries/wasi-nn/cmake/wasi_nn.cmake

@@ -3,27 +3,6 @@
 
 list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR})
 
-if(WAMR_BUILD_WASI_NN_TFLITE EQUAL 1)
-  # Find tensorflow-lite
-  find_package(tensorflow_lite REQUIRED)
-endif()
-
-if(WAMR_BUILD_WASI_NN_OPENVINO EQUAL 1)
-  if(NOT DEFINED ENV{OpenVINO_DIR})
-    message(FATAL_ERROR
-        "OpenVINO_DIR is not defined. "
-        "Please follow https://docs.openvino.ai/2024/get-started/install-openvino.html,"
-        "install openvino, and set environment variable OpenVINO_DIR."
-        "Like OpenVINO_DIR=/usr/lib/openvino-2023.2/ cmake ..."
-        "Or OpenVINO_DIR=/opt/intel/openvino/ cmake ..."
-    )
-  endif()
-
-  list(APPEND CMAKE_MODULE_PATH $ENV{OpenVINO_DIR})
-  # Find OpenVINO
-  find_package(OpenVINO REQUIRED COMPONENTS Runtime)
-endif()
-
 #
 # wasi-nn general
 set(WASI_NN_ROOT ${CMAKE_CURRENT_LIST_DIR}/..)
@@ -42,22 +21,46 @@ add_compile_definitions(
 #
 # - tflite
 if(WAMR_BUILD_WASI_NN_TFLITE EQUAL 1)
+  find_package(tensorflow_lite REQUIRED)
+
   add_library(
     wasi_nn_tflite
     SHARED
       ${WASI_NN_ROOT}/src/wasi_nn_tensorflowlite.cpp
   )
 
+  target_include_directories(
+    wasi_nn_tflite
+    PUBLIC
+      ${tensorflow_lite_SOURCE_DIR}
+  )
+
   target_link_libraries(
     wasi_nn_tflite
     PUBLIC
       libiwasm
       tensorflow-lite
   )
+
+  install(TARGETS wasi_nn_tflite DESTINATION lib)
 endif()
 
 # - openvino
 if(WAMR_BUILD_WASI_NN_OPENVINO EQUAL 1)
+  if(NOT DEFINED ENV{OpenVINO_DIR})
+    message(FATAL_ERROR
+        "OpenVINO_DIR is not defined. "
+        "Please follow https://docs.openvino.ai/2024/get-started/install-openvino.html,"
+        "install openvino, and set environment variable OpenVINO_DIR."
+        "Like OpenVINO_DIR=/usr/lib/openvino-2023.2/ cmake ..."
+        "Or OpenVINO_DIR=/opt/intel/openvino/ cmake ..."
+    )
+  endif()
+
+  list(APPEND CMAKE_MODULE_PATH $ENV{OpenVINO_DIR})
+  # Find OpenVINO
+  find_package(OpenVINO REQUIRED COMPONENTS Runtime)
+
   add_library(
     wasi_nn_openvino
     SHARED
@@ -71,4 +74,37 @@ if(WAMR_BUILD_WASI_NN_OPENVINO EQUAL 1)
       openvino::runtime
       openvino::runtime::c
   )
-endif()
+
+  install(TARGETS wasi_nn_openvino DESTINATION lib)
+endif()
+
+# - llamacpp
+
+if(WAMR_BUILD_WASI_NN_LLAMACPP EQUAL 1)
+  find_package(cjson REQUIRED)
+  find_package(llamacpp REQUIRED)
+
+  add_library(
+    wasi_nn_llamacpp
+    SHARED
+      ${WASI_NN_ROOT}/src/wasi_nn_llamacpp.c
+  )
+
+  target_include_directories(
+    wasi_nn_llamacpp
+    PUBLIC
+      ${cjson_SOURCE_DIR}
+  )
+
+  target_link_libraries(
+    wasi_nn_llamacpp
+    PUBLIC
+      libiwasm
+      cjson
+      common
+      ggml
+      llama
+  )
+
+  install(TARGETS wasi_nn_llamacpp DESTINATION lib)
+endif()

core/iwasm/libraries/wasi-nn/include/wasi_nn_types.h

@@ -43,6 +43,11 @@ typedef enum {
     security,
     // The operation failed for an unspecified reason.
     unknown,
+    // for WasmEdge-wasi-nn
+    end_of_sequence = 100,  // End of Sequence Found.
+    context_full = 101,     // Context Full.
+    prompt_tool_long = 102, // Prompt Too Long.
+    model_not_found = 103,  // Model Not Found.
 } wasi_nn_error;
 
 /**
@@ -140,6 +145,9 @@ typedef uint32_t graph_execution_context;
 typedef wasi_nn_error (*LOAD)(void *, graph_builder_array *, graph_encoding,
                               execution_target, graph *);
 typedef wasi_nn_error (*LOAD_BY_NAME)(void *, const char *, uint32_t, graph *);
+typedef wasi_nn_error (*LOAD_BY_NAME_WITH_CONFIG)(void *, const char *,
+                                                  uint32_t, void *, uint32_t,
+                                                  graph *);
 typedef wasi_nn_error (*INIT_EXECUTION_CONTEXT)(void *, graph,
                                                 graph_execution_context *);
 typedef wasi_nn_error (*SET_INPUT)(void *, graph_execution_context, uint32_t,
@@ -154,6 +162,7 @@ typedef wasi_nn_error (*BACKEND_DEINITIALIZE)(void *);
 typedef struct {
     LOAD load;
     LOAD_BY_NAME load_by_name;
+    LOAD_BY_NAME_WITH_CONFIG load_by_name_with_config;
     INIT_EXECUTION_CONTEXT init_execution_context;
     SET_INPUT set_input;
     COMPUTE compute;