Task-Level Model Demo

Introduction

Each task in sleepKIT has a corresponding demo mode that allows you to run a task-level demonstration using the specified backend inference engine (e.g. PC or EVB). This is useful to showcase the model’s performance in real-time and to verify its accuracy in a real-world scenario. Similar to other modes, the demo can be invoked either via CLI or within sleepkit python package. At a high level, the demo mode performs the following actions based on the provided configuration parameters:

1. Load the configuration data (e.g. configuration.json)
2. Load the desired features (e.g. FS-W-A-5)
3. Load the trained model (e.g. model.keras)
4. Initialize inference engine backend (e.g. pc or evb)
5. Generate input data (e.g. x, y)
6. Perform inference on backend (e.g. model.predict)
7. Generate report (e.g. report.html)

---

Inference Backends

sleepKIT includes two built-in backend inference engines: PC and EVB. Additional backends can be easily added to the sleepKIT framework by creating a new backend class and registering it to the backend factory.

PC Backend

The PC backend is used to run the task-level demo on the local machine. This is useful for quick testing and debugging of the model.

1. Create / modify configuration file (e.g. stage-class-4.json)
2. Ensure “pc” is selected as the backend in configuration file.
3. Run demo sleepkit --mode demo --task stage --config ./configs/stage-class-4.json
4. HTML report will be saved to ${job_dir}/report.html

EVB Backend

The EVB backend is used to run the task-level demo on an Ambiq EVB. This is useful to showcase the model’s performance in real-time and to verify its accuracy in a real-world scenario.

1. Create / modify configuration file (e.g. stage-class-2.json)
2. Ensure “evb” is selected as the backend in configuration file.
3. Plug EVB into PC via two USB-C cables.
4. Build and flash firmware to EVB cd evb && make && make deploy
5. Run demo sleepkit --mode demo --task stage --config ./configs/stage-class-4.json
6. HTML report will be saved to ${job_dir}/report.html

Bring-Your-Own-Backend

Similar to datasets, tasks, and models, the demo mode can be customized to use your own backend inference engine. sleepKIT includes a backend factory (BackendFactory) that is used to create and run the backend engine.

How it Works

1. Create a Backend: Define a new backend class that inherits from the sk.InferenceBackend base class and implements the required methods. This skeleton must be implemented before use.

   Python example

   import sleepkit as sk
   import numpy.typing as npt
   
   class CustomBackend(sk.InferenceBackend):
       def __init__(self, params: sk.TaskParams) -> None:
           self.params = params
   
       def open(self):
           raise NotImplementedError("Implement the backend operation")
   
       def close(self):
           raise NotImplementedError("Implement the backend operation")
   
       def set_inputs(self, inputs: npt.NDArray):
           raise NotImplementedError("Implement the backend operation")
   
       def perform_inference(self):
           raise NotImplementedError("Implement the backend operation")
   
       def get_outputs(self) -> npt.NDArray:
           raise NotImplementedError("Implement the backend operation")

2. Register the Backend: Register the new backend with the sk.BackendFactory by calling the register method. This method takes the backend name and the backend class as arguments.

   Python example

   import sleepkit as sk
   sk.BackendFactory.register("custom", CustomBackend)

3. Use the Backend: The new backend can now be used by setting the backend flag in the demo configuration settings.

   Python example

   import sleepkit as sk
   
   from pathlib import Path
   
   params = sk.TaskParams.model_validate_json(Path("configuration.json").read_text())
   params.backend = "custom"
   
   task = sk.TaskFactory.get("detect")
   
   task.demo(params)

   OR by creating the backend directly:

   Python example

   import sleepkit as sk
   
   from pathlib import Path
   
   params = sk.TaskParams.model_validate_json(Path("configuration.json").read_text())
   params.backend = "custom"
   
   backend = sk.BackendFactory.get("custom")(params)

---

Usage

The following is an example of a task-level demo report for the sleep staging task.

CLI

Terminal

sleepkit -m demo -t stage -c ./configuration.json

Python

Python example

from pathlib import Path
import sleepkit as sk

task = sk.TaskFactory.get("stage")
params = sk.TaskParams.model_validate_json(Path("configuration.json").read_text())
task.demo(params)

---

Arguments

Please refer to TaskParams for the list of arguments that can be used with the demo command.

---