2.1 - Cache 的一般設計 L1 Cache 通常分為 I-Cache 和 D-Cache，為 private cache I-Cache 需要能夠在一個 cycle 內，讀取多條的指令 D-Cache 需要能在一個 cycle 內，處理多條 load/store 指令的訪問，因此需要使用 multi-port 的設計 Instruction fetch 的時候，會向 L1 Cache 嘗試讀取指令，因此 L1 Cache 也是 pipeline 的一部分 L1 Cache 必須保持跟 CPU 相近的速度，因此通常是透過 SRAM 實現的 L2 Cache 通常都是指令和資料一起 share 整個 cache，容量通常以 MB 為單位 現在大部份的 multi-core 都是 shared 同一個 L3 Cache L2 Cache 被訪問的頻率通常不是很高 (L1 Cache miss 才會訪問 L2 Cache)，因此不需要使用 multi-port 的設計 需要盡可能的提昇 L2 Cache 的命中率，因為如果 L2 Cache miss，且沒 L3 Cache 的話，就需要去訪問 DRAM 了，訪問時間會很長 2.1.1 - Cache 的組成方式 Set-associative cache： ...

1.2 - 普通處理器的流水線 1.2.2 - 流水線的劃分 Pipeline CPU，其 cycle time 由最長 cycle time 的 pipeline stage 所決定 因此，最好每個 pipeline stage 的 cycle time 都是差不多長的 解決各個 pipeline stage cycle time 不平衡的方法： 合： 將多個 pipeline stages 合併成一個 stage，例如： Fetch (7 ns) & Decode (3 ns) | Operand fetch (8 ns) & Execute (5 ns) | Memory (10 ns) & Write back (3 ns) 此方法將 pipeline stages 從 5 個降為 3 個，原本各個 pipeline stage cycle time 不平衡的情況也變成：10 ns | 13 ns | 13 ns 適用於對於性能要求不高的 CPU，例如：ARM7、ARM9、Cortex-M0、Cortex-M3 因為 pipeline stage 的 cycle time 增加了，從原本的 max(7 ns, 3 ns, 8 ns, 5 ns, 10 ns, 3 ns) ⇒ 10 ns，增加為 max(10 ns, 13 ns, 13 ns) ⇒ 13 ns，cycle time 增加，就代表 CPU 的 frequency 會降低 拆： 將 pipeline stage 拆成更小的 stages，例如： Fetch (7 ns) ⇒ Fetch1 (3.5 ns) & Fetch2 (3.5 ns) 適用於高性能 CPU，因為可以提昇 CPU 的 frequency 缺點： 增加所需的硬體元件，例如：需要多個 pipeline registers 功耗會增大 較深的 pipeline 也會增加 branch misprediction 的 penalty 1.2.3 - 指令間的相依性 Instructions hazard： ...

⚠️ This demo is based on PetaLinux 2024.1 release. PetaLinux Tools Documentation: Reference Guide (UG1144) (2024.1 English) Download 2024.1 PetaLinux Tools Installer & ZCU102 BSP: https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/embedded-design-tools/2024-1.html Switch to bash: 1 $ bash Set $SHELL to /bin/bash: 1 $ export SHELL=/bin/bash Disable dash as the default system shell (/bin/sh): 1 $ sudo dpkg-reconfigure dash Select: Install PetaLinux Tools Installer: ...

⚠️ The code is based on: https://gitlab.com/riseproject/riscv-optee/linux/-/tree/dev-optee-mpxy Commit ID: df5dc01764820f113312f7a39f221b49985bbd7a 1 2 3 4 5 6 7 8 9 10 // arch/riscv/kernel/mpxy-sbi.c struct sbi_mpxy { void *shmem; phys_addr_t shmem_phys_addr; bool active; }; // Define per-cpu varible: sbi_mpxy. DEFINE_PER_CPU(struct sbi_mpxy, sbi_mpxy); do_initcalls() … sbi_mpxy_init() sbi_mpxy_init() Check SBI version and if MPXY extension is supported by OpenSBI. ...

⚠️ The code is based on: https://gitlab.com/riseproject/riscv-optee/linux/-/tree/dev-optee-mpxy Commit ID: df5dc01764820f113312f7a39f221b49985bbd7a TEE supplicant /etc/init.d/S30-tee-supplicant /dev/teepriv0 main() process_one_request() process_one_request() read_request() Issue TEE_IOC_SUPPL_RECV ioctl to receive the TEE supplicant request from OP-TEE. This will be blocked until TEE supplicant request is received. Spawn a new thread to process for the new request: thread_main() process_one_request() The original thread will continue to handle the TEE supplicant request from OP-TEE: If RPC command: OPTEE_MSG_RPC_CMD_LOAD_TA: Call load_ta() to load the TA according to the UUID. load_ta() will call TEECI_LoadSecureModule() to load the TA. TEECI_LoadSecureModule() will call fopen(), ftell() to open TA and get the size of TA. If the buffer size (ta_size) is not enough to hold the TA, return the required size to let the caller increase the buffer size and try again. Otherwise, call fread() to read TA and save it to the buffer. … Call write_response() to send the TEE supplicant response to OP-TEE. write_response() Issue TEE_IOC_SUPPL_SEND ioctl to send the TEE supplicant response to OP-TEE. This will unblock Call wait_for_completion_interruptible(**&req->c**) to wait for TEE supplicant to process.

⚠️ The code is based on: https://gitlab.com/riseproject/riscv-optee/linux/-/tree/dev-optee-mpxy Commit ID: df5dc01764820f113312f7a39f221b49985bbd7a OP-TEE provides a pseudo Trusted Application (PTA): drivers/tee/optee/device.c in order to support device enumeration. In other words, OP-TEE driver invokes this application to retrieve a list of Trusted Applications which can be registered as devices on the TEE bus. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 // drivers/tee/optee/optee_private.h /** * struct optee - main service struct * @supp_teedev: supplicant device * @teedev: client device * @ops: internal callbacks for different ways to reach secure * world * @ctx: driver internal TEE context * @smc: specific to SMC ABI * @ffa: specific to FF-A ABI * @call_queue: queue of threads waiting to call @invoke_fn * @notif: notification synchronization struct * @supp: supplicant synchronization struct for RPC to supplicant * @pool: shared memory pool * @rpc_param_count: If > 0 number of RPC parameters to make room for * @scan_bus_done flag if device registation was already done. * @scan_bus_work workq to scan optee bus and register optee drivers */ struct optee { struct tee_device *supp_teedev; struct tee_device *teedev; const struct optee_ops *ops; struct tee_context *ctx; union { struct optee_smc smc; struct optee_ffa ffa; }; struct optee_shm_arg_cache shm_arg_cache; struct optee_call_queue call_queue; struct optee_notif notif; struct optee_supp supp; struct tee_shm_pool *pool; unsigned int rpc_param_count; bool scan_bus_done; struct work_struct scan_bus_work; }; 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 // drivers/tee/optee/optee_private.h /** * struct optee_ops - OP-TEE driver internal operations * @do_call_with_arg: enters OP-TEE in secure world * @to_msg_param: converts from struct tee_param to OPTEE_MSG parameters * @from_msg_param: converts from OPTEE_MSG parameters to struct tee_param * * These OPs are only supposed to be used internally in the OP-TEE driver * as a way of abstracting the different methogs of entering OP-TEE in * secure world. */ struct optee_ops { int (*do_call_with_arg)(struct tee_context *ctx, struct tee_shm *shm_arg, u_int offs, bool system_thread); int (*to_msg_param)(struct optee *optee, struct optee_msg_param *msg_params, size_t num_params, const struct tee_param *params); int (*from_msg_param)(struct optee *optee, struct tee_param *params, size_t num_params, const struct optee_msg_param *msg_params); }; do_initcalls() … optee_core_init() optee_core_init() ...

⚠️ The code is based on: https://gitlab.com/riseproject/riscv-optee/linux/-/tree/dev-optee-mpxy Commit ID: df5dc01764820f113312f7a39f221b49985bbd7a Kernel provides a TEE bus infrastructure where a Trusted Application is represented as a device identified via Universally Unique Identifier (UUID) and client drivers register a table of supported device UUIDs. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 // include/linux/tee_core.h /** * struct tee_device - TEE Device representation * @name: name of device * @desc: description of device * @id: unique id of device * @flags: represented by TEE_DEVICE_FLAG_REGISTERED above * @dev: embedded basic device structure * @cdev: embedded cdev * @num_users: number of active users of this device * @c_no_user: completion used when unregistering the device * @mutex: mutex protecting @num_users and @idr * @idr: register of user space shared memory objects allocated or * registered on this device * @pool: shared memory pool */ struct tee_device { char name[TEE_MAX_DEV_NAME_LEN]; const struct tee_desc *desc; int id; unsigned int flags; struct device dev; struct cdev cdev; size_t num_users; struct completion c_no_users; struct mutex mutex; /* protects num_users and idr */ struct idr idr; struct tee_shm_pool *pool; }; 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 // include/linux/tee_core.h /** * struct tee_desc - Describes the TEE driver to the subsystem * @name: name of driver * @ops: driver operations vtable * @owner: module providing the driver * @flags: Extra properties of driver, defined by TEE_DESC_* below */ #define TEE_DESC_PRIVILEGED 0x1 struct tee_desc { const char *name; const struct tee_driver_ops *ops; struct module *owner; u32 flags; }; 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 // include/linux/tee_core.h /** * struct tee_driver_ops - driver operations vtable * @get_version: returns version of driver * @open: called when the device file is opened * @release: release this open file * @open_session: open a new session * @close_session: close a session * @system_session: declare session as a system session * @invoke_func: invoke a trusted function * @cancel_req: request cancel of an ongoing invoke or open * @supp_recv: called for supplicant to get a command * @supp_send: called for supplicant to send a response * @shm_register: register shared memory buffer in TEE * @shm_unregister: unregister shared memory buffer in TEE */ struct tee_driver_ops { void (*get_version)(struct tee_device *teedev, struct tee_ioctl_version_data *vers); int (*open)(struct tee_context *ctx); void (*release)(struct tee_context *ctx); int (*open_session)(struct tee_context *ctx, struct tee_ioctl_open_session_arg *arg, struct tee_param *param); int (*close_session)(struct tee_context *ctx, u32 session); int (*system_session)(struct tee_context *ctx, u32 session); int (*invoke_func)(struct tee_context *ctx, struct tee_ioctl_invoke_arg *arg, struct tee_param *param); int (*cancel_req)(struct tee_context *ctx, u32 cancel_id, u32 session); int (*supp_recv)(struct tee_context *ctx, u32 *func, u32 *num_params, struct tee_param *param); int (*supp_send)(struct tee_context *ctx, u32 ret, u32 num_params, struct tee_param *param); int (*shm_register)(struct tee_context *ctx, struct tee_shm *shm, struct page **pages, size_t num_pages, unsigned long start); int (*shm_unregister)(struct tee_context *ctx, struct tee_shm *shm); }; 1 2 3 4 5 6 7 8 9 10 11 // include/linux/tee_core.h /** * struct tee_shm_pool - shared memory pool * @ops: operations * @private_data: private data for the shared memory manager */ struct tee_shm_pool { const struct tee_shm_pool_ops *ops; void *private_data; }; 1 2 3 4 5 6 7 8 9 10 11 12 13 14 // include/linux/tee_core.h /** * struct tee_shm_pool_ops - shared memory pool operations * @alloc: called when allocating shared memory * @free: called when freeing shared memory * @destroy_pool: called when destroying the pool */ struct tee_shm_pool_ops { int (*alloc)(struct tee_shm_pool *pool, struct tee_shm *shm, size_t size, size_t align); void (*free)(struct tee_shm_pool *pool, struct tee_shm *shm); void (*destroy_pool)(struct tee_shm_pool *pool); }; 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 // include/linux/tee_drv.h /** * struct tee_context - driver specific context on file pointer data * @teedev: pointer to this drivers struct tee_device * @data: driver specific context data, managed by the driver * @refcount: reference counter for this structure * @releasing: flag that indicates if context is being released right now. * It is needed to break circular dependency on context during * shared memory release. * @supp_nowait: flag that indicates that requests in this context should not * wait for tee-supplicant daemon to be started if not present * and just return with an error code. It is needed for requests * that arises from TEE based kernel drivers that should be * non-blocking in nature. * @cap_memref_null: flag indicating if the TEE Client support shared * memory buffer with a NULL pointer. */ struct tee_context { struct tee_device *teedev; void *data; struct kref refcount; bool releasing; bool supp_nowait; bool cap_memref_null; }; do_initcalls() … tee_init() tee_init() ...

⚠️ The code is based on: https://gitlab.com/riseproject/riscv-optee/opensbi/-/tree/dev-optee-mpxy Commit ID: 7d4c90953afe3bd86f9e2501bd4c2501e8db1898 init_coldboot() sbi_mpxy_init() sbi_mpxy_init() Allocate struct mpxy_state for each domain. sbi_platform_mpxy_init() Call platform-defined mpxy_init(). e.g. For generic platform: fdt_mpxy_init(). fdt_mpxy_init() Iterate MPXY drivers in fdt_mpxy_drivers[]. MPXY drivers are generated at compile time: 1 2 3 4 5 6 7 8 9 10 11 // lib/utils/mpxy/objects.mk libsbiutils-objs-$(CONFIG_FDT_MPXY) += mpxy/fdt_mpxy.o libsbiutils-objs-$(CONFIG_FDT_MPXY) += mpxy/fdt_mpxy_drivers.o carray-fdt_mpxy_drivers-$(CONFIG_FDT_MPXY_RPMI_MBOX) += fdt_mpxy_rpmi_mbox libsbiutils-objs-$(CONFIG_FDT_MPXY_RPMI_MBOX) += mpxy/fdt_mpxy_rpmi_mbox.o carray-fdt_mpxy_drivers-$(CONFIG_FDT_MPXY_MM) += fdt_mpxy_mm libsbiutils-objs-$(CONFIG_FDT_MPXY_MM) += mpxy/fdt_mpxy_mm.o carray-fdt_mpxy_drivers-$(CONFIG_FDT_MPXY_OPTEED) += fdt_mpxy_opteed libsbiutils-objs-$(CONFIG_FDT_MPXY_OPTEED) += mpxy/fdt_mpxy_opteed.o 1 2 3 4 5 // lib/utils/mpxy/fdt_mpxy_drivers.carray HEADER: sbi_utils/mpxy/fdt_mpxy.h TYPE: struct fdt_mpxy NAME: fdt_mpxy_drivers 1 2 3 4 5 6 7 // include/sbi_utils/mpxy/fdt_mpxy.h struct fdt_mpxy { const struct fdt_match *match_table; int (*init)(void *fdt, int nodeoff, const struct fdt_match *match); void (*exit)(void); }; Call drv->init(). ...

⚠️ The code is based on: https://gitlab.com/riseproject/riscv-optee/opensbi/-/tree/dev-optee-mpxy Commit ID: 7d4c90953afe3bd86f9e2501bd4c2501e8db1898 mpxy_opteed_init() Match compatible string: "riscv,sbi-mpxy-opteed". Allocate channel. opteed_domain_setup() Setup domain for OP-TEE dispatcher by looking up the domain specified by opensbi-domain-instance phandle in DTS. Assign the domain name to opteed_domain_name. Get channel ID from DTS property: riscv,sbi-mpxy-channel-id. Initialize channel: 1 2 3 4 5 6 // lib/utils/mpxy/fdt_mpxy_opteed.c channel->channel_id = channel_id; channel->send_message = mpxy_opteed_send_message; channel->attrs.msg_proto_id = SBI_MPXY_MSGPROTO_TEE_ID; channel->attrs.msg_data_maxlen = PAGE_SIZE; Register channel: sbi_mpxy_register_channel() ...

⚠️ The code is based on: https://gitlab.com/riseproject/riscv-optee/optee_os/-/tree/dev-optee-mpxy Commit ID: 75df9ba41a404aec897399ead0ff0aebcbff48ca Base addresses: U-Boot SPL: 0x80000000 OpenSBI: 0x80100000 OP-TEE: 0xf1000000 U-Boot proper: 0x81200000 / Relocated to: 0x17f76f000, relocate offset = 0xfe558000 E.g. 0x81217b66 → 0x17f76fb66; 0x81232884 → 0x17f78a884 Linux Kernel: 0x80200000 / 0xffffffff80000000 1 2 3 (gdb) add-symbol-file u-boot-spl 0x80000000 (gdb) add-symbol-file fw_dynamic.elf 0x80100000 (gdb) add-symbol-file tee.elf 0xf1000000 1 2 # For the relocated U-Boot (gdb) add-symbol-file u-boot -o 0xfe558000 When traps, mscratch/sscratch: If 0: Trap from kernel. If !0: Trap from user. See thread_init_per_cpu(), thread_trap_vect(). thread_init_per_cpu() sets mscratch/sscratch to 0 to indicate that the trap is from kernel. Fast Calls and Yielding Calls Fast Calls execute atomic operations. The call appears to be atomic from the perspective of the calling PE, and returns when the requested operation has completed. Yielding Calls start operations that can be pre-empted by a Non-secure interrupt. The call can return before the requested operation has completed.