DFX-MGR provides infrastructure to abstract configuration and hardware resource management for dynamic deployment of Xilinx based accelerator across different platforms.

DFX-MGR is merged in Petalinux 2021.1 onwards and the recipe is called ‘dfx-mgr’. The recipe is not enabled by default and user is expected to enable it. As of today, DFX-MGR can be used for dynamic loading/unloading of accelerators to PL (Programmable Logic) or AI Engine. The functionality is tested for loading/unloading of Flat shell (i.e. shell which does not have any reconfigurable partitions) and DFX shell (i.e. shell which contains static and dynamic region). As you can see in the diagram below DFX-MGR can load a 3RP design and a 2RP design with the corresponding accelerators dynamically without having to reboot the board

Once you compile petalinux by enabling the dfx-mgr recipe, you should have dfx-mgrd and dfx-mgr-client in /usr/bin of rootfs and libdfx-mgr.so in /usr/lib. The config file daemon.conf can be found in /etc/dfx-mgrd/. Config file is mandatory, refer the files section for details of it. A default daemon.conf will be copied by the recipe and the user is expected to update as required.

DFX-MGR recognizes the designs under /lib/firmware/xilinx on device filesystem, location can be updated in daemon.conf. Designs could be downloaded using dnf or any other package manager or manually copied to the previously mentioned location. Each folder under /lib/firmware/xilinx will be treated as a base shell design wherein the folder is expected to have a shell.json file. Base design folder can then have sub-folder for each of the accelerators. Each accelerator is expected to have a accel.json. Have a look at below folder structure for more understanding and the details of json config files.

The expected directory structure under /lib/firmware/xilinx which contains a 3x1 PL shell design, AIE accel and a flat shell design. 3x1 base_design shell has an accelerator called FFT which contains three different partial bitstreams for three slots of base shell. base_design needs to have base shell bitstream and shell.json. DFX-MGR expects '_slot#' as subfolders for each of the accelerators for DFX designs. Place the bitstream corresponding to that slot in the respective subfolder along with accel.json file.

It is not mandatory to have all the partial bitstreams for each slot but DFX-MGR will fail to load an accelerator to the slot if no partial design is found.

$ls /lib/firmware/xilinx/base_design
base_Shell.pdi base.dtbo shell.json FFT AIE_Accel

$ls /lib/firmware/xilinx/base_design/FFT/
FFT_slot0 FFT_slot1 FFT_slot2

$ls /lib/firmware/xilinx/base_design/FFT/FFT_slot0/
partial.pdi partial.dtbo accel.json

$ls /lib/firmware/xilinx/base_design/FFT/FFT_slot1/
partial.pdi partial.dtbo accel.json

$ls /lib/firmware/xilinx/base_design/FFT/FFT_slot2/
partial_slot2.pdi partial.dtbo accel.json

$ls /lib/firmware/xilinx/base_design/AIE_accel
AIE_accel_slot3

$ls /lib/firmware/xilinx/base_design/AIE_accel/AIE_accel_slot3
aie.xclbin accel.json

$ls /lib/firmware/xilinx/flat_shell/
Flat_shell.bit.bin flat_shell.dtbo shell.json

As shown above, base_design has first three slots (0,1,2) corresponding to PL slots from the FFT subfolders, and AIE_accel has slot3. When DFX-MGR successfully loads FFT accel to one of the slots, -listPackage output would show active slot as 0,1 or 2. When AIE_accel is programmed, it will show active slot as 3. If AIE were to have multiple slots, the subfolders would have to be named 3,4 and so on.

DFX-MGR is started on linux bootup and reads the config file /etc/dfx-mgrd/daemon.conf from device for any config settings. Any change to daemon.conf will need a restart of the /usr/bin/dfx-mgrd on target.

For eg. If you want to add another location for packages on the filesystem, append absolute path to the location in "firmware_location" and restart the daemon. The limitations around directory structure as explained above still apply as for "/lib/firmware/xilinx".

$cat /etc/dfx-mgrd/daemon.conf
{
  "default_accel":"/etc/dfx-mgrd/default_firmware", //Optional: echo the
                             //package-name to default_firmware for any
                             // accelerator to be loaded on start of daemon
 "firmware_location": ["/lib/firmware/xilinx"]  //Required:Package directories
                        // monitored by DFX-MGR }

shell.json describes the base shell configuration information. Optional fields can be skipped if not desired.

One of the below type should be used for shell_type as per your design. XRT-FLAT: dfx-mgr will program the PL and update /etc/vart.conf on target with the path to the active xclbin on success. PL_FLAT: dfx-mgr will program the PL bitstream and treat the design as static. PL_DFX: dfx-mgr will treat the design as DFX with number of slots as mentioned in json.

$cat shell.json 
 { 
  "shell_type" : "PL_DFX",// Required: valid values are XRT_FLAT/PL_FLAT/PL_DFX
  "num_pl_slots": 3, //Required: Number of pl slots in your base shell design
  "num_aie_slots":1, //Required: Number of aie slots in your base shell design
  "load_base_design": "no" //Optional : If the shell was loaded during boot
              //and you want to skip loading base shell when loading the
              // accelerators to the slot.
  "device_name" : "a0010000.SIHA_Manager", //optional: IP name
  "reg_base" : "", //Optional: IP device base address 
  "reg_size" : "", //Optional 
  "clock_device_name" : "a0000000.clk_wiz", //optional
  "clock_reg_base" : "",//optional
  "clock_reg_size" : "", //optional
  "isolation_slots": [
	// optional, specify the register offsets and the corresponding values to be
	// written for each slot to bring it out of isolation, expects the length
    //of array equal to number of slots 
	{ "offset": ["0x4000","0x4004","0x4008","0x400c"],//Address offset for first slot
	"values": ["0x01","0x01", "0x0b", "0x02"] }, //Value to be written to above regs
	{ "offset": ["0x5000","0x5004","0x5008","0x500c"],
	"values": ["0x01","0x01", "0x0b", "0x02"]},
	{ "offset": ["0x6000","0x6004","0x6008","0x600c"],
	"values": ["0x01","0x01","0x0b", "0x02"] }]
	 }

accel.json describes the accelerator configuration. Optional fields can be skipped if not desired. Flat shell designs are not required to have accel.json since they do not have reconfigurable partition.

SIHA_PL_DFX: Use this option for PL accelerators build with siha hw manager, this enables some extra steps required to bring the slots out of isolation in addition to programming the bitstream. XRT_AIE_DFX: Use this option for AIE accelerator which expects an xclbin to program the AIE. XRT_PL_DFX: Use this option for XRT based PL accelerator.

$cat accel.json
{ 
  "accel_type": "",  //Required: supported types are SIHA_PL_DFX/XRT_AIE_DFX/XRT_PL_DFX
  "accel_devices":[ // Optional: list of IP devices corresponding to this slot design
  { "dev_name": "20100000000.accel",
   "reg_base":"", 
   "reg_size":"" }],
  "sharedMemoryConfig": {
   "sizeInKB": "",
   "sharedMemType" : ""},
  "dataMoverConfig": { //Optional: skip
this node if application handles its own dma
  "dma_dev_name":"a4000000.dma",
  "dma_driver":"vfio-platform",
  "dma_reg_base":"",
  "dma_reg_size":"",
  "iommu_group":"0",
  "Bus": "platform",
  "HWType": "mcdma",
  "max_buf_size":"8388608",
  "dataMoverCacheCoherent": "Yes",
  "dataMoverVirtualAddress": "Yes",
  "dataMoverChnls":[
  {"chnl_id": 0,
   "chnl_dir":"ACAPD_DMA_DEV_W" },
  { "chnl_id": 0,
   "chnl_dir": "ACAPD_DMA_DEV_R" }]
  },
  "AccelHandshakeType": "”,
  "fallbackBehaviour": "software" //Optional: If hw accelerator fails to load,
                    //DFX-MGR will try software fallback
} 

DFX-MGR depends on external libraries/frameworks like libdfx, libwebsockets, XRT, i-notify etc. The recommended way to compile this repo is using yocto where the required dependency are taken care of in the recipe.

If not using yocto then dependent libaries will need to be provided to cmake using appropriate -DCMAKE_LIBRARY_PATH.

To compile using yocto in 2021.1 onwards, do bitbake dfx-mgr.

You would need to provide dependency libraries using -DCMAKE_LIBRARY_PATH for cmake.

There is generic cmake toolchian file for generic Linux which is in cmake/platforms/cross-linux-gcc.cmake

Set the path where you have all the dependent libraries in build.sh.

$ cd dfx-mgr 
$ mkdir build 
$ cd build 
$ ../build.sh

After build successfully completes, libdfx-mgr.so can be found under build/usr/local/lib and binary under build/usr/local/bin.

dfx-mgrd daemon should mostly be running on linux startup. Assuming it is running, you can use below commands from command line to load/unload accelerators.

1. Command to list the packages present on target filesystem under /lib/firmware/xilinx.

An alternative command line utility which some users might be familiar with can also be used xmutil listapps.

$dfx-mgr-client -listPackage
Accelerator    Accel_type          Base   Base_type  #slots(PL+AIE) Active_slot

  flat_shell	  PL_FLAT    flat_shell     PL_FLAT           (0+0)          -1
  AIE_accel   XRT_AIE_DFX   base_design      PL_DFX           (3+1)          3,
        FFT   SIHA_PL_DFX   base_design      PL_DFX           (3+1)         0,1

In the above output, AIE_accel is currently programmed to AIE and hence the slot shows as 3. FFT is programmed to first two slots out of three slots of PL. flat_shell is not programmed and show -1. Flat shell design do not have a dynamic reconfigurable partition and hence #slots shows as 0.

2. Command to load one of the above listed accelerator. For dfx designs, base shell would be loaded first if not already loaded and then accelerator will be loaded to one of the free slots. If the shell is already loaded then only the accelerator will be loaded to one of free slots.

Equivalent xmutil command is xmutil loadapp FFT.

$dfx-mgr-client -load FFT

3. Command to remove accelerator from the slot. If there is not accel in the mentioned slot, this command will do nothing.

Equivalent xmutil command is xmutil unloadapp 1.

$dfx-mgr-client -remove 1

Users can write application to interact with daemon. Refer to example source code in repo/example/sys/linux/load_accel.c for a simple example how to load an accelerator.

For DFX designs we recommend using graph api's which provide good abstract way to create a visual model of the RP's and inter connections. Refer to some example code under <repo>/example/sys/linux/graph.

1. DFX-MGR uses i-notify for firmware file updates and i-notify doesn't work with network filesystem. Hence it is recommended to NOT boot linux over NFS for correct functinality of DFX-MGR daemon.

2. DFX-MGR package names i.e. firmware folder names are limited to 64 character currently and absolute path lengths are limited to 512 char. Hence avoid creating long filenames. If absolutely required for longer names, file an issue in the repo and will be adderssed on priority.

3. I/O nodes doesn't support zero copy.