Data is not working with UPG-VPP integrated with Open5GS

[BG00818733]

Hi,

I have setup my 5G Core with Open5GS integrated with UPG-VPP (UPF) on AWS infrastructure as per installation guides.
https://github.com/s5uishida/open5gs_5gc_ueransim_vpp_upf_dpdk_sample_config
https://github.com/s5uishida/install_vpp_upf_dpdk

MY UE is getting registered, but data is not working.

my upg configurations are.
startup.conf
unix {
nodaemon
log /tmp/vpp.log
full-coredump
gid vpp
interactive
cli-listen /run/vpp/cli.sock
exec /root/openair-upf/init.conf
}

api-trace {
on
}

cpu {
main-core 0
corelist-workers 1-3
}

api-segment {
gid vpp
}

dpdk {
dev 0000:00:06.0 {name n3}
dev 0000:00:07.0 {name n4}
dev 0000:00:08.0 {name n6}
}

plugins {
path /usr/lib/x86_64-linux-gnu/vpp_plugins/
plugin ikev2_plugin.so {disable}
plugin dpdk_plugin.so {enable}
plugin upf_plugin.so {enable}
}

init.conf
set interface ip table n6 0
set interface mtu 9000 n6
set interface ip address n6 10.1.12.6/28
set interface state n6 up

set interface ip table n4 0
set interface mtu 9000 n4
set interface ip address n4 10.1.11.6/28
set interface state n4 up

set interface ip table n3 0
set interface mtu 9000 n3
set interface ip address n3 10.1.10.6/28
set interface state n3 up

ip route add 0.0.0.0/0 table 0 via 10.1.12.14 n6

upf pfcp endpoint ip 10.1.11.6 vrf 0

upf node-id fqdn 10.1.11.6

upf nwi name internet vrf 0

upf specification release 16

upf gtpu endpoint ip 10.1.12.6 nwi internet teid 0x000004d2/2

[BG00818733]

One additional thing while starting the UPG VPP, i am getting below error.

root@ip-10-1-1-180:/home/ubuntu# /usr/bin/vpp -c /root/openair-upf/startup.conf
dpdk [warn ]: not enough DPDK crypto resources
dpdk [warn ]: unsupported tx offloads requested on port 0: multi-segs
dpdk [warn ]: unsupported tx offloads requested on port 1: multi-segs
dpdk [warn ]: unsupported tx offloads requested on port 2: multi-segs
dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Not enough cryptodevs
_______ _ _ _____ ___
/ / _ \ (_) | | / / _ / _
_/ // // / / / _ \ | |/ / / /
// /()/_/ |_// //

vpp# interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
0: upip_ip_interface_ip:1298: No NWI IP found, using first interface IP
0: upip_ip_interface_ip:1298: No NWI IP found, using first interface IP

Please suggest your inputs and help us to fix this.

[BG00818733]

UPF Session output:

vpp# show upf session
CP F-SEID: 0x00000000000005a9 (1449) @ 10.1.11.11
UP F-SEID: 0x00000000000005a9 (1449) @ 10.1.11.6
PFCP Association: 0
TEID assignment per choose ID
PDR: 1 @ 0x7f1dc72bf2b0
Precedence: 255
PDI:
Fields: 0000000c
Source Interface: Core
Network Instance: internet
UE IP address (destination):
IPv4 address: 10.45.0.2
SDF Filter [1]:
permit out ip from any to assigned
Outer Header Removal: no
FAR Id: 1
URR Ids: [1] @ 0x7f1dc6432f30
QER Ids: [1] @ 0x7f1dc6432f60
PDR: 2 @ 0x7f1dc72bf330
Precedence: 255
PDI:
Fields: 0000000d
Source Interface: Access
Network Instance: internet
Local F-TEID: 827107826 (0x314ca9f2)
IPv4: 10.1.12.6
UE IP address (source):
IPv4 address: 10.45.0.2
SDF Filter [1]:
permit out ip from any to assigned
Outer Header Removal: GTP-U/UDP/IPv4
FAR Id: 2
URR Ids: [] @ 0x0
QER Ids: [1] @ 0x7f1dc72c0580
PDR: 3 @ 0x7f1dc72bf3b0
Precedence: 1000
PDI:
Fields: 00000001
Source Interface: CP-function
Network Instance: internet
Local F-TEID: 564920489 (0x21ac00a9)
IPv4: 10.1.12.6
Outer Header Removal: GTP-U/UDP/IPv4
FAR Id: 1
URR Ids: [] @ 0x0
QER Ids: [1] @ 0x7f1dc72c05b0
PDR: 4 @ 0x7f1dc72bf430
Precedence: 1
PDI:
Fields: 00000009
Source Interface: Access
Network Instance: internet
Local F-TEID: 827107826 (0x314ca9f2)
IPv4: 10.1.12.6
SDF Filter [1]:
permit out 58 from ff02::2 to assigned
Outer Header Removal: GTP-U/UDP/IPv4
FAR Id: 3
URR Ids: [] @ 0x0
QER Ids: [] @ 0x0
FAR: 1
Apply Action: 00000002 == [FORWARD]
Forward:
Network Instance: internet
Destination Interface: 0
Outer Header Creation: [GTP-U/UDP/IPv4],TEID:00000001,IP:10.1.10.11
FAR: 2
Apply Action: 00000002 == [FORWARD]
Forward:
Network Instance: internet
Destination Interface: 1
FAR: 3
Apply Action: 00000002 == [FORWARD]
Forward:
Network Instance: internet
Destination Interface: 3
Outer Header Creation: [GTP-U/UDP/IPv4],TEID:00000001,IP:10.1.11.11
URR: 1
Measurement Method: 0002 == [VOLUME]
Reporting Triggers: 0002 == [VOLUME THRESHOLD]
Status: 0 == []
Start Time: 2024/02/27 11:29:23:214
vTime of First Usage: 0.0000
vTime of Last Usage: 0.0000
Volume
Up: Measured: 0, Theshold: 0, Pkts: 0
Consumed: 0, Quota: 0
Down: Measured: 0, Theshold: 0, Pkts: 0
Consumed: 0, Quota: 0
Total: Measured: 0, Theshold: 104857600, Pkts: 0
Consumed: 0, Quota: 0

[s5uishida]

@BG00818733

I haven't tried it, but the following may be helpful.

https://s3-docs.fd.io/vpp/24.02/usecases/vppcloud/vppinaws.html

$ sudo modprobe uio
$ sudo insmod kmod/igb_uio.ko

Are you binding your network devices with this module?

[BG00818733]

Hi @s5uishida. Thanks for your response.
Currently we are binding the ENA devices with vfio-pci

Please suggest your inputs

[s5uishida]

@BG00818733

You may try igb_uio instead of vfio-pci.
example)

dpdk-devbind.py -b igb_uio 0000:00:06.0 --force

[BG00818733]

@s5uishida Will it work for Ubuntu 22.04 with EC2 Instance Type c5.2xl.

Actually I am getting below error while loading igb_uio.
modprobe igb_uio
modprobe: FATAL: Module igb_uio not found in directory /lib/modules/6.2.0-1017-aws

Please suggest

[BG00818733]

Hi @s5uishida With Ubuntu 20.04. I have installed the igb_uio using the command.
sudo apt-get install -y dpdk-igb-uio-dkms

Now After binding the ENA with igb_ui, still we are facing the same issues.
dpdk-devbind.py -b igb_uio 0000:00:06.0 --force
dpdk-devbind.py -b igb_uio 0000:00:07.0 --force
dpdk-devbind.py -b igb_uio 0000:00:08.0 --force

Please suggest your inputs.

[BG00818733]

Hi @s5uishida,

Can below errors could be the reason for data not working. Please share your views on this.

vpp# interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
0: upip_ip_interface_ip:1298: No NWI IP found, using first interface IP
0: upip_ip_interface_ip:1298: No NWI IP found, using first interface IP

[BG00818733]

Hi @s5uishida

We tried further with m5 instance type categories on AWS EC2 and facing the same dpdk binding errors as mentioned above.
Additionally, we tried the same in VMs on a baremetal server and we are not seeing any errors.

Hence, we request your support for the difference in Cloud Infrastructure which is causing below dpdk binding errors.

root@ip-10-1-1-244:/home/ubuntu# /usr/bin/vpp -c /root/openair-upf/startup.conf
clib_sysfs_prealloc_hugepages:260: pre-allocating 20 additional 2048K hugepages on numa node 0
0: clib_sysfs_prealloc_hugepages:260: pre-allocating 8 additional 2048K hugepages on numa node 0
dpdk [warn ]: not enough DPDK crypto resources
dpdk [warn ]: unsupported tx offloads requested on port 0: multi-segs
dpdk [warn ]: unsupported tx offloads requested on port 1: multi-segs
dpdk [warn ]: unsupported tx offloads requested on port 2: multi-segs
dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Not enough cryptodevs
_______ _ _ _____ ___
/ / _ \ (_) | | / / _ / _
_/ // // / / / _ \ | |/ / / /
// /()/_/ |_// //

vpp# interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95
interface [error ]: hw_add_del_mac_address: dpdk_add_del_mac_address: mac address add/del failed: -95

Your inputs will be appreciated.

Thanks and Regards,
Bharat Gupta

[BG00818733]

Hi All,

This issues worked for us by disabling the source/destination check in AWS.
Hence closing this issue.

Regards,
Bharat