Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure JEP 454 (Foreign Function and Memory API) works correctly with Mandrel for JDK 22 (24.0) #678

Open
jerboaa opened this issue Feb 1, 2024 · 6 comments
Open

Ensure JEP 454 (Foreign Function and Memory API) works correctly with Mandrel for JDK 22 (24.0) #678

jerboaa opened this issue Feb 1, 2024 · 6 comments
Labels
affects/24.0 bug Something isn't working not-Stale pending upstream Issue or PR pending action on upstream repository (oracle/graal)

Comments

@jerboaa
Copy link
Collaborator

jerboaa commented Feb 1, 2024

Description

With JDK 22, JEP 454 went out of preview. For GraalVM this currently means needing to ship svm-foreign.jar in the correct location of the distro. Specifically, next to svm.jar in $MANDREL_HOME/lib/svm/builder. Then, native-image builds pick up the jar and add it to the module path when the generator runs.

However, after fixing that, not everything seems to be fully working (might be my test case, but still).

How To Reproduce

$ cat FFM.java
import static java.lang.foreign.ValueLayout.ADDRESS;
import static java.lang.foreign.ValueLayout.JAVA_INT;
import static java.lang.foreign.ValueLayout.JAVA_LONG;

import java.lang.foreign.Arena;
import java.lang.foreign.FunctionDescriptor;
import java.lang.foreign.Linker;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.SymbolLookup;
import java.lang.foreign.ValueLayout;
import java.lang.invoke.MethodHandle;
import java.lang.invoke.MethodHandles;
import java.lang.invoke.MethodType;
import java.util.Arrays;
import java.util.List;

public class FFM {
	static int compare(MemorySegment elem1, MemorySegment elem2) {
		MemorySegment s1 = elem1.get(ADDRESS.withTargetLayout(ADDRESS), 0);
		String one = s1.getString(0);
		MemorySegment s2 = elem2.get(ADDRESS.withTargetLayout(ADDRESS), 0);
		String two = s2.getString(0);
		assert one != null;
		assert two != null;
		int value = one.compareTo(two);
		if (value == 0) {
			return 0;
		} else {
			return value < 0 ? -1 : 1;
		}
	}

	public static void main(String[] args) throws Throwable {
		// 1. Find foreign function on the C library path
		Linker linker = Linker.nativeLinker();
		SymbolLookup stdlib = linker.defaultLookup();
		MethodHandle qsort = linker.downcallHandle(stdlib.find("qsort").orElseThrow(),
				FunctionDescriptor.ofVoid(ADDRESS, JAVA_LONG, JAVA_LONG, ADDRESS));
		MethodHandle comparHandle = MethodHandles.lookup().findStatic(FFM.class, "compare",
				MethodType.methodType(int.class, MemorySegment.class, MemorySegment.class));
		// 2. Allocate on-heap memory to store four strings
		String[] javaStrings = { "mouse", "cat", "dog", "car" };
		// 3. Use try-with-resources to manage the lifetime of off-heap memory
		try (Arena offHeap = Arena.ofConfined()) {
			MemorySegment comparFunc = linker.upcallStub(comparHandle,
					/*
					 * A Java description of a C function implemented by a Java method!
					 */
					FunctionDescriptor.of(JAVA_INT,
							ValueLayout.ADDRESS.withTargetLayout(ADDRESS),
							ValueLayout.ADDRESS.withTargetLayout(ADDRESS)),
					offHeap);
			// 4. Allocate a region of off-heap memory to store four pointers
			MemorySegment pointers = offHeap.allocate(ValueLayout.ADDRESS, javaStrings.length);
			// 5. Copy the strings from on-heap to off-heap
			for (int i = 0; i < javaStrings.length; i++) {
				MemorySegment cString = offHeap.allocateFrom(javaStrings[i]);
				pointers.setAtIndex(ValueLayout.ADDRESS, i, cString);
			}
			// 6. Sort the off-heap data by calling the foreign function
			qsort.invoke(pointers, (long) javaStrings.length, ADDRESS.byteSize(), comparFunc);

			// 7. Copy the (reordered) strings from off-heap to on-heap
			for (int i = 0; i < javaStrings.length; i++) {
				MemorySegment cString = pointers.getAtIndex(ValueLayout.ADDRESS, i);
				javaStrings[i] = cString.reinterpret(ADDRESS.byteSize()).getString(0);
			}
		} // 8. All off-heap memory is deallocated here
		List<String> list = Arrays.asList(javaStrings);
		if (!Arrays.equals(javaStrings, new String[] { "car", "cat", "dog", "mouse" })) {
			throw new AssertionError("Lists not sorted! Got:" + list);
		} else {
			System.out.println("Test passed! List sorted: " + list);
		}
	}
}
$ javac FFM.java
$ native-image --verbose -H:+UnlockExperimentalVMOptions -H:+ForeignAPISupport --no-fallback FFM
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/jni-config.json
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/reflect-config.json
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/resource-config.json
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/proxy-config.json
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/serialization-config.json
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/predefined-classes-config.json
Apply jar:file:https:///disk/graal/upstream-sources/graal/mandrel-build/lib/svm/library-support.jar!/META-INF/native-image/com.oracle.svm/thirdparty/native-image.properties
Executing [
HOME=/home/sgehwolf \
LANG=en_US.UTF-8 \
PATH=/home/sgehwolf/.local/bin:/home/sgehwolf/bin:/home/sgehwolf/Documents/eclipse/eclipse-2022/eclipse:/home/sgehwolf/.local/bin:/home/sgehwolf/bin:/home/sgehwolf/Documents/eclipse/eclipse-2022/eclipse:/usr/lib64/qt-3.3/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin \
PWD=/home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example \
USE_NATIVE_IMAGE_JAVA_PLATFORM_MODULE_SYSTEM=true \
/disk/graal/upstream-sources/graal/mandrel-build/bin/java \
-XX:+UseParallelGC \
-XX:+UnlockExperimentalVMOptions \
-XX:+EnableJVMCI \
-Dtruffle.TrustAllTruffleRuntimeProviders=true \
-Dtruffle.TruffleRuntime=com.oracle.truffle.api.impl.DefaultTruffleRuntime \
-Dgraalvm.ForcePolyglotInvalid=true \
-Dgraalvm.locatorDisabled=true \
-Dsubstratevm.HostLibC=glibc \
--add-exports=java.base/com.sun.crypto.provider=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.access=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.event=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.loader=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.logger=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.misc=org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile,org.graalvm.nativeimage.pointsto \
--add-exports=java.base/jdk.internal.module=org.graalvm.nativeimage.base,org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.org.objectweb.asm=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.perf=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.platform=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.ref=org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile \
--add-exports=java.base/jdk.internal.reflect=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.util=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.vm.annotation=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.vm=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.invoke.util=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.net.www=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.net=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.nio.ch=org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile \
--add-exports=java.base/sun.reflect.annotation=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.reflect.generics.factory=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.reflect.generics.reflectiveObjects=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.reflect.generics.repository=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.reflect.generics.scope=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.reflect.generics.tree=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.security.jca=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.security.provider=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.security.ssl=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.security.util=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.security.x509=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.text.spi=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util.calendar=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util.cldr=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util.locale.provider=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util.locale=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util.resources=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util=org.graalvm.nativeimage.builder \
--add-exports=java.management/com.sun.jmx.mbeanserver=org.graalvm.nativeimage.builder \
--add-exports=java.management/sun.management=org.graalvm.nativeimage.builder,org.graalvm.nativeimage.pointsto \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.aarch64=jdk.graal.compiler,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.amd64=jdk.graal.compiler,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.code.site=jdk.graal.compiler,org.graalvm.nativeimage.builder \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.code.stack=jdk.graal.compiler,org.graalvm.nativeimage.builder \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.code=jdk.graal.compiler,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile,org.graalvm.nativeimage.pointsto,org.graalvm.truffle.compiler \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.common=jdk.graal.compiler,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.pointsto \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot.aarch64=jdk.graal.compiler \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot.amd64=jdk.graal.compiler \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot.riscv64=jdk.graal.compiler \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot=jdk.graal.compiler,org.graalvm.nativeimage.builder \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.meta=jdk.graal.compiler,org.graalvm.nativeimage.base,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile,org.graalvm.nativeimage.pointsto,org.graalvm.truffle.compiler \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.riscv64=jdk.graal.compiler,org.graalvm.nativeimage.builder \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.runtime=jdk.graal.compiler,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.pointsto \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.services=jdk.graal.compiler,org.graalvm.nativeimage.builder \
--add-exports=jdk.jfr/jdk.jfr.events=org.graalvm.nativeimage.builder \
--add-exports=jdk.jfr/jdk.jfr.internal.event=org.graalvm.nativeimage.builder \
--add-exports=jdk.jfr/jdk.jfr.internal.jfc=org.graalvm.nativeimage.builder \
--add-exports=jdk.jfr/jdk.jfr.internal=org.graalvm.nativeimage.builder \
--add-exports=jdk.management/com.sun.management.internal=org.graalvm.nativeimage.builder \
-XX:-UseJVMCICompiler \
-Xss10m \
-XX:MaxRAMPercentage=47.640579346032816 \
-XX:GCTimeRatio=9 \
-XX:+ExitOnOutOfMemoryError \
-Djava.awt.headless=true \
'-Dorg.graalvm.vendor=GraalVM Community' \
-Dorg.graalvm.vendorurl=https://github.com/graalvm/mandrel/issues \
-Dorg.graalvm.vendorversion=Mandrel-24.0.0-dev \
-Dorg.graalvm.version=24.0.0-dev \
-Dcom.oracle.graalvm.isaot=true \
-Djava.system.class.loader=com.oracle.svm.hosted.NativeImageSystemClassLoader \
-Xshare:off \
--upgrade-module-path \
/disk/graal/upstream-sources/graal/mandrel-build/lib/jvmci/graal.jar \
-Djdk.reflect.useOldSerializableConstructor=true \
-Djdk.internal.lambda.disableEagerInitialization=true \
-Djdk.internal.lambda.eagerlyInitialize=false \
-Djava.lang.invoke.InnerClassLambdaMetafactory.initializeLambdas=false \
-Djava.lang.invoke.MethodHandle.DONT_INLINE_THRESHOLD=-1 \
-Djava.lang.invoke.MethodHandle.PROFILE_GWT=false \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.code=jdk.graal.compiler \
--add-modules=ALL-DEFAULT \
--module-path \
/disk/graal/upstream-sources/graal/mandrel-build/lib/jvmci/collections.jar:/disk/graal/upstream-sources/graal/mandrel-build/lib/jvmci/word.jar:/disk/graal/upstream-sources/graal/mandrel-build/lib/jvmci/nativeimage.jar:/disk/graal/upstream-sources/graal/mandrel-build/lib/truffle/truffle-compiler.jar:/disk/graal/upstream-sources/graal/mandrel-build/lib/svm/builder/native-image-base.jar:/disk/graal/upstream-sources/graal/mandrel-build/lib/svm/builder/svm-foreign.jar:/disk/graal/upstream-sources/graal/mandrel-build/lib/svm/builder/pointsto.jar:/disk/graal/upstream-sources/graal/mandrel-build/lib/svm/builder/objectfile.jar:/disk/graal/upstream-sources/graal/mandrel-build/lib/svm/builder/svm.jar \
--module \
org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner \
-keepalive \
/proc/26519/comm \
-imagecp \
/home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example \
-imagemp \
/disk/graal/upstream-sources/graal/mandrel-build/lib/svm/library-support.jar \
-H:CLibraryPath=/disk/graal/upstream-sources/graal/mandrel-build/lib/svm/clibraries/linux-amd64 \
-H:Path@driver=/home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example \
-H:+UnlockExperimentalVMOptions@user \
-H:+ForeignAPISupport@user \
-H:FallbackThreshold@user+api=0 \
-H:-UnlockExperimentalVMOptions@user \
-H:JNIConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/jni-config.json+api=META-INF/native-image/jni-config.json \
-H:ReflectionConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/reflect-config.json+api=META-INF/native-image/reflect-config.json \
-H:ResourceConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/resource-config.json+api=META-INF/native-image/resource-config.json \
-H:DynamicProxyConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/proxy-config.json+api=META-INF/native-image/proxy-config.json \
-H:SerializationConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/serialization-config.json+api=META-INF/native-image/serialization-config.json \
-H:PredefinedClassesConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/predefined-classes-config.json+api=META-INF/native-image/predefined-classes-config.json \
'-H:Class@explicit main-class=FFM' \
'-H:Name@main-class lower case as image name=ffm' \
-H:ImageBuildID@driver=6a95d3de-3ea3-49cd-e1c6-74d3ccfd42e7 \
'-H:Features@jar:file:https:///disk/graal/upstream-sources/graal/mandrel-build/lib/svm/library-support.jar!/META-INF/native-image/com.oracle.svm/thirdparty/native-image.properties+api=com.oracle.svm.thirdparty.gson.GsonFeature' \
-H:Color@driver=always \
-H:+BuildOutputProgress@driver \
-H:+BuildOutputLinks@driver
]
========================================================================================================================
GraalVM Native Image: Generating 'ffm' (executable)...
========================================================================================================================
[1/8] Initializing...                                                                                    (2.8s @ 0.18GB)
 Java version: 22-beta+33-ea, vendor version: Mandrel-24.0.0-dev
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 13.2.1)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 1 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
------------------------------------------------------------------------------------------------------------------------
 1 experimental option(s) unlocked:
 - '-H:+ForeignAPISupport' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 26.49GB of memory (42.3% of 62.56GB system memory, determined at start)
 - 12 thread(s) (100.0% of 12 available processor(s), determined at start)
[2/8] Performing analysis...  [*****]                                                                   (10.2s @ 0.38GB)
    4,387 reachable types   (76.6% of    5,727 total)
    4,725 reachable fields  (43.0% of   11,001 total)
   20,329 reachable methods (49.3% of   41,200 total)
    1,417 types,   228 fields, and   962 methods registered for reflection
       57 types,    57 fields, and    52 methods registered for JNI access
       0 foreign downcalls registered
        4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                               (1.7s @ 0.52GB)
[4/8] Parsing methods...      [*]                                                                        (1.4s @ 0.53GB)
[5/8] Inlining methods...     [***]                                                                      (0.9s @ 0.52GB)
[6/8] Compiling methods...    [***]                                                                     (11.2s @ 0.40GB)
[7/8] Laying out methods...   [*]                                                                        (2.0s @ 0.50GB)
[8/8] Creating image...       [*]                                                                        (1.6s @ 0.56GB)
   7.35MB (42.74%) for code area:    11,784 compilation units
   9.53MB (55.37%) for image heap:  116,272 objects and 47 resources
 332.78kB ( 1.89%) for other data
  17.21MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
   5.91MB java.base                                            2.19MB byte[] for code metadata
1009.52kB svm.jar (Native Image)                               1.53MB byte[] for java.lang.String
 113.17kB java.logging                                         1.09MB java.lang.String
  64.36kB org.graalvm.nativeimage.base                         1.02MB java.lang.Class
  47.59kB jdk.proxy1                                         377.01kB com.oracle.svm.core.hub.DynamicHubCompanion
  45.84kB jdk.proxy3                                         310.09kB heap alignment
  35.63kB FFM                                                286.59kB byte[] for general heap data
  26.78kB jdk.internal.vm.ci                                 262.31kB java.util.HashMap$Node
  22.03kB org.graalvm.collections                            247.43kB java.lang.Object[]
  11.42kB jdk.proxy2                                         217.12kB byte[] for reflection metadata
  20.10kB for 4 more packages                                  2.04MB for 1211 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                        1.3s (3.8% of total time) in 158 GCs | Peak RSS: 1.01GB | CPU load: 9.42
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/ffm (executable)
========================================================================================================================
Finished generating 'ffm' in 32.9s.
$  ./ffm 
WARNING: A restricted method in java.lang.foreign.Linker has been called
WARNING: java.lang.foreign.Linker::downcallHandle has been called by FFM in an unnamed module
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

Exception in thread "main" com.oracle.svm.core.foreign.ForeignFunctionsRuntime$UnregisteredForeignStubException: Cannot perform downcall with leaf type (long,long,long,long,long)void as it was not registered at compilation time.
	at org.graalvm.nativeimage.foreign/com.oracle.svm.core.foreign.ForeignFunctionsRuntime.getDowncallStubPointer(ForeignFunctionsRuntime.java:75)
	at org.graalvm.nativeimage.foreign/com.oracle.svm.core.foreign.NativeEntryPointInfo.makeEntryPoint(NativeEntryPointInfo.java:88)
	at java.base@22-beta/jdk.internal.foreign.abi.NativeEntryPoint.make(Target_jdk_internal_foreign_abi_NativeEntryPoint.java:60)
	at java.base@22-beta/jdk.internal.foreign.abi.DowncallLinker.getBoundMethodHandle(DowncallLinker.java:87)
	at java.base@22-beta/jdk.internal.foreign.abi.x64.sysv.CallArranger.arrangeDowncall(CallArranger.java:132)
	at java.base@22-beta/jdk.internal.foreign.abi.x64.sysv.SysVx64Linker.arrangeDowncall(SysVx64Linker.java:62)
	at java.base@22-beta/jdk.internal.foreign.abi.AbstractLinker.lambda$downcallHandle0$0(AbstractLinker.java:106)
	at java.base@22-beta/jdk.internal.foreign.abi.SoftReferenceCache$Node.get(SoftReferenceCache.java:49)
	at java.base@22-beta/jdk.internal.foreign.abi.SoftReferenceCache.get(SoftReferenceCache.java:38)
	at java.base@22-beta/jdk.internal.foreign.abi.AbstractLinker.downcallHandle0(AbstractLinker.java:103)
	at java.base@22-beta/jdk.internal.foreign.abi.AbstractLinker.downcallHandle(AbstractLinker.java:85)
	at FFM.main(FFM.java:37)
	at java.base@22-beta/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)

Configuration:

Mandrel 24.0 with JDK 22+33.

Expected behavior

Reproducer works the same as when run with java:

$ java --enable-native-access=ALL-UNNAMED FFM
Test passed! List sorted: [car, cat, dog, mouse]

Additional context

Where svm-foreign.jar is located changed from 23.1 (JDK 21) to 24.0 (JDK 22). It used to live in an svm-preview folder. There also seems to be a difference on how --enable-native-access works (one cannot silence the warning with the switch in native-image).

GraalVM for JDK 22 community (self-built) seems to have a similar issue.
@jerboaa jerboaa added the bug Something isn't working label Feb 1, 2024
@jerboaa jerboaa mentioned this issue Feb 1, 2024
Closed
@jerboaa
Copy link
Collaborator Author

jerboaa commented Feb 1, 2024 •
edited
Loading

Going by the discussion in oracle#8012, specifically this:

Foreign Function and Memory API support (OpenJKD Project Panama): Since support in native image
is still under development, it needs to be enabled explicitly at image build time (and then some things
will work, and some things will not work).
https://github.com/oracle/graal/pull/7980

It might not be 100% ready yet for JDK 22.

@jerboaa
Copy link
Collaborator Author

jerboaa commented Feb 6, 2024

There also seems to be a difference on how --enable-native-access works (one cannot silence the warning with the switch in native-image).

-J--enable-native-access=ALL-UNNAMED needs to be passed to native-image at build time to achieve this silencing.

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 8, 2024

This issue appears to be stale because it has been open 30 days with no activity. This issue will be closed in 7 days unless Stale label is removed, a new comment is made, or not-Stale label is added.

@github-actions github-actions bot added the Stale label Mar 8, 2024
@jerboaa jerboaa added not-Stale and removed Stale labels Mar 8, 2024
@jerboaa
Copy link
Collaborator Author

jerboaa commented Mar 21, 2024

I've tested this with the GraalVM for JDK 22 Community release here (GA released yesterday):
https://github.com/graalvm/graalvm-ce-builds/releases/tag/jdk-22.0.0

Seems to have the same issue:

$ rm -rf META-INF && mkdir -p META-INF/native-image && /disk/graal/builds/graalvm-community-openjdk-22+36.1/bin/java -agentlib:native-image-agent=config-output-dir=META-INF/native-image --enable-native-access=ALL-UNNAMED FFM && /disk/graal/builds/graalvm-community-openjdk-22+36.1/bin/native-image -J--enable-native-access=ALL-UNNAMED --verbose -H:+UnlockExperimentalVMOptions -H:+ForeignAPISupport --no-fallback FFM
Test passed! List sorted: [car, cat, dog, mouse]
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/jni-config.json
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/reflect-config.json
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/resource-config.json
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/proxy-config.json
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/serialization-config.json
Apply file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/predefined-classes-config.json
Apply jar:file:https:///disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/library-support.jar!/META-INF/native-image/com.oracle.svm/thirdparty/native-image.properties
Apply jar:file:https:///disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/library-support.jar!/META-INF/native-image/com.oracle.svm/polyglot/native-image.properties
Executing [
HOME=/home/sgehwolf \
LANG=en_US.UTF-8 \
PATH=/home/sgehwolf/.local/bin:/home/sgehwolf/bin:/home/sgehwolf/Documents/eclipse/eclipse-2022/eclipse:/home/sgehwolf/.local/bin:/home/sgehwolf/bin:/home/sgehwolf/Documents/eclipse/eclipse-2022/eclipse:/usr/lib64/qt-3.3/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin \
PWD=/home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example \
USE_NATIVE_IMAGE_JAVA_PLATFORM_MODULE_SYSTEM=true \
/disk/graal/builds/graalvm-community-openjdk-22+36.1/bin/java \
-XX:+UseParallelGC \
-XX:+UnlockExperimentalVMOptions \
-XX:+EnableJVMCI \
-Dtruffle.TrustAllTruffleRuntimeProviders=true \
-Dtruffle.TruffleRuntime=com.oracle.truffle.api.impl.DefaultTruffleRuntime \
-Dgraalvm.ForcePolyglotInvalid=true \
-Dgraalvm.locatorDisabled=true \
-Dsubstratevm.HostLibC=glibc \
--add-exports=java.base/com.sun.crypto.provider=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.access=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.event=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.loader=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.logger=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.misc=org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile,org.graalvm.nativeimage.pointsto \
--add-exports=java.base/jdk.internal.module=org.graalvm.nativeimage.base,org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.org.objectweb.asm=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.perf=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.platform=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.ref=org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile \
--add-exports=java.base/jdk.internal.reflect=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.util=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.vm.annotation=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal.vm=org.graalvm.nativeimage.builder \
--add-exports=java.base/jdk.internal=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.invoke.util=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.net.www=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.net=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.nio.ch=org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile \
--add-exports=java.base/sun.reflect.annotation=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.reflect.generics.factory=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.reflect.generics.reflectiveObjects=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.reflect.generics.repository=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.reflect.generics.scope=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.reflect.generics.tree=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.security.jca=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.security.provider=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.security.ssl=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.security.util=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.security.x509=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.text.spi=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util.calendar=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util.cldr=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util.locale.provider=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util.locale=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util.resources=org.graalvm.nativeimage.builder \
--add-exports=java.base/sun.util=org.graalvm.nativeimage.builder \
--add-exports=java.management/com.sun.jmx.mbeanserver=org.graalvm.nativeimage.builder \
--add-exports=java.management/sun.management=org.graalvm.nativeimage.builder,org.graalvm.nativeimage.pointsto \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.aarch64=jdk.graal.compiler,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.amd64=jdk.graal.compiler,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.code.site=jdk.graal.compiler,org.graalvm.nativeimage.builder \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.code.stack=jdk.graal.compiler,org.graalvm.nativeimage.builder \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.code=jdk.graal.compiler,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile,org.graalvm.nativeimage.pointsto,org.graalvm.truffle.compiler \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.common=jdk.graal.compiler,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.pointsto \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot.aarch64=jdk.graal.compiler \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot.amd64=jdk.graal.compiler \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot.riscv64=jdk.graal.compiler \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot=jdk.graal.compiler,org.graalvm.nativeimage.builder \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.meta=jdk.graal.compiler,org.graalvm.nativeimage.base,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.objectfile,org.graalvm.nativeimage.pointsto,org.graalvm.truffle.compiler \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.riscv64=jdk.graal.compiler,org.graalvm.nativeimage.builder \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.runtime=jdk.graal.compiler,org.graalvm.nativeimage.builder,org.graalvm.nativeimage.pointsto \
--add-exports=jdk.internal.vm.ci/jdk.vm.ci.services=jdk.graal.compiler,org.graalvm.nativeimage.builder \
--add-exports=jdk.jfr/jdk.jfr.events=org.graalvm.nativeimage.builder \
--add-exports=jdk.jfr/jdk.jfr.internal.event=org.graalvm.nativeimage.builder \
--add-exports=jdk.jfr/jdk.jfr.internal.jfc=org.graalvm.nativeimage.builder \
--add-exports=jdk.jfr/jdk.jfr.internal=org.graalvm.nativeimage.builder \
--add-exports=jdk.management/com.sun.management.internal=org.graalvm.nativeimage.builder \
-XX:+UseJVMCINativeLibrary \
-Xss10m \
-XX:MaxRAMPercentage=47.64222951120384 \
-XX:GCTimeRatio=9 \
-XX:+ExitOnOutOfMemoryError \
-Djava.awt.headless=true \
'-Dorg.graalvm.vendor=GraalVM Community' \
-Dorg.graalvm.vendorurl=https://www.graalvm.org/ \
'-Dorg.graalvm.vendorversion=GraalVM CE 22+36.1' \
-Dorg.graalvm.version=24.0.0 \
-Dcom.oracle.graalvm.isaot=true \
-Djava.system.class.loader=com.oracle.svm.hosted.NativeImageSystemClassLoader \
-Xshare:off \
-Djdk.reflect.useOldSerializableConstructor=true \
-Djdk.internal.lambda.disableEagerInitialization=true \
-Djdk.internal.lambda.eagerlyInitialize=false \
-Djava.lang.invoke.InnerClassLambdaMetafactory.initializeLambdas=false \
-Djava.lang.invoke.MethodHandle.DONT_INLINE_THRESHOLD=-1 \
-Djava.lang.invoke.MethodHandle.PROFILE_GWT=false \
--enable-native-access=ALL-UNNAMED \
--add-modules=ALL-DEFAULT \
--module-path \
/disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/builder/native-image-base.jar:/disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/builder/svm-foreign.jar:/disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/builder/pointsto.jar:/disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/builder/objectfile.jar:/disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/builder/svm.jar \
--module \
org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner \
-keepalive \
/proc/93559/comm \
-imagecp \
/home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example \
-imagemp \
/disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/library-support.jar \
-H:CLibraryPath=/disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/clibraries/linux-amd64 \
-H:Path@driver=/home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example \
-H:+UnlockExperimentalVMOptions@user \
-H:+ForeignAPISupport@user \
-H:FallbackThreshold@user+api=0 \
-H:-UnlockExperimentalVMOptions@user \
-H:JNIConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/jni-config.json+api=META-INF/native-image/jni-config.json \
-H:ReflectionConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/reflect-config.json+api=META-INF/native-image/reflect-config.json \
-H:ResourceConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/resource-config.json+api=META-INF/native-image/resource-config.json \
-H:DynamicProxyConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/proxy-config.json+api=META-INF/native-image/proxy-config.json \
-H:SerializationConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/serialization-config.json+api=META-INF/native-image/serialization-config.json \
-H:PredefinedClassesConfigurationResources@file:https:///home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/META-INF/native-image/predefined-classes-config.json+api=META-INF/native-image/predefined-classes-config.json \
'-H:Class@explicit main-class=FFM' \
'-H:Name@main-class lower case as image name=ffm' \
-H:ImageBuildID@driver=72e4cf08-08a7-37d1-140d-369e254b8717 \
'-H:Features@jar:file:https:///disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/library-support.jar!/META-INF/native-image/com.oracle.svm/thirdparty/native-image.properties+api=com.oracle.svm.thirdparty.gson.GsonFeature' \
'-H:Features@jar:file:https:///disk/graal/builds/graalvm-community-openjdk-22+36.1/lib/svm/library-support.jar!/META-INF/native-image/com.oracle.svm/polyglot/native-image.properties+api=com.oracle.svm.polyglot.groovy.GroovyIndyInterfaceFeature,com.oracle.svm.polyglot.scala.ScalaFeature' \
-H:Color@driver=always \
-H:+BuildOutputProgress@driver \
-H:+BuildOutputLinks@driver
]
========================================================================================================================
GraalVM Native Image: Generating 'ffm' (executable)...
========================================================================================================================
[1/8] Initializing...                                                                                    (2.8s @ 0.19GB)
 Java version: 22+36, vendor version: GraalVM CE 22+36.1
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 13.2.1)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 1 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
------------------------------------------------------------------------------------------------------------------------
 1 experimental option(s) unlocked:
 - '-H:+ForeignAPISupport' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 26.49GB of memory (42.3% of 62.55GB system memory, determined at start)
 - 12 thread(s) (100.0% of 12 available processor(s), determined at start)
[2/8] Performing analysis...  [*****]                                                                   (10.9s @ 0.47GB)
    4,369 reachable types   (76.4% of    5,715 total)
    4,695 reachable fields  (42.9% of   10,950 total)
   20,207 reachable methods (49.2% of   41,074 total)
    1,413 types,   231 fields, and   959 methods registered for reflection
       57 types,    57 fields, and    52 methods registered for JNI access
       0 foreign downcalls registered
        4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                               (2.1s @ 0.47GB)
[4/8] Parsing methods...      [*]                                                                        (1.1s @ 0.41GB)
[5/8] Inlining methods...     [***]                                                                      (0.9s @ 0.45GB)
[6/8] Compiling methods...    [***]                                                                     (10.9s @ 0.39GB)
[7/8] Laying out methods...   [*]                                                                        (1.8s @ 0.49GB)
[8/8] Creating image...       [*]                                                                        (1.9s @ 0.56GB)
   7.35MB (42.87%) for code area:    11,712 compilation units
   9.52MB (55.56%) for image heap:  115,037 objects and 47 resources
 276.72kB ( 1.58%) for other data
  17.13MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
   5.89MB java.base                                            2.18MB byte[] for code metadata
1012.93kB svm.jar (Native Image)                               1.50MB byte[] for java.lang.String
 113.51kB java.logging                                         1.08MB java.lang.String
  64.54kB org.graalvm.nativeimage.base                         1.02MB java.lang.Class
  47.59kB jdk.proxy1                                         388.11kB heap alignment
  45.84kB jdk.proxy3                                         375.46kB com.oracle.svm.core.hub.DynamicHubCompanion
  35.76kB FFM                                                286.56kB byte[] for general heap data
  26.87kB jdk.internal.vm.ci                                 260.48kB java.util.HashMap$Node
  22.07kB org.graalvm.collections                            219.66kB java.lang.Object[]
  11.42kB jdk.proxy2                                         219.03kB java.lang.String[]
  20.18kB for 4 more packages                                  2.04MB for 1207 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
 CPU:  Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
                        1.4s (4.0% of total time) in 173 GCs | Peak RSS: 1.00GB | CPU load: 8.93
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
 /home/sgehwolf/Documents/mandrel/bugs/upstream-mandrel/ffm_jdk22_example/ffm (executable)
========================================================================================================================
Finished generating 'ffm' in 33.6s.
$ ./ffm 
Exception in thread "main" com.oracle.svm.core.foreign.ForeignFunctionsRuntime$UnregisteredForeignStubException: Cannot perform downcall with leaf type (long,long,long,long,long)void as it was not registered at compilation time.
	at org.graalvm.nativeimage.foreign/com.oracle.svm.core.foreign.ForeignFunctionsRuntime.getDowncallStubPointer(ForeignFunctionsRuntime.java:75)
	at org.graalvm.nativeimage.foreign/com.oracle.svm.core.foreign.NativeEntryPointInfo.makeEntryPoint(NativeEntryPointInfo.java:88)
	at java.base@22/jdk.internal.foreign.abi.NativeEntryPoint.make(Target_jdk_internal_foreign_abi_NativeEntryPoint.java:60)
	at java.base@22/jdk.internal.foreign.abi.DowncallLinker.getBoundMethodHandle(DowncallLinker.java:87)
	at java.base@22/jdk.internal.foreign.abi.x64.sysv.CallArranger.arrangeDowncall(CallArranger.java:132)
	at java.base@22/jdk.internal.foreign.abi.x64.sysv.SysVx64Linker.arrangeDowncall(SysVx64Linker.java:62)
	at java.base@22/jdk.internal.foreign.abi.AbstractLinker.lambda$downcallHandle0$0(AbstractLinker.java:106)
	at java.base@22/jdk.internal.foreign.abi.SoftReferenceCache$Node.get(SoftReferenceCache.java:49)
	at java.base@22/jdk.internal.foreign.abi.SoftReferenceCache.get(SoftReferenceCache.java:38)
	at java.base@22/jdk.internal.foreign.abi.AbstractLinker.downcallHandle0(AbstractLinker.java:103)
	at java.base@22/jdk.internal.foreign.abi.AbstractLinker.downcallHandle(AbstractLinker.java:85)
	at FFM.main(FFM.java:37)
	at java.base@22/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)

So I don't think we can do any more on the Mandrel side.

@jerboaa
Copy link
Collaborator Author

jerboaa commented Mar 21, 2024

I've opened oracle#8616 upstream to track this.

@jerboaa jerboaa added pending upstream Issue or PR pending action on upstream repository (oracle/graal) affects/24.0 labels Mar 21, 2024
@jerboaa
Copy link
Collaborator Author

jerboaa commented Mar 28, 2024

I've opened oracle#8616 upstream to track this.

The tracker for this upstream is oracle#8113. Documentation as to what works and what not is at:
https://www.graalvm.org/jdk22/reference-manual/native-image/native-code-interoperability/foreign-interface/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects/24.0 bug Something isn't working not-Stale pending upstream Issue or PR pending action on upstream repository (oracle/graal)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jerboaa