Java UDF Functional Specification

Introduction

The term native UDF is used to distinguish UDFs which use the standard C linkage conventions. C, C++, and Delphi compilers can create UDF libraries (DLLs or Unix shared libraries) which use the C calling convention.

Native UDFs are deployed natively as shared UDF libraries (.dll or .so files), loaded dynamically as needed. Java UDFs are not deployed as a native shared library. Since native UDFs are encoded natively, native InterBase type representations are passed directly to/from native UDFs.

For example, a language such as Delphi can represent the C data type ISC_QUAD* directly using the Delphi syntax

type

   ISC_QUAD = record
     isc_quad_high : Integer ;
     isc_quad_low : Cardinal ;
   end;

PISC_QUAD = ^ISC_QUAD;

Such a pointer may be passed on the stack to the Delphi UDF in exactly the same way as it would be passed to a C UDF. So executing native UDFs for any compiled language supporting C data type representations and C calling conventions requires no additional support within the InterBase engine.

Java, on the other hand, is not compiled, and with no pointers or structures, has no language support for C data type representations, and does not support native C calling conventions. So executing Java UDFs requires communication with a Java Virtual Machine (JVM), and Java objects will need to be created within the InterBase engine in a portable way, independently of the object representation used by any particular JVM implementation.

The Java Native Interface (JNI) provides a mechanism for creating and manipulating portable Java objects as internal C structures and pointers. So for example, the representation for an 8-byte C date structure (ISC_QUAD*) can be converted by the engine to a java.sql.Date object before being passed to a Java UDF method. The java.sql.Date object representation would be created within the engine in a portable way via the JNI, but the actual underlying C representation for the object would be peculiar to whatever JVM implementation happens to be embedded. Similarly, the representation for an InterBase C string (char*) can be converted by the engine to a Java Unicode java.lang.String object before being passed to a Java UDF method.

So there are three key requirements for supporting Java UDFs from within InterBase

1. Provide support for describing Java UDF structures in so far as Java UDF descriptions differ from native UDF descriptions.
2. Provide support for converting InterBase native datatypes to/from JNI representations within the engine before passing/returning such representations to/from a Java UDF.
3. Provide a means to embed and configure a Java Virtual Machine (JVM) for interpreting Java UDFs during SQL execution.

The functional implications of these requirements are detailed in section User Interface/Usability, but may be summarized briefly as follows

• A DDL syntax to distinguish Java UDFs from native UDFs. Java UDF invocations (DML) will be syntactically identically with native UDF invocations, but must be distinguishable semantically. Furthermore, Java UDF descriptions are different from native UDF descriptions. For example, unlike native UDFs, Java UDFs are not known by a MODULE_NAME and ENTRY_POINT. Instead, Java UDF DDL must describe a Java UDF's class and method name. In addition, Java UDF DDL will use Java UDF datatype names, and will not employ descriptors such as FREE_IT, and BY VALUE, which are only meaningful in the context of native UDFs and pointer-based memory management.
• A system table for describing Java UDFs and associated metadata extract.
• A set of Java UDF datatypes and associated Java classes.
• An extended mechanism for InterBase error handling in support of recoverable Java exceptions.
• JVM configuration options, including the specification of a Java class path.
• An installation which deploys a Java Runtime Environment (JRE).

This specification assumes an understanding of native UDFs, and does not attempt to describe the meaning of UDFs in general, except where the behavior of Java UDFs differs from that of native UDFs.

Also note that Sun changed the name of Java 1.2 to Java 2, and JDK 1.2 to Java 2 SDK. These terms are now used synonymously.

Description

Support for Java UDFs (User Defined Functions) will allow for an external library of Java classes and methods to be utilized anywhere that a SQL function may be used. This provides for runtime SQL execution to perform data manipulation tasks by communicating directly with a Java Virtual Machine (JVM) local to the InterBase server. Through the use of the Java Native Interface (JNI) we can embed and use a Java VM in a standardized way that will work with any VM implementation supporting the JNI. All future evolutions of the JNI will maintain complete binary compatibility.

The recursive evaluation (execution) of SQL containing a Java UDF invocation will perform all necessary conversions of the UDF arguments and UDF return values from the InterBase native datatype structures to the corresponding data and character representations used by Java. For example, InterBase DATEs, TIMEs, and TIMESTAMPs will be converted to Java objects of class java.sql.Date, java.sql.Time, and java.sql.Timestamp respectively. Blob UDFs will be supported through the use of a customized public Java Blob library which invokes callbacks to the InterBase engine for performing gets and puts on Blob segments.

User Interface/Usability

When defining a Java UDF, there are two declarations to consider. The first declaration is for the actual Java Method in some external Java class library. The second declaration is for the UDF itself as declared to a database. Although the two declarations must correspond by the typing of their arguments and return value, they are nonetheless distinct declarations, and they will be referred to as the Java Method Declaration and the Java UDF Declaration.

For native UDF declarations, UDF type names, rather than C type names, are used to denote the types of UDF arguments and return values. For example,

DECLARE EXTERNAL FUNCTION foo
  BLOB,
// C type is a blob structure pointer *Blob
  CSTRING(n),
// C type is *char, InterBase type is CHAR or VARCHAR
  NUMERIC(n),
// C type is ISC_QUAD*
int or short,
//InterBase type is INT64, INTEGER, or SMALLINT.
  RETURNS TIMESTAMP
// C type is ISC_QUAD* from the ibase.h file
  ENTRY_POINT "C-function-name"
  MODULE_NAME "udflib.dll";

Similarly for Java UDFs, an extended class of UDF type names are used to denote the types of UDF arguments and UDF return values rather than the actual Java type names used in the Java Method Declaration. For example,

DECLARE EXTERNAL JAVA FUNCTION foo
  BLOB,
// Java type is class com.borland.interbase.Blob
  JSTRING(n),
// Java type is java.lang.String, InterBase type is CHAR or VARCHAR
  NUMERIC(n),
// Java type is java.math.BigInteger
  RETURNS TIMESTAMP
// Java type is java.sql.Timestamp
  CLASS "class-name"
  METHOD "method-name";

Although Java type names are not used in the Java UDF declaration, each UDF datatype corresponds strictly with a Java type or class.

When a Java UDF is invoked by InterBase, arguments are provided whose engine native types must be converted to the corresponding Java types. Because of the necessary datatype conversions from InterBase native structures to Java representations, Java UDF invocations must be distinguishable from native UDF invocations. Given a Java UDF declaration, it must also be possible [for the user] to infer the Java types of the corresponding method arguments and method return value. Therefore the SQL syntax for declaring a UDF must provide a means to indicate that the UDF is a Java UDF, as well as provide a means to indicate the Java UDF types of the UDF arguments and UDF return value.

DECLARE EXTERNAL JAVA FUNCTION udf-name
[ java-udf-datatype .,..]
[ RETURNS { java-udf-datatype
          | PARAMETER argument-position } ]
CLASS "class-name"
METHOD "method-name";

java-udf-datatype ::=
            JSTRING (maximum-character-length)
          | NUMERIC(p,s) | NUMERIC(p) | DECIMAL(p,s) | DECIMAL(p)
          | DATE | TIME | TIMESTAMP
          | BLOB
          | DOUBLE PRECISION
          | INTEGER
          | SMALLINT

package com.borland.interbase;
/**
 * This class represents a Blob as passed to a Java UDF.
 * A Blob UDF cannot open or close a Blob,
 * but instead invokes Blob methods to perform Blob access.
 * A UDF that returns a Blob does not actually define a return value.
 * Instead, the return-Blob must be passed as the last
 * input parameter to the UDF.
 **/

public class Blob

{
  /**
   * Read a Blob segment into a buffer, and return the number
   * of bytes read.
   **/

  public int getSegment (byte[] buffer)

  /**
   * Write a Blob segment of bytesToPut bytes from a buffer.
   **/

  public void putSegment (byte[] buffer, int bytesToPut);

  /**
   * Returns the total number of segments in the Blob.
   **/

  public long numberOfSegments ();

  /**
   * The size, in bytes, of the largest single segment in the Blob.
   **/

  public int maxSegmentLength ();

  /**
   * Returns the actual total size, in bytes, of the Blob.
   **/

  public long size ();

}

LOAD_JAVA_VIRTUAL_MACHINE TRUE

<interbase-dir>/java_udfs

JAVA_UDF_CLASSPATH c:interbasejava_udfs;d:fredsUdfsmathUdfs.jar

JAVA_UDF_NATIVE_LIBRARY_PATH d:fredsUdfNativeLibs

isc_dpb_load_java_virtual_machine
isc_dpb_java_udf_classpath
isc_dpb_java_udf_native_library_path

CONNECT "employee.gdb"
LOAD_JAVA_VIRTUAL_MACHINE JAVA_UDF_CLASSPATH "d:java_udfs";

JAVA_VIRTUAL_MACHINE_LIBRARY "c:\jdk1.2\jre\bin\classic\jvm.dll"

// Return a function pointer to the JNI function
// "AttachCurrentThread" in a variable JVM library.

     void *findAttachCurrentThread (char *jvmLibrary)

     {

       HINSTANCE hVM = LoadLibrary (jvmLibrary);
       if (hVM == NULL) return NULL;
       return GetProcAddress (hVM, "AttachCurrentThread");

     }

// Return a function pointer to the JNI function
// "AttachCurrentThread" in a variable JVM library.

     void *findAttachCurrentThread (char* jvmLibrary)

     {

       void *libVM = dlopen (jvmLibrary, RTLD_LAZY);
       if (libVM == NULL) return NULL;
       return dlsym (libVM, "AttachCurrentThread");

     }

Requirements and Constraints

The Java analog to the native UDF module name (.dll file) is the Java classpath. The Java classpath must be known in advance at the time the JVM is initialized. The Java classpath is not specified at UDF declaration time. Rather, the Java classpath is specified at server startup time. This has the disadvantage that the classpath is fixed for the life of the server, but has the advantage that if Java libraries are renamed, the Java classpath can be reconfigured without having to redeclare the UDF to the database. Whereas with native UDFs, the .dll module name is hardwired in the system tables for the UDF. So if a .dll module name changes, the native UDF must be redeclared to the database.

Like native UDFs, the invocation of a Java UDF will release the engine thread lock by performing a thread-exit before transferring control to the Java runtime (invoking the UDF). When the Java UDF returns, a thread-enter will be performed to regain the thread lock on the engine.

The JVM port must provide native Java thread support for the deployed platform. Therefore initial support for Java UDFs will be for Win32 only. Here's a quote from Sun's JNI FAQ (http://java.sun.com/products/jdk/faq/jnifaq.html ):

"The Solaris Java VM shipped with JDK 1.1 is not suitable for embedding into certain native applications. Because it depends on a user-level thread package to implement Java threads, the VM overrides a number of system calls in order to support non-blocking I/O. This may have undesirable affects on the hosting native application. In addition, the Invocation API function AttachCurrentThread is not supported on Solaris.

We plan to fix these problems in the near future by releasing a Java VM directly supported by Solaris native threads."

This non-native Java thread implementation was known as green threads.

But further information is now to be found at the new Java 2 JNI FAQ ( http://java.sun.com/products/jdk/faq/jni-j2sdk-faq.html#nativethreads ):

"8. (Solaris) Has support for native threads gotten any better? Yes. As of JDK/JRE 1.1.3 you could download a Solaris Native Threads Pack which was fully supported. In the Java 2 SDK, native threads is integrated into the release. If you use the invocation API on Solaris to embed the JVM into your application, we recommend the use of the native threads VM (see also Q11 on linker issues on Solaris).

Lest you ask, we have always used only native threads on the Win32 platforms."

This will need to be tested directly on Solaris for confirmation. Note that we must embed the native threads VM since green threads and native threads don't mix, and of course InterBase already links with -lthread -lc which is required for access to Solaris native threads.

Because of significant differences in the JNI API between Java 1 and Java 2, only Java 2 and above will be supported.

The JVM port must support the Java 2 JNI interfaces.

The blr and dyn generation for Java UDFs is deferred as a design consideration.

Thread-safety of Java UDFs is up to the author of the Java UDF class library. However, this is a relatively easy task in Java.

Performance of Java UDFs will be inferior to that of native UDFs because of the necessary internal conversions from native InterBase datatypes to Java objects.

Migration Issues

Although RDB$ENTRYPOINT and RDB$MODULE_NAME are nullable fields, the modified system table for RDB$FUNCTIONS could affect applications which do not allow for a null RDB$ENTRYPOINT or RDB$MODULE_NAME.

Open Issues

• Any way to force garbage collection of JVM by InterBase?
• Server-wide UDFs - this is a desirable feature for both native and java UDFs.

Syntax Conventions

The syntax diagram conventions mostly follow BNF, with a few variations to enhance readability. Here is a description of the general rules for specifying syntax in this extended BNF. Please be aware that BNF is a high-level specification syntax, and is not a low-level LALR(1) syntax as used by parser generators such as YACC. The LALR(1) syntax for Java UDFs is deferred as a design consideration. These rules are taken from the book "SQL Instant Reference" by Martin Gruber, Sybex Publishing.

• The symbol ::=means "is defined as". It is used to further clarify parts of a statement's syntax diagram.
• Keywords appear in all uppercase letters. These reserved words are literals that are actually written as part of the statement.
• Place holders for specific values, such as domain-name in the CREATE DOMAIN domain-name statement, appear in italic type. These place holders identify the type of value that should be used in a real statement; they are not literals to be written as part of the statement. This is not a standard BNF convention.
• Optional portions of a statement appear in square brackets ([ and ]).
• A vertical bar (|) indicates that whatever precedes it may optionally be replaced by whatever follows it.
• Braces ({ and }) indicate that everything within them is to be regarded as a whole for the purpose of grouping.
• Ellipses (...) indicate that the preceding portion of the statement may be repeated any number of times.
• Ellipses with an interposed comma (.,..) indicates that the preceding portion may be repeated any number of times, with the individual occurrences separated by commas. The final occurence should not be followed by a comma. Note: This is not a standard BNF convention; it is used for clarity and simplicity.
• Parenthesis () used in syntax diagrams are literals. They indicate that parenthesis are to be used in forming the statement. They do not specify a way of grouping the diagram as braces or square brackets do.

Reference Documents

• http://java.sun.com/products/jdk/faq/jnifaq.html
• http://java.sun.com/products/jdk/1.2/docs/guide/jni/spec/jniTOC.doc.html
• http://java.sun.com/products/jdk/1.2/docs/guide/jni/jni-12.html
• http://java.sun.com/products/jdk/1.2/jre/
• http://java.sun.com/products/jdk/1.2/jre/LICENSE
• http://java.sun.com/products/jdk/1.2/jre/README
• http://java.sun.com/products/jdk/1.2/runtime.html
• http://java.sun.com/products/jdk/faq/jni-j2sdk-faq.html
• Solaris Native Threads Pack
• The Java Native Interface: Programmer's Guide and Specification
• InterBase Data Definition Guide, Programmer's Guide, and Language Reference
• JNI Specification