How To Build k-NN Plugin?

Greetings all,

I appreciate if someone can share with us the process to build the k-NN plugin.

When I run “./gradlew build” I get the below error:

Could not find blas (missing: BLAS_LIBRARIES)

I actually installed blas openblas and lapack but none of them helped.

The above was done on Rocky Linux 8 (which is supposed yo be equivalent to CentOS 8 and RHEL 8)

Thanks

I believe that blas stuff is very new. There is an open issue on opensearch-build about how the libraries are going to be integrated. @jmazane might know more.

@asfoorial are you trying to get to the new faiss stuff or just wanting to build a stable version of the library?

I am trying to see how I can extend it. Step zero is to build a stable version of the plugin.

Ah, ok - glad to hear it! Try building off a branch, for example: GitHub - opensearch-project/k-NN at 1.1 - @jmazane is a developer on the team, so he probably can give you more pointers.

The way OpenSearch repos work is that the main branch is always the active development branch, so it could be broken, etc. Stable versions are in version labeled branches.

I tried… but got the below error.

FAILURE: Build failed with an exception.

  • What went wrong:
    A problem occurred configuring root project ‘opensearch-knn’.

Could not resolve all artifacts for configuration ‘:classpath’.
Could not find org.opensearch.gradle:build-tools:1.1.0-SNAPSHOT.
Searched in the following locations:
- https://repo.maven.apache.org/maven2/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/maven-metadata.xml
- https://repo.maven.apache.org/maven2/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/build-tools-1.1.0-SNAPSHOT.pom
- https://plugins.gradle.org/m2/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/maven-metadata.xml
- https://plugins.gradle.org/m2/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/build-tools-1.1.0-SNAPSHOT.pom
- file:/home/hasan/.m2/repository/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/maven-metadata.xml
- file:/home/hasan/.m2/repository/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/build-tools-1.1.0-SNAPSHOT.pom
- https://jcenter.bintray.com/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/maven-metadata.xml
- https://jcenter.bintray.com/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/build-tools-1.1.0-SNAPSHOT.pom
Required by:
project :

Finally I got it working. Switching to 1.1 fixed one thing. But then I had to build OpenSearch and push it to a local maven repo. Also, it 9nly accepted compiling with JDK14.

Glad to hear you got it. Yeah, I’ve been bitten by the JDK thing before.

After all the struggle I found out that the kNN plugin is not extensible! Basically I wanted to create a search plugin that utilizes classes from the kNN plugin. So I added extended.plugins=opensearch-knn to the plugin descriptor. Later I got an error saying it is not extensible.

Is there a way to make it extensible?

Hi @asfoorial ,

It looks like to make the plugin extensible, the k-NN plugin would need to implement the ExtensiblePlugin interface.

There is no a reason (I know of) why we shouldn’t be able to implement this interface, but I need to look into what this would require a little bit closer. I will create an issue on the Github repo and move discussion over to there.

Side note: jni is changing in the 1.2 release. I am planning on updating developer guide with better instructions. Sorry for the inconvenience when building.

Jack

This is good news. I look forward for having kNN becoming extensible. I am sure it will pave the way for several extension projects in the future. Already we have seen projects leveraging kNN in past meetings and this step will enable them to work within the OpenSearch ecosystem.

1 Like

Thats great to hear! I have not gotten a chance to look into it yet. I have been busy with 1.2 release, but will take a look after that.

1 Like