Logstash-oss with non-removable x-pack

Hi,

anyone else experienced that the logstash-oss version from the original elastic site contain x-pack stuff which to my understanding should not be in the oss packages?

  • install oss version from website (sidenote, oss doesn’t seem to be in the repository)
    curl https://artifacts.elastic.co/downloads/logstash/logstash-oss-6.7.0.rpm -o logstash-oss-6.7.0.rpm

  • when starting the oss version following error pops up:
    May 02 11:53:05 host logstash[8117]: [ERROR] 2019-05-02 11:53:05.366 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) no such file to load -- x-pack/logstash_registry

  • ok, then I thought, just let’s remove logstash x-pack plugin
    /usr/share/logstash/bin/logstash-plugin remove x-pack
    -> nope :thinking:
    WARNING: All illegal access operations will be denied in a future release wAs of 6.3+ X-Pack is part of the default distribution and cannot be uninstalled; to remove all X-Pack features, you must install the OSS-only distribution of Logstash. ERROR: Operation aborted, cannot remove plugin, message: x-pack not a removable plugin

  • it was not working either way, so I had to remove the x-pack plugin from ugly way:
    mv -v /usr/share/logstash/x-pack /tmp/

Maybe I have overlooked something or maybe elastic doesn’t have a highest interest in creating functional oss packages?

thanks,
nean

Update:
i created an issue at github:

also I found another workaround:
one can deactivate/comment load_xpack unless LogStash::OSS in the code/file /usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb like below

def setup!
  #load_xpack unless LogStash::OSS
  load_available_plugins
  execute_universal_plugins
end

Update:
Elastic applied patches -> https://github.com/elastic/logstash/issues/10783 ,
However, fully fixed OSS packages will be available only as of version 6.8.1 and 7.2 , unfortunately those versions are quite ahead and not yet supported by ODfE.

Currently, with ODfE 1.0.0 and Elasticsearch/Logstash 7.0.1 one has to manually apply the hacks to get rid of the issues caused by the x-pack code fragments.
In addition ilm - index lifecycle management is exclusively for x_pack (a x-pack feature) and causing issues on 7.0.1, so its recommended to disable it in the logstash config:

output {
  elasticsearch { 
    hosts =>  [.. ]
    ilm_enabled => false
  }
}

As I could figure out, when using OSS packages (also >7.2), there is still x-pack related code in there, which probably slowly, or in worst case, never, get removed and might cause issues in the future.

Question to ODfE:
What is the strategy to limit/get rid of issues caused by x-pack code in OSS packages?

thanks,
nean