Logstash exited with code 1

Hi,

I am new to OpenSearch. I already have elastic and kibana up and running… now I want to pull log from api (using http poller) and push the log from logstash to elastic… but having this kind of problem.

logstash      | [2021-09-28T02:53:05,233][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash      | [2021-09-28T02:53:05,245][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash      | [2021-09-28T02:53:05,590][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"bf4885fc-a55a-4c28-895a-a885cc4db297", :path=>"/usr/share/logstash/data/uuid"}
logstash      | [2021-09-28T02:53:06,689][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
logstash      | [2021-09-28T02:53:07,097][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"if\", [A-Za-z0-9_-], '\"', \"'\", \"}\" at line 30, column 1 (byte 638) after output {\n  opensearch {\n    hosts => [\"https://odfe-node1:9200\"]\n    index => \"logstash-index-test\"\n    user => \"admin\"\n    password => \"admin\"\n    ssl => true\n    cacert => \"./logstash_pipeline/root-ca.pem\"\n    ilm_enabled => false\n  }\n\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:384:in `block in converge_state'"]}
logstash      | [2021-09-28T02:53:07,194][INFO ][logstash.runner          ] Logstash shut down.
logstash      | [2021-09-28T02:53:07,204][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
logstash      | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash      |         at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
logstash      |         at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
logstash      |         at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
logstash exited with code 1

my docker compose file

version: '3.7'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:1.13.2
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster
      - node.name=odfe-node1
      - discovery.seed_hosts=odfe-node1
      - cluster.initial_master_nodes=odfe-node1
      - bootstrap.memory_lock=true #
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" #
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 #
        hard: 65536
    volumes:
      - odfe-data1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:1.13.2
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_HOSTS: https://odfe-node1:9200
    networks:
      - odfe-net
  logstash:
    image: docker.elastic.co/logstash/logstash-oss:8.0.0-alpha2
    container_name: logstash
    volumes:
    - type: bind
      source: ./logstash_pipeline/
      target: /usr/share/logstash/pipeline
      read_only: true
    ports:
      - "5044:5044"
      - "80:80"
    networks:
      - odfe-net
    depends_on:
      - odfe-node1

volumes:
  odfe-data1:

networks:
  odfe-net:

** Logstash confiugration**

input{
  http_poller {
    urls => {
      authentication => {
        method => get
        url => "webapi/api/"
        headers => {
          "Content-Type" => "application/json"
        }
      }
    }
    request_timeout => 60
    schedule => { every => "5s"}
    codec => "json"
  }
}
output {
  opensearch {
    hosts => ["https://odfe-node1:9200"]
    index => "logstash-index-test"
    user => "admin"
    password => "admin"
    ssl => true
    cacert => "./logstash_pipeline/root-ca.pem"
    ilm_enabled => false
  }
}

can anybody help me with it?
thank you

You can’t use that version of Logstash - you need the OpenSearch Output Plugin.

Oh okay.

May I know, how the structure for logstash if I would like to send events to Logstash through HTTP? Is it the same or need changes?

Thank you.

Are you talking about logstash->OpenSearch/Open Distro?

Yes, for logstash configuration.

input{
  http_poller {
    urls => {
      authentication => {
        method => get
        url => "webapi/api/"
        headers => {
          "Content-Type" => "application/json"
        }
      }
    }
    request_timeout => 60
    schedule => { every => "5s"}
    codec => "json"
  }
}

These configurations are wrong, so which structure is the right one?.

Looking more deeply at your error message vs what you posted I noticed that your posted config is 27 lines long and the error message is complaining about an unexpected character on line 30. Do you have some sort of extra character in your config file?

There is nothing after line 27, but now I have another error regarding the logstash. Please refer to this error low below.

::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
logstash      | warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
logstash      | LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
logstash      |           create at org/logstash/execution/ConvergeResultExt.java:135
logstash      |              add at org/logstash/execution/ConvergeResultExt.java:60
logstash      |   converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:402
logstash      | [2021-10-04T03:42:06,313][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
logstash      | [2021-10-04T03:42:06,333][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:402:in `block in converge_state'"]}
logstash      | [2021-10-04T03:42:06,361][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
logstash      | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash      |         at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
logstash      |         at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
logstash      |         at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

My logstash configuration is still the same as above.

Really appreciate any help.

It’s hard to say - I don’t see anything that stands out, to be honest. I would try to reduce your surface area to eliminate variable.

I have the same problem. I am trying to use logstash latest version 7.16.1 with Opendistro 1.13.3 to overcome log4j vulnerability. I understand we need to have opensearch output plugin to make them work.

I created a docker image as:
FROM docker.elastic.co/logstash/logstash:7.16.1
RUN bin/logstash-plugin install logstash-output-opensearch

When applied via Helm charts, getting error:

[2021-12-15T07:35:45,752][ERROR][logstash.outputs.opensearch] Unknown setting ‘ilm_enabled’ for opensearch
[2021-12-15T07:35:45,835][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>“Java::JavaLang::IllegalStateException”, :message=>“Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.”, :backtrace=>[“org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)”, “org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)”, “org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)”, “org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)”, “org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)”, “org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)”, “org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)”, “usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)”, “org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)”, “org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)”, “org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)”, “org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)”, “org.jruby.RubyClass.newInstance(RubyClass.java:939)”, “org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)”, “org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)”, “usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)”, “usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)”, “org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)”, “org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)”, “org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)”, “usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383)”, “org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)”, “org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)”, “org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)”, “org.jruby.runtime.Block.call(Block.java:139)”, “org.jruby.RubyProc.call(RubyProc.java:318)”, “org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)”, “java.base/java.lang.Thread.run(Thread.java:829)”]}
warning: thread “Converge PipelineAction::Create” terminated with exception (report_on_exception is true):
LogStash::Error: Don’t know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:135
add at org/logstash/execution/ConvergeResultExt.java:60
converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:396
[2021-12-15T07:35:45,839][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>“Don’t know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>”}
[2021-12-15T07:35:45,846][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don’t know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>[“org/logstash/execution/ConvergeResultExt.java:135:in `create’”, “org/logstash/execution/ConvergeResultExt.java:60:in `add’”, “/usr/share/logstash/logstash-core/lib/logstash/agent.rb:396:in `block in converge_state’”]}
[2021-12-15T07:35:45,855][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]