Logstash OpenSearch Plugin Doc Instructions result in failure: logstash pipeline doesn't like colons

OpenDistro
1

Hi - there is a similar topic but the three problems/solutions identified we’re not the same as the one I have encountered.
Logstash-OSS with Opensearch Plugin Failed to Execute Action

I am following the instructions for the Logstash OpenSearch plugin:
Logstash OpenSearch Plugin Instructions

The OpenSearch container is up and running fine.

Then I try to start Logstash by copy/pasting the instructions and I get the same error mentioned in the other post. In this case I was able to narrow it down to the ‘:’ in the hosts config. It does not like that character. I see in the docs that OpenSearch index names can’t have ‘:’, but it should be fine for hosts. Perhaps there is a work around?

I’m running these commands from Windows 11 PowerShell. I have been able to get the Docker OpenSearch and Dashboards all up and running. Now I’m trying to learn Logstash so I can config log file ingestion.

Actual Error:

[2021-12-31T03:11:30,110][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [A-Za-z0-9_-], [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_], \",\", \"]\" at line 3, column 21 (byte 66) after output {\n   opensearch {\n     hosts => [https", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383:in `block in converge_state'"]}
[2021-12-31T03:11:30,269][INFO ][logstash.runner          ] Logstash shut down.
[2021-12-31T03:11:30,286][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

If I change hosts to the following it stops complaining:
hosts => [“localhost”]

Then it doesn’t like the special chars in the index either, like ‘.’ and ‘%’. So I change that to the following and the whole thing starts up fine:
index => “opensearch”

2

Humm. I’ve personally used these instructions and they’ve worked for me (famous last words). Are you passing the configuration with the -e like on the docs page or passing a configuration file?

3

I copy pasted so it’s using the -e. I took a different tac and followed the Logstash instructions from the elastic site and that is all working fine. So - don’t know why this doesn’t work but I’m past the issue now as I’m running based on a config file.

4

What instructions did you use?

5

I was following docker instruction steps 1-4 which gave me the trouble indicated above.

I pivoted from that and just wanted to learn Logstash so I followed the instructions here:
Elastic Logstash Instructions

6

OK - so, to be clear, you are doing something like logstash -f someconfig.conf and it’s working where the instructions using the -e are not?

7

Yes. Are you not able to duplicate the behavior from step 4? It’s just me?

8

That’s right - it works on Linux and Mac for sure. I’m 90% sure that the documentation isn’t accounting for different quoting or line end semantics in PowerShell.