I have created a slack destination with a webhook, which does not need any credentials,
however, when i try to test one of the triggers that i have, i can see that it goes to an error state
when I check the logs in elasticsearch i see an error similar to below trace
[2019-05-28T00:19:17,486][ERROR][c.a.o.a.d.f.SlackDestinationFactory] [master1] Exception publishing Message: DestinationType: SLACK, DestinationName:Test Alert Destination, Url: https://hooks.slack.com/services/blah/blah/blaaaaaaaaaaaaaaaaaah, Message: {"text":"Failed Login Attempt on \n\n Monitor Windows authentication failures just entered alert status. Please investigate the issue.\n- Trigger: Windows authentication failures - Trigger\n- Severity: 1\n- Period start: 2019-05-28T00:18:12.394Z\n- Period end: 2019-05-28T00:19:12.394Z"}
org.apache.http.conn.ConnectTimeoutException: Connect to hooks.slack.com:443 [hooks.slack.com/99.86.34.156] failed: connect timed out
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:150) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar:4.5.2]
at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.getHttpResponse(DestinationHttpClient.java:115) ~[alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.execute(DestinationHttpClient.java:80) ~[alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.factory.SlackDestinationFactory.publish(SlackDestinationFactory.java:42) [alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.factory.SlackDestinationFactory.publish(SlackDestinationFactory.java:29) [alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.lambda$publish$0(Notification.java:43) [alerting-notification-0.9.0.0.jar:?]
at java.security.AccessController.doPrivileged(Native Method) [?:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.publish(Notification.java:41) [alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.model.destination.Destination.publish(Destination.kt:167) [opendistro_alerting-0.9.0.0.jar:0.9.0.0]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner.runAction(MonitorRunner.kt:393) [opendistro_alerting-0.9.0.0.jar:0.9.0.0]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner.runMonitor(MonitorRunner.kt:196) [opendistro_alerting-0.9.0.0.jar:0.9.0.0]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner.runMonitor$default(MonitorRunner.kt:166) [opendistro_alerting-0.9.0.0.jar:0.9.0.0]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner$runJob$1.run(MonitorRunner.kt:160) [opendistro_alerting-0.9.0.0.jar:0.9.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.7.1.jar:6.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[?:?]
at java.net.Socket.connect(Socket.java:591) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2]
... 27 more
[2019-05-28T00:19:17,492][INFO ][c.a.o.a.m.MonitorRunResult] [master1] Internal error: org.apache.http.conn.ConnectTimeoutException: Connect to hooks.slack.com:443 [hooks.slack.com/99.86.34.156] failed: connect timed out. See the Elasticsearch.log for details
java.lang.IllegalStateException: org.apache.http.conn.ConnectTimeoutException: Connect to hooks.slack.com:443 [hooks.slack.com/99.86.34.156] failed: connect timed out
at com.amazon.opendistroforelasticsearch.alerting.destination.factory.SlackDestinationFactory.publish(SlackDestinationFactory.java:46) ~[alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.factory.SlackDestinationFactory.publish(SlackDestinationFactory.java:29) ~[alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.lambda$publish$0(Notification.java:43) ~[alerting-notification-0.9.0.0.jar:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.Notification.publish(Notification.java:41) ~[alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.model.destination.Destination.publish(Destination.kt:167) ~[opendistro_alerting-0.9.0.0.jar:0.9.0.0]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner.runAction(MonitorRunner.kt:393) ~[opendistro_alerting-0.9.0.0.jar:0.9.0.0]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner.runMonitor(MonitorRunner.kt:196) [opendistro_alerting-0.9.0.0.jar:0.9.0.0]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner.runMonitor$default(MonitorRunner.kt:166) [opendistro_alerting-0.9.0.0.jar:0.9.0.0]
at com.amazon.opendistroforelasticsearch.alerting.MonitorRunner$runJob$1.run(MonitorRunner.kt:160) [opendistro_alerting-0.9.0.0.jar:0.9.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.7.1.jar:6.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to hooks.slack.com:443 [hooks.slack.com/99.86.34.156] failed: connect timed out
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:150) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar:4.5.2]
at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.getHttpResponse(DestinationHttpClient.java:115) ~[alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.execute(DestinationHttpClient.java:80) ~[alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.factory.SlackDestinationFactory.publish(SlackDestinationFactory.java:42) ~[alerting-notification-0.9.0.0.jar:?]
... 15 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[?:?]
at java.net.Socket.connect(Socket.java:591) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar:4.5.2]
at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.getHttpResponse(DestinationHttpClient.java:115) ~[alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.client.DestinationHttpClient.execute(DestinationHttpClient.java:80) ~[alerting-notification-0.9.0.0.jar:?]
at com.amazon.opendistroforelasticsearch.alerting.destination.factory.SlackDestinationFactory.publish(SlackDestinationFactory.java:42) ~[alerting-notification-0.9.0.0.jar:?]
... 15 more
However,
I could post the same json payload to the webhook i have configured with below curl command from the same host which OP-Elasticsearch and Kibana was installed
[root@SourceHost ~]# curl -X POST -H "Content-Type: application/json" -d '{"text":"Failed Login Attempt on \n\n Monitor Windows authentication failures just entered alert status. Please investigate the issue.\n- Trigger: Windows authentication failures - Trigger\n- Severity: 1\n- Period start: 2019-05-28T00:18:12.394Z\n- Period end: 2019-05-28T00:19:12.394Z"}' https://hooks.slack.com/services/blah/blah/blaaaaaaaaaaaaaaaaaah
ok
[root@SourceHost ~]#
What could be the reason for this.