RPM Distributions delayed again?

Does anyone have any status on the RPM distributions? If I remember correctly, this was supposed to be a GA (1.0) feature. It was then pushed to 1.1, but then 1.0.1 got released at the 1.1 planned date and 1.1 pushed further to the right. Now it looks like even the ‘1.1’ tag got removed from the RPM issue in GitHub (OpenSearch / OpenSearch Dashboards - RPM distribution (X64) · Issue #27 · opensearch-project/opensearch-build · GitHub). Is there any information that can be shared about when we will get a RPM for OpenSearch similar to what has consistently been available for ElasticSearch? Thanks in advance for your insights.

2 Likes

Oh, this was disappointing news to me I was waiting for the 1.1 to release before upgrading from OpenDistro because the rpm support OpenSearch Project Roadmap · GitHub

x64 RPM is in 1.2 on Nov 9. ARM RPM is going to be in 1.3. As @oscark noted, it’s on the road map.

A lot of this was affected by needing to fork the clients and produce an output plugin for Logstash (see: OpenSearch Roadmap & Release Update). It’s nothing technical, it’s just a bottleneck in capacity. Good news is the team is automating at they go along so it should pay off later with the ability more rapidly release.

But I do hear you both - RPM and DEB are super vital to serious use for a lot of folks.

2 Likes

x64 deb will be added in 1.2 @searchymcsearchface?

@yuvaraj Yup. DEB x64 is in 1.2.

Can you please give us an update about the delivery plan of RPM?
Looks like this topic is moved to 1.3 release in Jan.22?

If this is the case, can you please provide a step-by-step manual how to create the RPM on local machine. (in my case for RHEL as target platform)

This is equally disappointing to me as it was two months ago. IMO it undermines the trust in OpenSearch to make changes to the version objectives a couple of days before it is supposed to release (twice!).

Basically I have postponed upgrading from Open Distro because I thought that the rpm will be release in the “next version” basically since OpenSearch was released, I will not wait for rpm to be released this time. I would have upgraded to 1.0 straight away if I had known that it would take ~6 months for rpms to get release.

There’s an old saying in Tennessee — I know it’s in Texas, probably in Tennessee — that says, fool me once, shame on — shame on you. Fool me — you can’t get fooled again.”

Hey folks. I hear you on this one - I get it that it can be frustrating. OpenSearch works on a release train model, so it looks like the RPM didn’t make it based on the timeline.

You might want to keep track of this via the issue if you aren’t already:

Let me first say that I find your communications great and from near as I can see on this board, you seem to be a great guy. Nothing of the below is all that important in the grand scheme of the universe and it certainly isn’t intended to be an attack or personal.

From Sep:

From Nov:

I was really excited at the notion of OpenSearch and I thought the actions of Elastic were disrespectful to the community that was a big part of their success. However, my enthusiasm for this fork has waned and continues to do so. With the continued prioritizations that can be seen, it seems like the name OpenSearch is a bit of a misnomer; AmazonSearch might be a better description.

Now, there is nothing wrong with that - clearly there are a lot of Amazon people contributing to the project, but I think when a feature as critical as RPM/installers, which you acknowledge is ‘super vital to serious use for a lot of folks’ are booted from 1.0, to 1.1, to 1.2, to 1.3, (to 1.never???), the project is saying that the open source community isn’t the target for this project. This isn’t some small feature about an idiosyncrasy of some off-the-beaten-path behavior - this is the primary means open source packages have been maintained and installed for decades, including from your fork of ElasticSearch.

The continued lack of priority for something as fundamental as a RPM/DEB installation for the largest platform (x86-64) is disappointing on a micro-level and as a commentary as to the state of the fork. At least in my admittedly limited experience, the responsiveness of the devs has been underwhelming (I did as you suggested and posted some comments and items in the referenced ticket back in June, but was ignored). I am sure there is a ton to communication happening inside the Amazon systems, but my experience is that it is absent in the open source facing tools. I sincerely hopes this changes for the better.

@justme OK- you make some really valid points here, and yeah, I think it needs further explanation. I wasn’t personally involved in the decision making but I was personally helping with the spec files for RPM last Thursday, so A LOT changed since that time.

I followed up with some folks closer to the decision and I was forwarded to a design review doc which was 12 pages (:flushed:) from 2 days ago. Let me try to summarize - the original idea was that it would just reuse the ODFE process which was based on gradle and a plugin called ospackage. It wasn’t ideal but made more sense when we were relying on the ES artifact vs our own artifact as we could easily bring that in. It also causes all the plugins to maintain their own workflow to create the plugin package for each release. Short/short: using the old process would be carrying forward technical debt and slowing the overall release process down until that technical debt was resolved.

Instead, the solution would be to move to a different tool, which would move the project in the net right direction and pay dividends in release velocity down the road.

There are a couple of more approvals to go through (open source reviews to confirm license compatibility of tools, etc.) but then we’ll publish the entire direction to the build repo. We’re also working on a markdown file that will let you generate your own RPM artifacts as a stop gap which will be put in the repo.

1 Like

I appreciate the response and your genuine effort to try and give some context. However, after reading your post, I can’t help but ask a few follow-up questions, if you don’t mind me probing. I have two areas of concern that pop into my head:

Area 1: Can I ask where this 12 page design document is? I have been trying to keep tabs on this, and with the exception of a couple of posts to the github issue in the last week (which made very little sense without the context in which they were making comments) there is no activity in that issue other than when they remove the label for the next release when it misses it. I am not accusing anyone of doing this in secret, but it appears there is significant direction happing behind the Amazon curtain and coordination of only Amazon employees. These types of behaviors will only serve to reinforce the ‘AmazonSearch’ instead of ‘OpenSearch’ moniker that the project at least stated it wanted to have and be. I suspect this is a symptom and not the only instance where this type of thing is occurring. If I am oblivious to the public forum/distribution list/whatever, then I apologize in advance for my ignorance. Is this something that you think can or will change, or has there been some rethinking on behalf of Amazon management and its role in this project?

Area 2: A little more uncomfortable, but how is it a feature that was supposed to ship months ago only has a design document written two days before the third release it was targeted for? And then to be written in a way that has significant technical debt? My general impression was that this kept getting slipped because you needed to do it correctly, which seems like a good technical call, but your description of the state of it two days before release is worrying. Again, I am not as concerned about this specific issue as what it may say about the trajectory of the project and prioritization of issues. I realize that everyone is trying their best against the list of priorities that are given to them to hammer out, but my point is that the priorities seem a little bent out of shape as I would expect for an open source project with the goals and aspirations of OpenSearch.

Admittedly, my two areas of concern are somewhat rhetorical. You seem to grasp some of the worrying signs from the perspective of the community and I perceive that you are a good intermediary into the Amazon side of the effort. I think you will more likely than not try to communicate such and try to find meaningful changes that could be made on both sides of the equation. I am relatively sure you will respond, but the progress and changes in the project over the next 3-6 months will speak louder, although I fully realize you may not be a decision maker in that process. I appreciate the opportunity to have an intelligent discourse on the issue. In any case, as I assume you are in the US, I wish you a good holiday next week!

Area 1: Yup, you’ll see a consolidated version posted as a markdown document - I was actually referring to that in the previous message. If there are things we can’t do (e.g. include an incompatible dependency), those parts are just dropped from the final version. You’re right context is important, so just dropping something in the repo would be puzzling. However, it’s a fair callout that more openness in these processes is better and I can honestly say I see more important decisions being made on github as the project has evolved. It’s not perfect yet, but the phrase “could this decision be made on github” is common in meetings.

Area 2: This was a moment where a final review didn’t go as planned. Generally there is a direction and then a final review. I’m sure you’ve had those moments where things look good initially and you’re just not satisfied with the end result.

Anyhow - RPM is a very active area of work. Glad to discuss it - I do know it’s important to a range of people.

Also, I’m not in the US, I’m in Canada. I appreciate the sentiment but my turkey day was well over a month ago. I’m here all next week :wink:

1 Like