SAAS-295 - versioning implemented

[ChineduUzoka]

SAAS-295 - versioning implemented

• created a generic pipeline that creates and uploads a new versioned release to s3

Jira-#

Additional Notes

• changes are not published to repo
• this needs doing but was not part of the jira story

[crsimmons]

Why are all the tests being deleted?

This would break the pipeline that builds new versions of the release https://concourse.prod.smarsh.cloud/teams/alcatraz/pipelines/mongodb_release_pipeline

EDIT: just took a look at the jira ticket and this makes a bit more sense now. It sounds like you're going to be adding some tests in elsewhere as part of the new pipelines and the tests in here were collateral. Is this right?

When you get your new pipelines running let us (delivery) know so we can remove our existing pipeline and start consuming your releases instead.

[washkinazy]

Why is the version changing?

[washkinazy]

I'm confused by this..

• If there here is a team actively working in this and if they have tests running why would we break that functionality?
• If there is a team working in this and using this release, were they reached out to at all?
• Is anyone still using this release?

[washkinazy]

I don't see any updates to the README

[crsimmons]

I'm confused by this..

* If there here is a team actively working in this and if they have tests running why would we break that functionality?

* If there is a team working in this and using this release, were they reached out to at all?

* Is anyone still using this release?

To fill in some more details, we made some updates to this release and created a pipeline to build it as part of the Credit Suisse work. We intend to continue using the release in the Azure deployment now that Credit Suisse has ended and our focus has been moved over to there. On Azure we are deploying all of Enterprise Archive in a single pipeline so Mongo is a part of that.

We're building and testing the release here and then fetching it in this job in order to make it available to our azure pipelines that run on a different concourse.

[DanielJonesEB]

Hey @washkinazy and @ChineduUzoka, it seems like through a lack of communication/visibility we've ended up doing similar things. I can't get access to JIRA (shakes fist at Centrify), so I've got limited context. If I say anything silly, please assume ignorance rather than malice.

As @crsimmons mentioned, this repo is pipelined and used already, with a monotonically-increasing semver stored in the version branch. The pipeline publishes to S3, but for some reason we never had this pipeline publishing GitHub releases, which may have reduced the visibility of this repo's ongoing usage. The approach we've taken in the pipeline was broadly the same as for https://github.com/Smarsh/storm_bosh_release

@crsimmons mentioned that this release is in use in delivery-azure-pipelines, where we consume the latest release from the S3 bucket, and deploy it as part of one monolithic Enterprise Archive pipeline we're building.

It'd be nice to maintain the same level of test coverage (or greater!), and for those tests to take place before a final release is published and a new version number bumped. The needs of the delivery-azure-pipelines code is to consume a release from that bucket, with an incrementing semver. Beyond that, there are no hard requirements as to how it gets built, but we will need to make changes if the naming format changes, the location of the built artifact changes, or the version number changes.

I mentioned to @nimeshjm yesterday before we found this PR that it'd be good to get all the PaaS, Data Services, Delivery and SRE folks together to discuss what we're trying to achieve and how we're trying to achieve it. I can understand the frustration of doing some work, and then finding out someone else has already done something similar in a different fashion. I'm keen to avoid duplicating efforts or creating frustrating situations for anyone.

For the avoidance of doubt, it's only Storm and MongoDB that we had already pipelined. I don't think we did any others.

I can see the need for being able to have different release streams depending on the base data service version; that's something that was out-of-scope in our use case and so we didn't factor into the equation. At the risk of making things worse by offering unwanted opinions, I can imagine a situation where Smarsh will need maintenance branches to exist for given versions of the data service, and separate pipelines emitting separate non-conflicting artifacts. This should be compatible with semver-ing things; any time there's a new version of the data service that would break consumers behaviour, we can bump the major version, and start a new maintenance branch for any later hotfixes. Having the underlying data service version number in the artifact name is convenient for visibility, but may lead to added complexity in consuming code.

[crsimmons]

We have also pipelined https://github.com/Smarsh/bosh-nfsserver in a similar fashion. I can't think of any other beyond the two @DanielJonesEB mentioned above.

[DanielJonesEB]

Not sure how I forgot about that when my name is on at least 50% of the commits! Goes to show how information can easily fall through the cracks :)

[washkinazy]

There's a lot here and it's better to get on a call.
I'll set something up for Monday a few members of DS and both @DanielJonesEB and @crsimmons to get on the same page. We can have followups if needed.

[ChineduUzoka]

Hey @washkinazy and @ChineduUzoka, it seems like through a lack of communication/visibility we've ended up doing similar things. I can't get access to JIRA (shakes fist at Centrify), so I've got limited context. If I say anything silly, please assume ignorance rather than malice.

As @crsimmons mentioned, this repo is pipelined and used already, with a monotonically-increasing semver stored in the version branch. The pipeline publishes to S3, but for some reason we never had this pipeline publishing GitHub releases, which may have reduced the visibility of this repo's ongoing usage. The approach we've taken in the pipeline was broadly the same as for https://github.com/Smarsh/storm_bosh_release

@crsimmons mentioned that this release is in use in delivery-azure-pipelines, where we consume the latest release from the S3 bucket, and deploy it as part of one monolithic Enterprise Archive pipeline we're building.

It'd be nice to maintain the same level of test coverage (or greater!), and for those tests to take place before a final release is published and a new version number bumped. The needs of the delivery-azure-pipelines code is to consume a release from that bucket, with an incrementing semver. Beyond that, there are no hard requirements as to how it gets built, but we will need to make changes if the naming format changes, the location of the built artifact changes, or the version number changes.

I mentioned to @nimeshjm yesterday before we found this PR that it'd be good to get all the PaaS, Data Services, Delivery and SRE folks together to discuss what we're trying to achieve and how we're trying to achieve it. I can understand the frustration of doing some work, and then finding out someone else has already done something similar in a different fashion. I'm keen to avoid duplicating efforts or creating frustrating situations for anyone.

For the avoidance of doubt, it's only Storm and MongoDB that we had already pipelined. I don't think we did any others.

I can see the need for being able to have different release streams depending on the base data service version; that's something that was out-of-scope in our use case and so we didn't factor into the equation. At the risk of making things worse by offering unwanted opinions, I can imagine a situation where Smarsh will need maintenance branches to exist for given versions of the data service, and separate pipelines emitting separate non-conflicting artifacts. This should be compatible with semver-ing things; any time there's a new version of the data service that would break consumers behaviour, we can bump the major version, and start a new maintenance branch for any later hotfixes. Having the underlying data service version number in the artifact name is convenient for visibility, but may lead to added complexity in consuming code.

Hey @washkinazy and @ChineduUzoka, it seems like through a lack of communication/visibility we've ended up doing similar things. I can't get access to JIRA (shakes fist at Centrify), so I've got limited context. If I say anything silly, please assume ignorance rather than malice.

As @crsimmons mentioned, this repo is pipelined and used already, with a monotonically-increasing semver stored in the version branch. The pipeline publishes to S3, but for some reason we never had this pipeline publishing GitHub releases, which may have reduced the visibility of this repo's ongoing usage. The approach we've taken in the pipeline was broadly the same as for https://github.com/Smarsh/storm_bosh_release

@crsimmons mentioned that this release is in use in delivery-azure-pipelines, where we consume the latest release from the S3 bucket, and deploy it as part of one monolithic Enterprise Archive pipeline we're building.

It'd be nice to maintain the same level of test coverage (or greater!), and for those tests to take place before a final release is published and a new version number bumped. The needs of the delivery-azure-pipelines code is to consume a release from that bucket, with an incrementing semver. Beyond that, there are no hard requirements as to how it gets built, but we will need to make changes if the naming format changes, the location of the built artifact changes, or the version number changes.

I mentioned to @nimeshjm yesterday before we found this PR that it'd be good to get all the PaaS, Data Services, Delivery and SRE folks together to discuss what we're trying to achieve and how we're trying to achieve it. I can understand the frustration of doing some work, and then finding out someone else has already done something similar in a different fashion. I'm keen to avoid duplicating efforts or creating frustrating situations for anyone.

For the avoidance of doubt, it's only Storm and MongoDB that we had already pipelined. I don't think we did any others.

I can see the need for being able to have different release streams depending on the base data service version; that's something that was out-of-scope in our use case and so we didn't factor into the equation. At the risk of making things worse by offering unwanted opinions, I can imagine a situation where Smarsh will need maintenance branches to exist for given versions of the data service, and separate pipelines emitting separate non-conflicting artifacts. This should be compatible with semver-ing things; any time there's a new version of the data service that would break consumers behaviour, we can bump the major version, and start a new maintenance branch for any later hotfixes. Having the underlying data service version number in the artifact name is convenient for visibility, but may lead to added complexity in consuming code.

The goal here is to adopt a standard or framework across mongodb, kafka and elasticsearch bosh release pipelines, a standard which is pretty much finished in the context of how we bump versions and create releases. The mongodb, kafka and elasticsearch pipelines are effectively the same except for integration and smoke tests yet the skeleton of the pipeline remains the same. So we have one code base for managing any bosh release includes the integration of "bosh vendoring" releases.

I believe its a win win solution for all as its makes Bosh cicd far more manageable and tailored towards Smarsh's needs.

I also believe its important that we support only one deployment framework of any service be it storm, mongodb, elasticsearch etc etc. company wide and understand that tests are a major part of this obviously. There should be one source of truth rather than many i.e one ansible role or one puppet module.

I would love to be part of this discussion by the way

Regards