10x faster release with sbt-sonatype 3.6

#1

I’m glad to announce this good news for Scala library maintainers!

I’ve just released sbt-sonatype 3.6 https://github.com/xerial/sbt-sonatype, which improved the upload performance to Sonatype significantly faster.

Previously, publishSigned task took a long time if you have hundreds of small artifact files in your projects. By using a local staging repository and bundle upload to Sonatype, thousands of files can be uploaded in several minutes. This will reduce the pain of maintaining cross-build projects that might have multiple sub-modules.

With sbt-sonatype 3.6, you only need two commands to publish your artifacts to Maven Central (Sonatype OSS repository):

  • publishSigned
  • sonatypeBundleRelease

sbt-sonatype 3.6 also added support for restarting your release from scratch by properly cleaning up the previously created stating repository at Sonatype. So even if your release fails for some reasons (e.g., network failures or outage of Sonatype API), you can safely restart the release process just by re-running sonatypeBundleRelease command.

Happy OSS programming!

4 Likes
#2

Wow! This sounds great!

Does this mean that you could run publishSigned from several different SBT processes and then run one sonatypeBundleRelease at the end? That would be really great for projects that need to publish for multiple versions of scala.js!

#3

For multi-module and cross-build projects, sbt-sonatype 3.4 will use the root project’s target/sonatype-staging folder for local staging, and release it to Sonatype at once. You don’t need to run multiple SBT processes.

#4

For more clarity, here is my example script to upload multiple projects including Scala 2.11, 2.12, 2.13 and Scala.js https://github.com/wvlet/airframe/blob/f20e50bc18a7f45436c7e5f8283232728b82e3ba/scripts/publish.sh#L14

#5

Sorry, I should have been clearer.

In projects like ScalaCheck the way to publish the project for multiple scala.js versions is to use an environment variable to control which version of the SBT plugin is used, and then launch SBT multiple times with different environment variables. Here’s a link to the plugin.sbt file that controls this.

This means that to do a ScalaCheck release I currently have to run SBT multiple times with different environment variables (e.g. SCALAJS_VERSION=0.6.28), and then manually use the Sonatype interface to finish the process. Could I use sonatypeBundleRelease to do this last step instead?

#6

Thanks. If it’s possible to consolidate all of your artifacts into a single local staging repository after running multiple SBT tasks, sonatypeBundleRelease can be used as the last step.

1 Like
#7

@non It would be better to change this setting to use the same local folder from multiple SBT tasks:

sonatypeBundleDirectory := (ThisBuild / baseDirectory).value / target.value.getName / "sonatype-staging" / s"${name.value}-${version.value}"
1 Like
#8

Now sbt-sonatype 3.6, including fixes for supporting complex multi-module projects and projects using sbt-dynver, is available.

#9

Note that the bundle mechanism was originally proposed by pshirshov, many thanks! :pray:

1 Like
#10

The bundle upload mechanism itself has been available in Sonatype since more than 5 years ago, and it has been used by many Maven users. Unfortunately, however, nobody in Scala community could connect these dots to sbt until pshirshov mentioned that approach in sbt and sbt-sonatype GitHub issues. Initially, I didn’t fully understand why this approach can be so fast.

I was somehow forced to work on sbt-sonatype because the performance of Sonatype API is becoming quite slow; some projects that could be released in 10 minutes a year ago takes 1 hour or more recently. I was quite reluctant to work on this 4-year old code of sbt-sonatype, but to mitigate the pain I added sonatypePrepare task to make the release cycle retryable, and shipped sbt-sonatype 3.0.

Aha! moment came later. After checking some existing GitHub tickets, I noticed the idea of Pshirshov again and tried to understand how it works by looking at the code of his Ant script.

The bundle upload code was surprisingly simple! So I’ve released sbt-sonatype 3.1 with the bundle upload support in a day. (The current version is sbt-sonatype 3.6, which fixes several minor bugs)

I’ve been complaining about sbt and sonatype for the slow publish performance. But blaming them was totally wrong. We simply didn’t know how to use Sonatype API appropriately. (To be fair, it was quite difficult to understand what this API endpoint does from this doc: https://repository.sonatype.org/nexus-staging-plugin/default/docs/path__staging_bundle_upload.html)

It seems more than 4K+ projects are using sbt-sonatype: https://github.com/search?q=sbt-sonatype&type=Code. I think this new release of sbt-sonatype will save a lot of time for these Scala developers.