Add Deprecation Process DIP

[JackStouffer]

cc @wilzbach @andralex @CyberShadow

[CyberShadow]

Thanks!

Recent debate on using dates vs. versions for deprecation deadlines:
dlang/phobos#6086 (comment)

CC @jmdavis, the deprecations czar.

[wilzbach]

Wow, excellent initial draft.

Here's what I miss:

• the one who introduces the deprecation has to ensure that it's not used anywhere in the core repositories (while this is mostly enforced with -de now, it might be good to be explicit).
• similarly, I think we should mention the Project Tester
• language features deprecations/PRs should update https://dlang.org/deprecate.html
• how deprecations are marked in the source code (let's standardize @@@DEPRECATED_2.082@@@)

[wilzbach]

We have the https://docarchives.dlang.io which are index by Google too, so we could do this earlier.

[wilzbach]

Maybe add something like:

This changelog entry should have a short motivation for the deprecation (or removal) and needs to describe what actions are required by the user to upgrade their codebase.

[wilzbach]

Imho the entire idea of -transition is nice, but completely failed in the past.
See e.g. the story of -transition=complex.
tl;dr: -transition=complex existed for 1.5 years and one couldn't even use standard library with it until recently dlang/dmd#7640.

Hence, I suggest to drop the idea of -transition for future deprecation and do the same as we do in Phobos and
have recently done in dmd too(https://github.com/dlang/dmd/pull/7760/files#diff-90e876a104f01c31442b3b307953221cR1173): just trigger the deprecation.

[wilzbach]

We don't follow SemVer at the moment. With the closest approximation (2.B.C), bumping B is called a minor release 😱

However, @MartinNowak plans to change the release process soon. It would mean that the meaning of "release" changes from every two months to six months. So we need to be a bit careful here. OTOH four versions is a good value for now and if Martin's release process change (DIP?) goes through, this can be easily adjusted.
Though, I really like our current bi-monthly releases cycle, it's a good measure of stable and getting new stuff every now and then.

[andralex]

cc @mdparker

[jmdavis]

This is way too fast. For years now, the policy has been to have a symbol deprecated but documented for a year and then deprecated and undocumented for a year. That has gotten complaints both that it is too quick and that it is too long, but it's tried to balance that. It's long enough that even projects that are rarely maintained have a decent chance to be updated but not so long that deprecated stuff is around forever.

As it stands, four major releases is only going to be something like 8 - 10 months - maybe 12 if we run into several delayed releases. And one release of being deprecated before being removed from the docs means that anyone who is not completely on top of things is just going to find the documentation missing the next time they rebuild their project. Even if we all agreed that the current process is too slow, I think that this is clearly too fast - especially when you consider that plenty of our users aren't constantly updating their compilers. Heck, with the documentation only being there for one major release, it'll probably be gone before the ldc guys update to that release, since they're frequently about a major release behind.

This should probably be more like 5 major releases as deprecated and documented and 5 as deprecated and undocumented with the caveat that it can be adjusted in either direction if deemed appropriate by the development team but that it should only be done so with very good reason (e.g. if having the symbol around is causing problems, it should probably go sooner, but that's rarely the case).

[mdparker]

That's a debate to have in the community review, methinks.

[jmdavis]

It's 2018 now.

[jmdavis]

Also, if we're deprecating a symbol, because we're replacing it, if possible, the new symbol should be introduced at least one release prior to deprecating the old one so that it's possible to build your project without deprecation messages on both the latest release and master.

[JackStouffer]

Addressed comments and added more details about language changes and exceptions to these rules.

[jmdavis]

ends up

[jmdavis]

so as to

[wilzbach]

Typo: Period

[JackStouffer]

Do you mean this sentence should be broken up?

[jmdavis]

No, you misspelled period. You wrote preiod.

[JackStouffer]

Oh, ha. Totally missed that.

[jmdavis]

I did too until Seb pointed it out.

[jmdavis]

ten deprecated with five undocumented would be better, since if we're actually on target for all releases, eight would only be 16 months rather than 24 like we do now. Ten would still only be 20 months if all releases are on time, but they get delayed frequently enough that that will probably push it into the 2 year range which is where we want to be. So, with 10 releases, we might be removing stuff a bit early, but with eight, it's pretty much a guarantee that we will unless releases start falling behind significantly.

[wilzbach]

How about adding a global summary table of the required actions?
Quick proposal:

action	when
add deprecation warning	immediately
add deprecation message to the docs	immediately
add deprecation changelog	immediately
remove deprecation from the docs	Fourth release
remove symbol + add final removal warnings	Fifth release

[wilzbach]

Walter doesn't like short urls, however I do like them and would love to use them. AFAICT we have never included a URL in a deprecation message. So we probably need to talk about this.

If short urls aren't possible, we need an alternative.
Idea: dep.dlang.io/<alias> (with e.g. redirect to a changelog entry)
I could setup dep.dlang.io with GH Pages, s.t. getting a new alias would be a matter of a simple one-line change PR.

[JackStouffer]

When I mentioned "pointer", I had in mind something like "see the 2.099 changelog for details"

[jmdavis]

Then you should probably give an example along those lines to make it clear.

[JackStouffer]

Ping @mdparker

[mdparker]

@JackStouffer I'm waiting on the new process document to be finalized before I move forward on any more DIPs. Waiting for feedback at the moment. Hopefully not much longer.

[wilzbach]

Mostly nits and ideas.

[wilzbach]

I think this will be the most controversial point of this DIP - especially because we a non-patch release might only be every six months soon. This should definitely get attention of @MartinNowak who plans to change this.

[jmdavis]

This is part of why we haven't been using releases for deprecations but have been using dates. Dates are consistent. Releases haven't been. The number 10 is based on the idea that we're getting a new major release approximately every other month but probably slip occasionally, so it's about 2 years. If we're going to change the release cycle, then this number isn't appropriate. And if we're not going to be at least reasonably consistent about when releases are, IMHO, it makes no sense whatsoever to use releases as the measurement for the deprecation cycle.

That being said, I hate the idea of major releases only coming out every 6 months. Not only would that suck from the standpoint of getting new stuff, but that effectively means that deprecations can only be introduced twice a year. And while ideally, we would only be deprecating stuff rarely, that could seriously delay us being able to get rid of something that we decide that we need to get rid of.

[JackStouffer]

Releasing new versions quickly/timely has been a huge benefit to D IMO, and has done great things for other large software projects like Firefox. I believe we move away from it at our detriment.

[mihails-strasuns-sociomantic]

Well, I can say for certain that in our case it is extremely unlikely to upgrade major compiler version more often than once in a year (as the process itself usually takes several months across different projects). With that in mind 2 years is a very useful term because it ensures that you can't possibly miss deprecation of some feature by skipping 1 year worth of versions.

Actual amount of major versions that cover that period does not matter that much in this context (less often major versions are beneficial for other reasons but it not within scope of this DIP).

[wilzbach]

Not only notified, they need to pass on the CI, which means that before removal of a feature the latest stable release of all projects on the Jenkins tester need to build without using this feature. Deprecations won't be shown, because almost all projects build without -de

[wilzbach]

FWIW while I successfully introduced -de to DMD, Phobos and Tools, DRuntime doesn't build with -de yet (dlang/druntime#1982)
Definitely not a requirement for this DIP, just an FYI.

[wilzbach]

I think an example of a function signature with a deprecation message would be good here.

[wilzbach]

Ouh so you were talking about this. As a user, this is a bit annoying.
OK, so I have to search for the changelog or function signature and hope I can find it. This takes at least three hops and is unnecessarily challenging.
I love the error messages from ShellCheck, e.g.

https://github.com/koalaman/shellcheck/wiki/SC2206

Can't we emit a similar deprecation message code? Phobos, dmd and druntime could use different leading numbers to avoid sync issues, but if numbers are too problematic because of merge conflicts (bumping the number), short keywords like DEPHEXSTRING or DEPENFORCEEX could work too?

[JackStouffer]

https://github.com/koalaman/shellcheck/wiki/SC2206

Maybe I'm missing something but I don't see anything about error messages on that page.

short keywords like DEPHEXSTRING or DEPENFORCEEX could work too?

That's basically just an error code, which really isn't helpful either. At the end of the day, to get details on the deprecation/migration path you still would have to go to google and type that in. In the case described in the DIP, at least there's a specific place the user is told to go to.

[jmdavis]

Normally, if you get a deprecation message, then you just look at the documentation for that symbol. I see no reason to complicate this with something like error codes (something that Walter has already expressed a dislike for BTW).

The deprecation message tells you that the symbol has been deprecated, and if the explanation is short enough, it also tells you what to use instead. If you want to know more, you look at the documentation for that symbol. For those who read the changelog, a changelog entry can give a heads up, but ultimately, it's the documentation on the symbol that's telling you what's going on. Let's please not complicate this.

[mihails-strasuns-sociomantic]

For that to work better, it would be convenient to provide persistent link to documentation entry for the specific version used.

[wilzbach]

For that to work better, it would be convenient to provide persistent link to documentation entry for the specific version used.

Do you mean that the changelog entries should use links like e.g. this one?

https://docarchives.dlang.io/v2.079.0/phobos/std_functional.html#binaryReverseArgs

[mihails-strasuns-sociomantic]

No, I mean actual deprecation messages printed by compiler. Less steps to take from message to relevant instructions -> more developers who actually read instructions as opposed to guessing -> less frustration

[quickfur]

What @mihails-strasuns-sociomantic requets requires (1) dlang.org keep an archive (lasting at least as long as the oldest deprecations) of all documentation for each release, and (2) maintain a consistent URL scheme that doesn't change over time. Both are good things, IMO, and we should definitely move in that direction. Then we can put actual URLs in the deprecation messages without the chaos that will ensue if the contents of the URL changes over time or if the URL itself becomes invalid over time. (Keep in mind that the URL must work even for previous releases of the compiler, so things like fixing a typo in the URL, etc., need to be done with care, e.g., with a webserver redirect in the right places.)

[wilzbach]

I think we need to clarify how the docs need to be updated too, e.g.

Deprecation: update the deprecatation list + add a warnings to the spec.

Error: remove from grammar + update deprecated list

[wilzbach]

s/preiod/period/

[JackStouffer]

Ping

[mdparker]

@JackStouffer Thanks for your patience. I'm moving on this stuff again. Once I get the older ones sorted, I'll get to this one. I ask that you first please update it to the layout in the new Template.md.

[wilzbach]

Does it need to be that long?
Once a symbol is removed from the documentation, the biggest cost is paid as it only triggers the deprecation messages, but no longer affects the learning process of newcomers (or other people looking at the doc for help).
If the argument is for the old symbols being searchable via Google, then there are now the docarchives.dlang.io with an indexed snapshot of every release + the changelog entries on dlang.org

Many projects also simply undocument their deprecated symbols with their removal (and sometimes put them on a special deprecated list) which imho is a better way - especially given that we constantly seem to forget to undocument the symbols - but I guess that's too much to ask for :/

[jmdavis]

Yes, and much as I hate to say it, I'm honestly sick of discussing this. This is the length that we came to after discussion on quite a few occasions and complaints both about it being too short and too long. We need to balance the ability to remove symbols that we don't want around anymore and not breaking existing code. Some folks in the past have pushed for never removing deprecated symbols, and both Andrei and Walter have expressed opinions along those lines at one time or another. For instance, at one point, Walter went to update one of his programs, and it didn't compile anymore, because some symbols had been deprecated and removed since the last time he'd updated it, and he was very unhappy about it. Those symbols had gone through the full two year deprecation process. Imagine how much more complaining we'd get along those lines if the length of the deprecation process were short. Every time that the length of the deprecation process comes up for discussion, I'm worried that Walter or Andrei is going to put their foot down and insist that it's not a problem to keep symbols around forever, and I really don't want to see that happen.

Various devs (usually those who are constantly up-to-date with the compiler that they use and probably who use master most of the time) like the idea of quickly deprecating stuff and then removing it, whereas other folks aren't as on top of things, and for them, fast deprecations and removals cause them problems. And the fact that ldc or gdc are not always being caught up with where dmd is just make that worse. Those needs need to be balanced, and if anything, over time, as our user base has grown and the amount of D code that's been written has grown, the need to avoid breaking code has grown.

After trial and error, we came to the process of having symbols deprecated and documented for one year and then deprecated and undocumented for one year, and over all, that's worked fairly well. It balances the needs of being able to remove old symbols while not breaking code, and it balances the need for folks to look at the documentation for the symbols they're replacing and the need to discourage the use of such symbols by not having them documented.

Yes, if there's a symbol that we really want to get rid of, the length of the deprecation process can be annoying, but we shouldn't be deprecating symbols very frequently, because that's just going to make folks complain about how D isn't stable, and it's a sign that we keep screwing up what we put in Phobos. But at least we have a process for removing such symbols, even if it's not lightning fast. There are other languages, that tend to be far more rigid with their standard library, and some stuff sits there forever even if it's deprecated.

I won't claim by any means that our deprecation process is perfect, but I think that how long it is and how we deal with the documentation balances things fairly well.

[wilzbach]

FWIW I wasn't tackling the two-year deprecation period, but the period of how long the deprecated symbol still stays on dlang.org (I don't like the long deprecation period, but I wasn't proposing to change it).

Some folks in the past have pushed for never removing deprecated symbols, and both Andrei and Walter have expressed opinions along those lines at one time or another.

The State of D survey has different results: https://rawgit.com/wilzbach/state-of-d/master/report.html#180373330 (only 2% of the respondents said "never break code")

After trial and error, we came to the process of having symbols deprecated and documented for one year and then deprecated and undocumented for one year, and over all, that's worked fairly well.
...
it balances the need for folks to look at the documentation for the symbols they're replacing and the need to discourage the use of such symbols by not having them documented.

Well, the only thing I was questioning whether we can remove the symbol earlier from the docs as two major things changed:

1. docarchives.dlang.io
2. We require visible changelog entries for every deprecation

and it's a sign that we keep screwing up what we put in Phobos.

Well, two weeks ago I made a "small" list of things that I would change if we can do breaking changes in Phobos. Unfortunately it's a very long list and I stopped at point 135 and being not even through a quarter of Phobos's symbols.

Anyhow I don't want to block this DIP with this, I just wanted to say that I personally feel that symbols should be undocumented sooner as they are still well visible for people searching for then and removing them helps to avoid encouragement of using deprecated symbols.

[jmdavis]

Having access to older documentation helps, but if the symbols are undocumented too quickly, it becomes harder for folks updating their code to figure out what the old symbol did exactly so that they can properly update their code. Having them documented for a year and then undocumented for a year came out of trying to balance the desire to remove the documentation as quickly as possible so that folks definitely wouldn't use the symbol and the desire to keep the documentation around the entire time so that folks could easily find it to figure out how to update their code to not use it. There have been folks who have argued in favor of removing the documentation ASAP and others have argued for keeping it around longer. IIRC, at one point, Walter even wanted to have the documentation around after the symbol was gone. The current approach was the compromise between the two extremes.

Now, we didn't use to have a way to access older documentation, so that does arguably change things, but I'm not sure that it's a good idea to speed up removing the documentation, especially since most folks are going to look at the main documentation, and I don't think that very many folks even know about docarchives.dlang.io. Having older versions of the documentation available has been discussed previously, but I think that this is the first that I've heard about there being anywhere that had it other than downloading the zip file for the specific compiler version. How would someone even know to look at docarchives.dlang.io to find older documentation?

[mdparker]

"code or not" -> code.

[mdparker]

"remove use of" -> remove all uses of

[mdparker]

"effected" -> affected

[mdparker]

"on schedule" -> on-schedule

[mdparker]

"added to the top of the code" -> prefixed to the line or block of code

"to be removed/disabled" -> which is being deprecated.

[mdparker]

"Any projects that are tested on the Project Tester that are broken should also have their maintainers notified." ->
Maintainers of any projects which are tested on the Project Tester and are broken by the deprecation should also be notified.

[mdparker]

"catch" -> find

[mdparker]

"This comment allows code to be easily searched" -> This comment allows source files to be conveniently searched

[mdparker]

"All removals (or changes that make the code private/package) of public functions, types, and modules must be accompanied with a deprecation period." -> When public functions, types, and modules are removed, or their protection attributes are made more restrictive, the changes must be accompanied by a deprecation period.

I notice this leaves out variables. Would it be correct to say "public symbols" rather than "functions, types and modules"?

[mdparker]

"pointer" -> reference

[mdparker]

"the users using" -> users of

[CyberShadow]

@mdparker Any reason you're not using one review with multiple comments, and instead posting individual comments? (Everyone here gets one email for each comment above.)

[mdparker]

"availible un-deprecated" -> available without deprecation

[mdparker]

"to [undeaD] to allow" -> to the undeaD repository, thereby allowing

[mdparker]

"that existing user code depends on" -> upon which existing code depends

The dangling preposition rule in and of itself is nonsense, but in formal writing I do find it distracting.

[mdparker]

Simultaneously to what?

[JackStouffer]

The depreciation

[mdparker]

"pointer" -> reference

[mdparker]

"many user's code" -> much user code

[JackStouffer]

This seems like a style preference. "I crashed many people's cars." is a valid sentence.

[mdparker]

"which turns something into a warning"

I think some clarification is needed here. I had to go back and read it twice. Perhaps "which causes warnings to be emitted where none were before."

[mdparker]

No reason other than cluelessness. Apologies for the spam!

…

On Wed, Mar 28, 2018 at 5:07 PM, Vladimir Panteleev < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In DIPs/DIP1xxx-JS.md <#108 (comment)>: > +In order to facilitate on schedule deprecations, a comment of the format +`@@@DEPRECATED_[version]@@@` should be added to the top of the code to be removed/disabled. +This comment allows code to be easily searched before every release to +catch all planned deprecations. + +### Public Functions, Types, and Modules + +All removals (or changes that make the code `private`/`package`) of public functions, +types, and modules must be accompanied with a deprecation period. + +The symbol(s) must be marked using the `deprecated` keyword with a message containing +the planned removal period. A pointer to more information should also be added. E.g. +"See the 2.080 changelog for more details" or "See the function documentation for more +details". The documentation of the symbol(s) must be updated noting the +deprecation and removal plan. The documentation should contain information to help +the users using the symbol(s) transition their code away from the symbol(s). @mdparker <https://github.com/mdparker> Any reason you're not using one review with multiple comments, and instead posting individual comments? (Everyone here gets one email for each comment above.) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#108 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6EDJITr2sNjWpDN19K8ZO6Xije_t70ks5ti0TQgaJpZM4Rz--4> .

-- Mike Parker

[wilzbach]

FYI: I finally got around proposing a DEPRECATED ddoc macro for consistent and prettier formatting of of deprecation messages in the spec + making it easier to grep for still documented and deprecated symbols.

dlang/dlang.org#2308

Not sure if it's too late to incorporate this in the DIP.

[wilzbach]

I just bumped into NodeJS's deprecation page and the majority of the deprecations are "Documentation-only"

https://nodejs.org/api/deprecations.html

So if we ever want to deprecate something, but the cost would be too high or it would bring too little value, just undocumenting it might be an easy way out.
Might be worthwhile to think about this in the DIP.

[jmdavis]

So if we ever want to deprecate something, but the cost would be too high or it would bring too little value, just undocumenting it might be an easy way out.

The main problem with that is that you still have to maintain it, so it doesn't help much. It does reduce how much the symbol would be used, but it would still be used, and we'd still have to deal with it. So, IMHO, it doesn't really fix the problem. If it's really bad enough that it needs to be deprecated, then it should be deprecated. And if it's not bad enough for it to be deprecated, then why remove it from the docs? And if it's so heavily used that we don't consider it reasonable to deprecate it, then removing it from the docs is just going to make it harder for folks to figure out how to deal with it when they encounter it, which they will, if it's that heavily used.

[wilzbach]

And if it's so heavily used that we don't consider it reasonable to deprecate it, then removing it from the docs is just going to make it harder for folks to figure out how to deal with it when they encounter it, which they will, if it's that heavily used.

The NodeJS "documentation-only" approach is to leave the symbol in the doc, but mark is as deprecated permanently in the docs (that's why I thought that approach is interesting for things that we know are bad, but don't won't to trigger deprecation messages for it). Anyhow, it was just a pointer.

then why remove it from the docs?

So that new code that gets written doesn't use this.

[mdparker]

Alright, I think we've covered the bases for the Draft Review. @JackStouffer If you're ready to go, we can brand this as DIP 1013 and launch the Community Review on Monday.

[JackStouffer]

Sounds good