WIP: Backports for 1.4-RC1 #34238

KristofferC · 2020-01-02T16:52:46Z

Backported PRs:

outline length assert in default_rng(tid) #34237 - outline length assert in default_rng(tid)
fix #33954, recursion through field types in is_derived_type #34223 - fix Compiler throws internal stack overflow error when plotting with Makie on Julia master #33954, recursion through field types in is_derived_type
fix codegen for getfield of homogeneous tuples #34243 - fix codegen for getfield of homogeneous tuples
fix #32442, broadcasting over non-offset arrays with mismatched axis eltypes #34230 - fix Broadcasting over views with non-Int indices fails (or creates an OffsetArray) #32442, broadcasting over non-offset arrays with mismatched axis eltypes
Update svd.jl #30239 - Update svd.jl
Fix trailing whitespace in svd docs introduced in cec4c3 #34299 - Fix trailing whitespace in svd docs introduced in cec4c3
add local to testset vars that should be local #34282 - add local to testset vars that should be local
fix #34286, regression in methods with empty tuple of types #34291 - fix Regression using methods with empty tuple for types #34286, regression in methods with empty tuple of types
make fallback signed and unsigned functions unchecked (as documented) #34287 - make fallback signed and unsigned functions unchecked (as documented)
add some compatibility shims for internal functions changed in 1.4 #34289 - add some compatibility shims for internal functions changed in 1.4
add a compatibility fallback for divrem and fldmod #34284 - add a compatibility fallback for divrem and fldmod
Simplify the GC.gc interface #34303 - Simplify the GC.gc interface
Update Mozilla CA certificate store to latest (01-01-2020) for libgit2 SSL. #34319 - Update Mozilla CA certificate store to latest (01-01-2020) for libgit2 SSL.
restore pathsep() and old sig of print_matrix_vdots for backwards compat #34314 - restore pathsep() and old sig of print_matrix_vdots for backwards compat
handle GlobalRef in getfield elim pass #34322 - handle GlobalRef in getfield elim pass
Update sprandn in docs. #34257 - Update sprandn in docs.
add hypot fallback for 2 Numbers of the same type #34316 - add hypot fallback for 2 Numbers of the same type
Use approximate comparison in LinearAlgebra diagonal tests. #34361 - Use approximate comparison in LinearAlgebra diagonal tests.
Generate an error for closure cfunctions on unsupported platforms. #34326 - Generate an error for closure cfunctions on unsupported platforms.
Fix initial value handling with flatten transducer #34369 - Fix initial value handling with flatten transducer
Change lowering of gc preserve #34379 - Change lowering of gc preserve
Force full collection in SharedArrays tests #34349 - Force full collection in SharedArrays tests
Fix CI hangs on win32 #34225 - Fix CI hangs on win32
Make it easier to extend lcm/gcd/gcdx to custom types #34129 - Make it easier to extend lcm/gcd/gcdx to custom types
Use UUID's to generate random tempname on Windows #33785 - Use UUID's to generate random tempname on Windows
Check return value of SystemFunction036 #34429 - Check return value of SystemFunction036
remove code forcing testsets to be compiled #34405 - remove code forcing testsets to be compiled
fix #34415, remove affinity setting code #34433- fix Julia seems to ignore process affinity settings on Linux #34415, remove affinity setting code (

@nanosoldier runbenchmarks(ALL, vs = ":release-1.3")

(cherry picked from commit d723cee)

KristofferC · 2020-01-02T17:00:00Z

I had the wrong merge target branch (master intstead of release-1.4) when starting nanosoldiers but I don't think anything has really been merged so should be ok.

KristofferC · 2020-01-02T17:37:24Z

@nanosoldier runtests(ALL, vs = ":release-1.3")

nanosoldier · 2020-01-03T00:35:43Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

nanosoldier · 2020-01-03T05:18:03Z

Your test job has completed - possible new issues were detected. A full report can be found here. cc @maleadt

KristofferC · 2020-01-03T09:59:27Z

Some failures from JuliaLang/Pkg.jl#1548 not being included here.

JeffBezanson · 2020-01-06T19:25:04Z

AIBECS: BSON
ARCHModels: numerical issue
AbstractNumbers: fails because div is now applicable to Complex
even though it doesn't work
AdaptiveDistanceFields: AxisAlgorithms: bad Vararg length; looks fixed (upgrade a dependency?)
Atom: pathsep not defined
BSON: JuliaIO/BSON.jl#63
BandedMatrices: no method matching bunchkaufman!(::Symmetric{Float32,BandedMatrix{Float32,Array{Float32,2},Base.OneTo{Int64}}}, ::Bool; check=false)

JeffBezanson · 2020-01-06T20:17:47Z

BitIntegers: #33459. we removed a fallback signed() method it used.
CBinding: jl_gc_total_bytes renamed; use Base.gc_bytes() instead
CategoricalArrays: different whitespace in output
CatmullClark: #33954, fixed
CurrenciesBase: FixedDecimals: another div issue (#34284)
DBFTables: output changed
DataInterpolations: Zygote needs to handle loopinfo FluxML/Zygote.jl#393
DecisionTree: ~~code coverage?~~ https://github.com/bensadeghi/DecisionTree.jl/issues/112
DiffEqParamEstim: Zygote
DiffEqSensitivity: Zygote
Distributions: NaN === NaN ?
Econometrics: output changed
EfficientGlobalOptimization: Zygote
ExactPredicates: #34252
FilePathsBase: BSON
FixedEffectModels: PosDefException: matrix is not positive definite; Cholesky factorization failed.
Flux: Zygote
GFlops: some numbers don't match; not sure
GLM: PosDefException: matrix is not positive definite; Cholesky factorization failed.
GeoArrays: curl issue?
GeometricFlux: Zygote
GeometricIntegratorsDiffEq: DecFP
HalfIntegers: div.jl issue with cld; packages need to implement div(a, b, RoundUp)
Hecke: maybe due to changes in reduce?
[1] _empty_reduce_error() at ./reduce.jl:292
[2] reduce_empty(::Function, ::Type{T} where T) at ./reduce.jl:302
[3] reduce_empty(::Base.BottomRF{typeof(max)}, ::Type{T} where T) at ./reduce.jl:317
[4] mapreduce_empty(::typeof(abs), ::Base.BottomRF{typeof(max)}, ::Type{T} where T) at ./reduce.jl:333
HomotopyContinuation: @allocated changes
Hyperspecialize: crash
IncrementalInference: numerical issue?
InfrastructureSystems: #34243
JWAS: zero(Any) ?
JuMP: output changes
KernelFunctions: Zygote

KristofferC · 2020-01-06T20:18:35Z

I started a bit (from the bottom) here: https://docs.google.com/spreadsheets/d/1ngalhKl9FEplWCWAWOfVKamZj-pkt4157pFGimb5wCw/edit?usp=sharing

vtjnash · 2020-01-06T22:11:39Z

DecisionTree was a known bug in that package (since fixed)

Hyperspecialize is probably an instance of #34252 (prior to this release, @testset used to expand to heuristics that would force off inference and optimization. It has forced compilation heuristics since 057fa2a, we just would not infer the code first.)

JeffBezanson · 2020-01-07T20:57:18Z

prior to this release, @testset used to expand to heuristics that would force off inference and optimization

Do you happen to know what that change was?

vtjnash · 2020-01-07T21:52:12Z

b9546c8

(cherry picked from commit 8b57f64)

fix #34206, fix #34207 (cherry picked from commit 007cb29)

…eltypes (#34230) In cases where we have multiple arrays with `OneTo` axes that do not share the same axis eltype, we should simply default to constructing a new array with `OneTo{Int}` axes. (cherry picked from commit 11e7c33)

* Update svd.jl * Update svd.jl Make doc more useful to the reader. * Fix grammar and one trailing whitespace * Update svd.jl * Fix doctests * Avoid duplicating documentation between mutating and non-mutating versions of svd functions. * Reorganize generalized svd docstring and doctests * Break some long lines Co-authored-by: Viral B. Shah <viral@juliacomputing.com> Co-authored-by: Andreas Noack <andreas@noack.dk> (cherry picked from commit cec4c32)

(cherry picked from commit 194a38b)

(cherry picked from commit 8dc0d93)

…ted) (#34287) (cherry picked from commit 37275c3)

…34289) (cherry picked from commit c33298d)

add more-compatible fallback for `divrem` (cherry picked from commit 8f53987)

(cherry picked from commit b0ed147)

sostock · 2020-01-08T09:47:11Z

HalfIntegers: div.jl issue with cld; packages need to implement div(a, b, RoundUp)

This is #34134. It is fixed on HalfIntegers master, but I wanted to wait for a decision on #34129 before releasing a new HalfIntegers version that contains the fix.

andreasnoack · 2020-01-08T09:54:38Z

Whitespace error fixed in #34299

(cherry picked from commit a3eb274)

KristofferC · 2020-01-15T09:11:19Z

Regarding cbrt getting 11x slower (and cos, sin etc) I think this is just benchmark noise.

On 1.3:

julia> @btime cbrt($(nextfloat(zero(Float64))));
  0.029 ns (0 allocations: 0 bytes)

and master

julia> @btime cbrt($(nextfloat(zero(Float64))));
  155.990 ns (0 allocations: 0 bytes)

Doing the "Ref trick":

1.3:

julia> @btime cbrt($(Ref(nextfloat(zero(Float64))))[]);
  178.467 ns (0 allocations: 0 bytes)

and master

julia> @btime cbrt($(Ref(nextfloat(zero(Float64))))[]);
  156.010 ns (0 allocations: 0 bytes)

KristofferC · 2020-01-15T09:28:04Z

Allocation increase in ["array", "index", "(\"sumvector\", \"1.0:0.00010001000100010001:2.0\")"] is MWEd with

julia> v = 1.0:0.00010001000100010001:2.0
1.0:0.00010001000100010001:2.0

julia> r = rand(1:length(v), 5);

julia> f(v, r) = v[r, 1]
f (generic function with 1 method)

julia> @btime f($v, $r);
  76.075 ns (3 allocations: 192 bytes)

(makes 2 allocations on 1.3). Changing to v[r] makes the allocation go back to 2.

sostock · 2020-01-16T09:59:30Z

Before releasing a new HalfIntegers version that is compatible with Julia 1.4 (and marking old versions as incompatible), I wanted to know whether #34129 could be included in 1.4 as well. I guess it’s too late for that now?

While it is not a bugfix, I think it is an important change to the implementation in #33910 (which was merged shortly before the 1.4 branch happened). Without #34129, it is rather inconvenient to extend gcd/lcm/gcdx to custom Real types (see #33910 (comment)).

(cherry picked from commit ade4713)

(cherry picked from commit 717d548)

(cherry picked from commit 78d1352)

(cherry picked from commit f53e144)

@test

There was a bug in initial value handling of `FlatteningRF` and the following example failed: @test mapfoldl( x -> (x, x), ((a, b), (c, d)) -> (min(a, c), max(b, d)), Iterators.flatten((1:2, 3:4)), ) == (1, 4) This is because `BottomRF(op.rf)` was called inside `FlatteningRF` where `op.rf` is already a "non-bottom" reducing function; here it's a `MappingRF`. As `BottomRF(rf)` forwards anything on the second argument on the first invocation as the first argument (accumulator) of the next calls, we need to make sure that this value is processed through `MappingRF` in the above example. However, if we do `BottomRF(op.rf)` where `op.rf` is a `MappingRF`, this `BottomRF` bypasses any processing that has to happen in `op.rf`. (cherry picked from commit 0ee3264)

This fixes #34247 by changing the way gc preserve is lowered. Instead of lowering it in a macro, lower it in the frontend. This allows us to use an SSA value directly for the return token of the gc begin expression. This bypasses the slot-renaming pass of the compiler, thus preventing the compiler from trying to save and restore the token. Of course, this kind of code would generally not be legal (because it uses an SSA value outside of the regular domination relation), but since this is a julia restriction, not an LLVM restriction, we can simply exempt gc_begin tokens from this particular validation. This works fine at the LLVM level also, because it doesn't have this particular restriction. It also doesn't have the same correctness problems as doing the same for non-token values, as the tokens get lowered away by the try/catch lowering before reaching the LLVM backend. (cherry picked from commit 07a16d6)

Fixes the same CI failure as #34336, but hopefully should be immediately mergeable without objection about what the interface is. (cherry picked from commit 3ed4e94)

After much debugging, it turns out that OpenBlas is sometimes hanging on exit due to bad assumptions in its thread management. The upstream discussion is at OpenMathLib/OpenBLAS#2350, but this should fix our frequent win32 CI failures in the meantime.

(cherry picked from commit 3549cd1)

(cherry picked from commit 85eb20f)

* Use UIUD to create random tempname on Windows * Use underscores and remove extension * Truncate to 10 chars the UUID * Generate the random name from a random byte array * Update file.jl (cherry picked from commit d759b5b)

In both places we call it. Also tabs-to-spaces for the new code to match the formatting of the rest of the code. (cherry picked from commit f120989)

(cherry picked from commit ed87ece)

This was removed before and added back apparently by mistake. (cherry picked from commit 5928786)

KristofferC · 2020-01-20T10:01:37Z

@sostock, your changes should be in this branch now.

outline rng length assert

9bd498d

(cherry picked from commit d723cee)

KristofferC added needs nanosoldier run This PR should have benchmarks run on it DO NOT MERGE Do not merge this PR! needs pkgeval Tests for all registered packages should be run with this change release Release management and versioning. labels Jan 2, 2020

KristofferC changed the base branch from master to release-1.4 January 2, 2020 16:59

JeffBezanson and others added 10 commits January 8, 2020 10:27

fix #33954, recursion through field types in is_derived_type (#34223)

03fc530

(cherry picked from commit 8b57f64)

fix codegen for getfield of homogeneous tuples (#34243)

e6b2a8e

fix #34206, fix #34207 (cherry picked from commit 007cb29)

add local to testset vars that should be local (#34282)

0835ee0

(cherry picked from commit 194a38b)

fix #34286, regression in methods with empty tuple of types (#34291)

f03279d

(cherry picked from commit 8dc0d93)

make fallback signed and unsigned functions unchecked (as documen…

bd2ac09

…ted) (#34287) (cherry picked from commit 37275c3)

add some compatibility shims for internal functions changed in 1.4 (#…

3797ea9

…34289) (cherry picked from commit c33298d)

restrict div fallback to Real (#34284)

8eee8d5

add more-compatible fallback for `divrem` (cherry picked from commit 8f53987)

Simplify the GC.gc interface. (#34303)

c9dba72

(cherry picked from commit b0ed147)

Fix trailing whitespace in svd docs introduced in cec4c32.

ca6895f

(cherry picked from commit a3eb274)

KristofferC mentioned this pull request Jan 15, 2020

Allow more general eltypes in sparse array multiplication #33205

Merged

KristofferC force-pushed the backports-release-1.4 branch from 7198d79 to d7d50d0 Compare January 15, 2020 10:56

KristofferC and others added 10 commits January 17, 2020 09:10

bump Pkg to release-1.4

df7473f

add hypot fallback for 2 Numbers of the same type (#34316)

a0d8713

(cherry picked from commit ade4713)

Update sprandn in docs. Fix #31283 (#34257)

671f7aa

(cherry picked from commit 717d548)

Use approximate comparison in LinearAlgebra diagonal tests.

9fbf6b3

(cherry picked from commit 78d1352)

Generate an error for closure cfunctions on unsupported platforms.

8860214

(cherry picked from commit f53e144)

Force full collection in SharedArrays tests (#34349)

2e9ceb8

Fixes the same CI failure as #34336, but hopefully should be immediately mergeable without objection about what the interface is. (cherry picked from commit 3ed4e94)

Fix CI hangs on win32

0665762

After much debugging, it turns out that OpenBlas is sometimes hanging on exit due to bad assumptions in its thread management. The upstream discussion is at OpenMathLib/OpenBLAS#2350, but this should fix our frequent win32 CI failures in the meantime.

Bump OpenBLAS BB release to include win32 threading patch (#34273)

0360daa

KristofferC force-pushed the backports-release-1.4 branch from 3405e7d to 0360daa Compare January 17, 2020 08:10

sostock and others added 6 commits January 20, 2020 09:31

Fix array-argument gcd for non-integers

d3ba27d

(cherry picked from commit 3549cd1)

Make it easier to extend lcm/gcd/gcdx to custom types

4bd73b2

(cherry picked from commit 85eb20f)

Use UUID's to generate random tempname on Windows (#33785)

eaec4a7

* Use UIUD to create random tempname on Windows * Use underscores and remove extension * Truncate to 10 chars the UUID * Generate the random name from a random byte array * Update file.jl (cherry picked from commit d759b5b)

Check return value of SystemFunction036 (#34429)

4865045

In both places we call it. Also tabs-to-spaces for the new code to match the formatting of the rest of the code. (cherry picked from commit f120989)

remove code forcing testsets to be compiled (#34405)

ea6f264

(cherry picked from commit ed87ece)

fix #34415, remove affinity setting code (#34433)

139696b

This was removed before and added back apparently by mistake. (cherry picked from commit 5928786)

ararslan removed DO NOT MERGE Do not merge this PR! needs nanosoldier run This PR should have benchmarks run on it needs pkgeval Tests for all registered packages should be run with this change labels Jan 20, 2020

ararslan added this to the 1.4 milestone Jan 20, 2020

ararslan merged commit 1c977b0 into release-1.4 Jan 20, 2020

ararslan deleted the backports-release-1.4 branch January 20, 2020 21:51

rfourquet mentioned this pull request Jan 21, 2020

[release-1.4] Set VERSION to 1.4.0-rc1 #34452

Merged

Uh oh!

WIP: Backports for 1.4-RC1 #34238

WIP: Backports for 1.4-RC1 #34238

Uh oh!

Conversation

KristofferC commented Jan 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KristofferC commented Jan 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KristofferC commented Jan 2, 2020

Uh oh!

nanosoldier commented Jan 3, 2020

Uh oh!

nanosoldier commented Jan 3, 2020

Uh oh!

KristofferC commented Jan 3, 2020

Uh oh!

JeffBezanson commented Jan 6, 2020

Uh oh!

JeffBezanson commented Jan 6, 2020 • edited by vtjnash Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KristofferC commented Jan 6, 2020

Uh oh!

vtjnash commented Jan 6, 2020

Uh oh!

JeffBezanson commented Jan 7, 2020

Uh oh!

vtjnash commented Jan 7, 2020

Uh oh!

sostock commented Jan 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreasnoack commented Jan 8, 2020

Uh oh!

KristofferC commented Jan 15, 2020

Uh oh!

KristofferC commented Jan 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sostock commented Jan 16, 2020

Uh oh!

KristofferC commented Jan 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

KristofferC commented Jan 2, 2020 •

edited

Loading

KristofferC commented Jan 2, 2020 •

edited

Loading

JeffBezanson commented Jan 6, 2020 •

edited by vtjnash

Loading

sostock commented Jan 8, 2020 •

edited

Loading

KristofferC commented Jan 15, 2020 •

edited

Loading