Deprecated range syntax


#9

That’s why I’m wondering if there was a legitimate technical reason for this deprecation, namely the failure of Doubles to represent some decimal numbers precisely. If I were using this construct in critical code, I would add a small epsilon to the end point to make sure it is not missed due to numerical roundoff. For example:

for (x <- 0.0 to 10.01 by 0.2)

Does BigDecimal makes that little trick unnecessary? (Easy enough to test, but I don’t feel like doing it right now.)


#10

If you want a series of equidistant Doubles, use Int indices and calculate, e.g.

__scala> val (min, max, step) = (1.0d, 10.0d, 0.01d)
min: Double = 1.0
max: Double = 10.0
step: Double = 0.01

val n = ((max - min)/step).toInt
n: Int = 900

val doubles = (0 to n).map(i => ((n - i)min + imax)/n)
doubles: scala.collection.immutable.IndexedSeq[Double] = Vector(1.0, 1.01, 1.02, 1.03, 1.04, 1.05, 1.06, 1.07, 1.08, 1.09, 1.1, 1.11, 1.12, 1.13, 1.14, 1.15, 1.16, 1.17, 1.18, 1.19, 1.2, 1.21, 1.22, 1.23, 1.24, 1.25, 1.26, 1.27, 1.28, 1.29, 1.3, 1.31, 1.32, 1.33, 1.34, 1.35, 1.36, 1.37, 1.38, 1.39, 1.4, 1.41, 1.42, 1.43, 1.44, 1.45, 1.46, 1.47, 1.48, 1.49, 1.5, 1.51, 1.52, 1.53, 1.54, 1.55, 1.56, 1.57, 1.58, 1.59, 1.6, 1.61, 1.62, 1.63, 1.64, 1.65, 1.66, 1.67, 1.68, 1.69, 1.7, 1.71, 1.72, 1.73, 1.74, 1.75, 1.76, 1.77, 1.78, 1.79, 1.8, 1.81, 1.82, 1.83, 1.84, 1.85, 1.86, 1.87, 1.88, 1.89, 1.9, 1.91, 1.92, 1.93, 1.94, 1.95, 1.96, 1.97, 1.98, 1.99, 2.0, 2.01, 2.02, 2.03, 2.04, 2.05, 2.06, 2.07, 2.08, 2.09, 2.1, 2.11, 2.12, 2.13, 2.14, 2.15, 2.16, 2.17, 2.18, 2.19,…

doubles.last
res0: Double = 10.0__

Best, Oliver


#11

The technical reason that floating-point ranges are deprecated is that precise addition of Double and Float is not guaranteed. In particular, it fails on addition of numbers with a decimal fraction, even if that fraction is “not tiny”, like 0.1. The library should not contain methods that mysteriously give you unreliable behavior, hence the deprecation.

The BigDecimal approach solves the issue because the math is performed without imprecision.

There’s no advantage to trying epsilon-shifting schemes because then you may as well just use integers to represent your decimal fraction, for example in the manner that Oliver demonstrated.

One could envision a macro that did the right thing for literals.


#12

Some literal-minded person with a vision ought to envision such a macro for literals.


#13

One can dispute whether Range.Double is useful, stepping with precision but delivering doubles, or whether there is another useful sense for double-stepping that could be controlled by a context, but please acknowledge that I am discussing the former, which is limited but by itself is unproblematic. Sometimes all you need or want is an unproblematic solution to an unproblematic problem.

I came back from the angry Java thread even angrier. You wouldn’t like me when I’m angry.


#14

It is fairly easy to write a routine that works right by using multiplication and division rather than addition. In fact, I did exactly that a long time ago for my scalar class (which represents physical scalars with units). Here it is:

def scalarSteps(start: Scalar, end: Scalar, step: Scalar): Vector[Scalar] = {

val inc = abs(step)
val sgn = Real(signum(step)) // convert to "Double"
val start1 = Real(start/inc)
val end1 = Real(end/inc) + 1e-10 * sgn

(BigDecimal(start1) to end1 by sgn)
    .map(_.toDouble).map(_ * inc).toVector
}

A simpler version of this (replace Scalar with Double) could be provided by default for so-called “Doubles” in Scala so that the deprecated syntax could be maintained and would work correctly. That would relieve users of spending time to figure out how to use BigDecimal. It would also result in a tiny performance penalty, but I would gladly take the slight hit in return for the convenience.


#15

For what it’s worth, It just occurred to me that if the human race had chosen base 8 (octal) instead of base 10 as the standard numeral system, we wouldn’t have this problem. People say we use base 10 because we have ten fingers, but actually we have 8 fingers and two thumbs! Too late to fix that one, I guess!


#16

That doesn’t work right on 0.1 to 0.299999999999 by 0.2.


#17

(How) is this different from 1 to 7 by 2?


#18

Binary-coded integers can represent whole decimal numbers exactly. Binary fractions, which is what Float and Double are, cannot represent decimal fractions exactly. So 1 to 7 by 2 is calculated without error.


#19

I’m sorry, but I truly do not see why you think Range.Double is “unproblematic”. What do you think about this behavior:

**Welcome to Scala 2.12.4 (OpenJDK 64-Bit Server VM, Java 1.8.0_171).
Type in expressions for evaluation. Or try :help.

Range.Double(0.0, 7.0, 1.0).last
res0: Double = 6.0

Range.Double(0.0, 0.7, 0.1).last
res1: Double = 0.7000000000000001**

Best, Oliver


#20

Even if you step with precision you run into surprises. What should the behavior of 0.1 until 3*0.1 by 0.1 be? More sneakily, suppose you have def steps(x: Double) = x until 3*x by x. Shall this sometimes give you three elements and sometimes two?

The imprecision can easily arrive in the input which is why a solution with precise stepping isn’t really a solution.


#21

That’s just a bug. The code comment is:

// XXX This may be incomplete.

And with the appropriate override,

scala> Range.Double(0, .7, .1).last
<console>:12: warning: method apply in object Double is deprecated (since 2.12.6): use Range.BigDecimal instead
       Range.Double(0, .7, .1).last
             ^
res1: Double = 0.6

#22

I still insist that in the age of literal types and macros, it’s not too much magic to insist on literals or at least take warning action. 0 to .7 by .1, give me BDs or Doubles or whatever seems to be expected.

Or at least enable a Propensive library to do it for me.


#23

That doesn’t work right on 0.1 to 0.299999999999 by 0.2.

But it’s “close enough for government work,” as they say!

I actually use something like this to discretize a bounding area for a numerical algorithm, and I definitely need to capture the end point. But I need to capture the end point even if it is in the middle of a step, which is a slightly different problem. I could just add the end point to the end of the sequence, but then I would usually be repeating the end point. So I came up with this little scheme:

def scalarStepsx(start: Scalar, end: Scalar, step: Scalar): Vector[Scalar] = {
// same as scalarSteps except guaranteed to include end point

val steps = scalarSteps(start, end, step)
if (areClose(steps.last, end)) steps else steps :+ end
}

def areClose(x: Scalar, y: Scalar) =
if (y == 0) x == 0 else abs(x / y - 1) < 1e-13


#24

That one also has counterexamples where it does the wrong thing (e.g. 0.1 to 0.300000000001 by 0.1). None of these are suitable for a library method that should act “intuitively”.


#25

@som-snytt - I don’t have any objection to a working macro. I’m not likely to be able to write one in a reasonable amount of time myself, though.

As Russ’s examples indicate, it’s tricky to get it working. The only really safe thing to do is pass literal numeric arguments into the BigDecimal string constructor, picking them directly out of the text of the code (not the Double literal computed by the compiler).


#26

I can’t sneak anything by you!

Seriously though, a person is extremely unlikely to actually use a number like 0.300000000001, and roundoff error will be a couple orders of magnitude less than 1e-12. Hence, I don’t see it as a practical issue. Nevertheless, I can understand that you cannot allow even the tiniest “loophole” in the standard language and library.


#27

Some applications actually hinge upon these kinds of differences–those that have chunked intervals where the intervals are used as a denominator, for instance, or those that count on hitting the endpoint exactly in order to generate a difference between a to b and a until b. This can be really important to get right if you’re, say, trying to generate angles between 0 and 2*Pi; overshooting on the last endpoint giving you a second approximately-zero angle can be a big deal.

I’d love to have a better story here, but unfortunately it is all too easy to have an “intuitive” result that’s just wrong. For example, people will reason, “Well, if I hit the endpoint exactly, to and until will be different, so I’ll just boost the endpoint up/down a tiny bit to make them the same,” and then they get weird unexpected behavior because it’s fighting secret heuristics in the algorithm put there to try to preserve a different kind of intuition.


#28

I can see that arithmetic with Doubles is imprecise, and that makes a naive range of Doubles unintuitive. But I don’t really see the problem anymore when you can make the steps of the range precise by using a BigDecimal underneath. The argument now is that you can give an imprecise result of a calculation with Doubles as input to the range (e.g. 0.1 until 3*0.1 by 0.1). But isn’t this just the case for everything one might do with Doubles? If that’s a reason not to have a range of Doubles, then shouldn’t you just remove Double itself?

For instance:

scala> Ordering[Double].equiv(0.3, 0.1 * 3)
res0: Boolean = false

Should we now deprecate Ordering[Double]?

Also, if you force people to use Range.BigDecimal instead, this is what’s going to happen:

scala> def someInput = 0.1 * 3
input: Double

scala> val range = BigDecimal("0.1") until someInput by 0.1
range: scala.collection.immutable.NumericRange.Exclusive[scala.math.BigDecimal] = NumericRange 0.1 until 0.30000000000000004 by 0.1

scala> range.last.toDouble
res1: Double = 0.3

Uglier code for the same result.