Unwarranted - or just very weird - Stepper.spliterator specialization

So, Stepper is @specialized (at least in Scala 2, I have no idea how it works in Scala 3).
Specialization if great, but it’s a leaky abstraction, and in particular there is one huge pitfall one has to avoid: separating the erased trait and its @specialized version in class linearization, because it it breaks the delegation chain and can lead to amusing bugs like a StackOverflowException by an infinite chain of calls to super. Back to the topic: the only method, which, per spec, should be specialized in Stepper is nextStep. It should not, in particular, specialize spliterator, because, first, Spliterator is not a specialized type (officially), and, second, it doesn’t even return a Spliterator[A], but a Spliterator[_] in Stepper and [B >: Int]Spliterator[B] in IntStepper. Even if Spliterator was specialized, this method should not be eligible to specialization.
And yet, here we go:

import java.util.function.IntConsumer
import java.util.{Iterator=>JavaIterator, Spliterator}
import scala.collection.{IntStepper, Stepper}

abstract class Stepper1[A] extends Stepper[A] {
  var hasStep = true
  override def estimateSize = if (hasStep) 1 else 0
  override def characteristics = 0
  override def trySplit() = null
}
trait PrimitiveStepper1[A] extends Stepper1[A] {
  override def spliterator[B >: A] = ???
  override def javaIterator[B >: A] = ???
}
class IntStepper1(elem :Int) extends Stepper1[Int] with IntStepper with PrimitiveStepper1[Int] {
  override def nextStep() = { 
    if (!hasStep) throw new NoSuchElementException
    hasStep = false
    elem 
	}
}

class LazyStepper[+A, +S <: Stepper[A]](init: => S) extends Stepper[A] {
	private[this] var underlying :S = _
	
	protected final def stepper :S = {
		if (underlying == null)
			underlying = init
		underlying
	}
	override def estimateSize :Long = stepper.estimateSize
	override def characteristics :Int = stepper.characteristics
	override def spliterator[B >: A] :Spliterator[_] = stepper.spliterator
	override def javaIterator[B >: A] :JavaIterator[_] = stepper.javaIterator
	
	override def hasStep :Boolean = stepper.hasStep
	override def nextStep() :A = stepper.nextStep()

	override def trySplit() :Stepper[A] = stepper.trySplit()
}
class LazyIntStepper(init: => IntStepper) extends LazyStepper[Int, IntStepper](init) with IntStepper {
		override def spliterator[B >: Int] = stepper.spliterator
		override def javaIterator[B >: Int] = stepper.javaIterator
		override def trySplit() = stepper.trySplit()  
}

val step = new LazyIntStepper(new IntStepper1(42))
step.spliterator

The code above should have thrown, by my understanding, a NotImplementedError. But it does not:

spliterator$mcI$sp:258, IntStepper (scala.collection)
spliterator$mcI$sp$:258, IntStepper (scala.collection)
spliterator$mcI$sp:673, IntStepper1 (net.noresttherein.sugar.collections)
spliterator$mcI$sp:932, LazyIntStepper (net.noresttherein.sugar.collections)
spliterator:932, LazyIntStepper (net.noresttherein.sugar.collections)
spliterator:931, LazyIntStepper (net.noresttherein.sugar.collections)

It’s worth noting, that this behaviour is not observed when calling IntStepper1.spliterator directly:


val step1 = new new IntStepper1(42)
step1.spliterator //throws

What’s happening here? LazyIntStepper should change nothing: the call to IntStepper1.spliterator is in both cases determined based on it being an IntStepper. The word ‘bug’ comes to mind, but, if so, it won’t get fixed as it doesn’t look like something that will preserve binary compatibility of the collection library. However, I have no idea what underhand magic the compiler is performing on critical types, so I am not reporting it for now. Still, if it’s a bug, I’ll gladly bump my reporting stat :stuck_out_tongue:

I’m a non-expert in these API in that I have managed to avoid using them, but following the advice of the compiler options, -Xsource:3 warns about your inferred overrides, and then the compiler will complain directly about your bad overrides.

My first diff:

@@ -1,4 +1,4 @@
-// scalac: -Xsource:3 -Xlint -quickfix:cat=scala3-migration
+// scalac: -Xsource:3 -Xlint

 import java.util.function.IntConsumer
 import java.util.{Iterator=>JavaIterator, Spliterator}
@@ -8,11 +8,11 @@ abstract class Stepper1[A] extends Stepper[A] {
   var hasStep = true
   override def estimateSize = if (hasStep) 1 else 0
   override def characteristics = 0
-  override def trySplit() = null
+  override def trySplit(): Stepper[A] = null
 }
 trait PrimitiveStepper1[A] extends Stepper1[A] {
-  override def spliterator[B >: A] = ???
-  override def javaIterator[B >: A] = ???
+  override def spliterator[B >: A]: Spliterator[_] = ???
+  override def javaIterator[B >: A]: JavaIterator[_] = ???
 }
 class IntStepper1(elem: Int) extends Stepper1[Int] with IntStepper with PrimitiveStepper1[Int] {
   override def nextStep() = {

with the result something like

stepper.scala:17: error: incompatible type in overriding
def javaIterator[B >: Int]: java.util.PrimitiveIterator.OfInt (defined in trait IntStepper)
  with override def javaIterator[B >: Int]: java.util.Iterator[_] (defined in trait PrimitiveStepper1);
 found   : [B >: Int]java.util.Iterator[_]
 required: [B >: Int]java.util.PrimitiveIterator.OfInt;
 other members with override errors are: trySplit
class IntStepper1(elem: Int) extends Stepper1[Int] with IntStepper with PrimitiveStepper1[Int] {
      ^
1 error

so the next edit

@@ -1,7 +1,6 @@
 // scalac: -Xsource:3 -Xlint

-import java.util.function.IntConsumer
-import java.util.{Iterator=>JavaIterator, Spliterator}
+import java.util.{Iterator=>JavaIterator, PrimitiveIterator, Spliterator}
 import scala.collection.{IntStepper, Stepper}

 abstract class Stepper1[A] extends Stepper[A] {
@@ -20,6 +19,9 @@ class IntStepper1(elem: Int) extends Stepper1[Int] with IntStepper with Primitiv
     hasStep = false
     elem
   }
+  override def trySplit(): IntStepper = null
+  override def spliterator[B >: Int]: Spliterator.OfInt = ???
+  override def javaIterator[B >: Int]: PrimitiveIterator.OfInt = ???
 }

 class LazyStepper[+A, +S <: Stepper[A]](init: => S) extends Stepper[A] {

which produced the result I think you wanted:

scala.NotImplementedError: an implementation is missing
        at scala.Predef$.$qmark$qmark$qmark(Predef.scala:344)
        at IntStepper1.spliterator$mcI$sp(stepper.scala:24)
        at LazyIntStepper.spliterator$mcI$sp(stepper.scala:46)

We got the covid this week, so I’m not at all sharp, this is just what the tooling tells me.

Apparently -quickfix is not quite ready to face live fire.

I see your class is calling:

scala> :javap LazyIntStepper#spliterator
         1: invokevirtual #61                 // Method spliterator:()Ljava/util/Spliterator$OfInt;

And I see that was your question, after all. I think the specialization in the covariant return is what the Scaladoc for Stepper calls “hand-specialized”, as opposed to automatically derived.

scala> :javap -public IntStepper#spliterator
  public default <B> java.util.Spliterator$OfInt spliterator();
  public default <B> java.util.Spliterator$OfInt spliterator$mcI$sp();

As I keep forgetting, the incantation for REPL is

$ scala -nobootcp -J--add-exports -Jjdk.jdeps/com.sun.tools.javap=ALL-UNNAMED

scala> :paste test/files/run/stepper.scala