I have a bug that I cannot explain. In the original code i have the following function:
// Fails
private def lazyMap[A,B](i: Iterable[A])(f: A => B): Iterable[B] = {
val convertable = new Iterable[B] {
override def iterator: Iterator[B] = {
val converter: AbstractIterator[B] = new AbstractIterator[B] {
private val iter = i.toIterator
override def hasNext: Boolean = iter.hasNext
override def next(): B = {
f(iter.next())
}
} // iterator
converter
}
}
convertable
}
I use this to process a data stream. However the iterator simply extracted a limited amount of elements and stopped (in my case 1 element). So I started experimenting. What I found was that if I place this line:
private val iter = i.toIterator
in different but supposedly equivalent places the behavior changes. More concretely,
if I use this version:
private def lazyMap[A,B](i: Iterable[A])(f: A => B): Iterable[B] = {
val convertable = new Iterable[B] {
private val iter = i.toIterator
override def iterator: Iterator[B] = {
val converter: AbstractIterator[B] = new AbstractIterator[B] {
override def hasNext: Boolean = iter.hasNext
override def next(): B = {
f(iter.next())
}
} // iterator
converter
}
}
convertable
}
or this version:
private def lazyMap[A,B](i: Iterator[A])(f: A => B): Iterable[B] = {
val convertable = new Iterable[B] {
override def iterator: Iterator[B] = {
val converter: AbstractIterator[B] = new AbstractIterator[B] {
private val iter = i
override def hasNext: Boolean = iter.hasNext
override def next(): B = {
f(iter.next())
}
} // iterator
converter
}
}
convertable
}
private def lazyMap[A,B](i: Iterable[A])(f: A => B): Iterable[B] = lazyMap(i.toIterator)(f)
it works correctly. So my question is, what is wrong with the first version? Why is it not equivalent to the last 2?
I have difficulties reproducing the results but I have included below a test that seems to reproduce something close to what I am experiencing in 2.12.6. With the version below I get the following expected output:
Not executed -------------------
e = List(0, 1, 2, 3, 4)
e = List(5, 6, 7, 8, 9)
e = List(10, 11, 12, 13, 14)
e = List(15, 16, 17, 18, 19)
e = List(20, 21, 22, 23, 24)
e = List(25, 26, 27, 28, 29)
e = List(30, 31, 32, 33, 34)
e = List(35, 36, 37, 38, 39)
List(135, 160, 185)
But if I use one of the other versions (just comment the appropriate lazyMap) I get:
Not executed -------------------
e = List(0, 1, 2, 3, 4)
e = List(5, 6, 7, 8, 9)
e = List(10, 11, 12, 13, 14)
e = List(15, 16, 17, 18, 19)
e = List(20, 21, 22, 23, 24)
e = List(0, 1, 2, 3, 4)
e = List(5, 6, 7, 8, 9)
e = List(10, 11, 12, 13, 14)
e = List(15, 16, 17, 18, 19)
e = List(20, 21, 22, 23, 24)
e = List(25, 26, 27, 28, 29)
e = List(30, 31, 32, 33, 34)
e = List(35, 36, 37, 38, 39)
List(135, 160, 185)
Notice how we print out 5 window but then stop and then start again.
Can anyone explain this to me?
Test code follows.
TIA
object LazyIterable {
// Ok
private def lazyMap1[A,B](i: Iterable[A])(f: A => B): Iterable[B] = {
val convertable = new Iterable[B] {
private val iter = i.toIterator
override def iterator: Iterator[B] = {
val converter: AbstractIterator[B] = new AbstractIterator[B] {
override def hasNext: Boolean = iter.hasNext
override def next(): B = {
f(iter.next())
}
} // iterator
converter
}
}
convertable
}
/*
// Fails
private def lazyMap1[A,B](i: Iterable[A])(f: A => B): Iterable[B] = {
val convertable = new Iterable[B] {
override def iterator: Iterator[B] = {
val converter: AbstractIterator[B] = new AbstractIterator[B] {
private val iter = i.toIterator
override def hasNext: Boolean = iter.hasNext
override def next(): B = {
f(iter.next())
}
} // iterator
converter
}
}
convertable
}*/
/*
// Fails
private def lazyMap1[A,B](i: Iterable[A])(f: A => B): Iterable[B] = {
val convertable = new Iterable[B] {
override def iterator: Iterator[B] = {
val iter = i.toIterator
val converter: AbstractIterator[B] = new AbstractIterator[B] {
override def hasNext: Boolean = iter.hasNext
override def next(): B = {
f(iter.next())
}
} // iterator
converter
}
}
convertable
}*/
def window[A](i: Iterable[A], ssize: Int, step: Int): Iterable[Iterable[A]] = {
val convertable = new Iterable[Iterable[A]] {
override def iterator: Iterator[Iterable[A]] = {
val converter = new AbstractIterator[Iterable[A]] {
val iter: Iterator[Iterable[A]] = lazyMap1(i)(e => e).sliding(ssize, step)
override def hasNext: Boolean = iter.hasNext
override def next(): Iterable[A] = iter.next()
}
converter
}
}
convertable
}
def main(args: Array[String]): Unit = {
val n = 40
val l1: Iterable[Int] = /*Stream.from(0).take(n)*/ Iterable.range(0,n)
val l1m1: Iterable[Int] = lazyMap1(l1)( _ * 1)
val l1m2: Iterable[Iterable[Int]] = window(l1m1, 5, 5)
val l1m3 = lazyMap1(l1m2){ e:Iterable[Int] => println(s"e = $e"); e.toList.sum }
val l1m4 = window(l1m3,5,5)
println("Not executed ------------------- ")
println(l1m4.last)
}
}