Side effecting givens in a final map with better-fors

SIP-62 better for `dropForMap` phase bug · Issue #24673 · scala/scala3 · GitHub fixed dropping the final map call in for comprehensions for side effecting functions, however, I believe a case was left out: when the final map call has a using clause, and resolving that implicit results in a side effect.

I noticed this when using traces with ZIO. ZIO maps ask for a Trace, and default to generating one when a more specific one is not present. Prior to upgrading to 3.9 from 3.7 (I know better-fors is a 3.8 feature), if a final map in a for comprehension produced a failing effect, the generated trace would include a frame for that map call, whereas after the upgrade, it does not.

This scastie demonstrates the difference in behavior when run with 3.7 versus 3.9:

I’m curious if this is intended or if this is another edge case for keeping the final map? If intended, could we either re-evaluate that decision or provide a mechanism to force keeping the map call?

1 Like

Thanks for the test case. I assume it is not intended, but I’ll check the docs.

The previous fix addressed valdefs aka “midstream assignments” and not side effects per se.

But I see that PR included a refactor to make the code “easier to read”, so now I may put that to the test, so to speak. Perhaps I will embarrass myself by not consulting Claude.

Worth adding that the spec is sly on this point.

But the equality of the last pattern and the result has to be able to be checked syntactically, so it is either a variable or a tuple of variables.

That is, the spec is syntax-driven, so that perhaps the current behavior is intended after all.

Ok, that’s interesting. So wrapping the yielded value in something like identity would preserve the final map call?

I hope that’s not the intention though, I would think the intention would be to not have any behavioral differences between different desugaring strategies.

I see the core team addressed the issue and will make a call.

I was about to emit a warning, but depending on what they decide, it will either warn when eliding or warn when not eliding.

The spec restriction is necessary but not sufficient; I could see merit in either decision, and there are other cases where side effects are elided, such as module loads (initialization of objects); that is not without ongoing debate or head scratching.

It wouldn’t be necessary to “warn” about such things if one could rely on “Claude” to always mention it during a “review”.

Edit: by “make a call”, I mean they will “make a decision”, not “make a call to map”.

I see there is a test case:

val _ = for x = 42; x <- MyOption(x) yield x

for which one might say, OK maybe. But also:

Welcome to Scala 3.3.9 (25.0.2, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.

scala> val xs = (1 to 10).toList
val xs: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

scala> for x <- xs; x = 42 yield x
-- [E005] Naming Error: --------------------------------------------------------
1 |for x <- xs; x = 42 yield x
  |             ^
  |             duplicate pattern variable: x
  |
  | longer explanation available when compiling with `-explain`
1 error found

Welcome to Scala 3.10.0 (25.0.2, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.

scala> val xs = (1 to 10).toList
val xs: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

scala> for x <- xs; x = 42 yield x
val res0: List[Int] = List(42, 42, 42, 42, 42, 42, 42, 42, 42, 42)

“Alignment with expectations” may require tweaks to either desugaring or to expectations.

Overall, this seems to work as intended. We drop the last map, so the map is not invoked and hence no given is resolved.

You could even have a similar situation, where we would normal side effects in the map and they should not be invoked as well. It would also not possible to check that properly, at least if I understand it correctly.

I still think it should not be the intention, but if it is, could we have some mechanism to indicate that we don’t want better-fors to apply to a particular for-comprehension?

We can for sure add some kind of a warning, it can be discussed in depth under the issue maybe?

I argue that such a mechanism exists: write out the chain of filter, map and flatMap that you specifically want.

1 Like

The intention is to have only one desugaring strategy — the new one.

We already know that SIP-62 can cause altered semantics under certain unusual circumstances. It says so right in the SIP itself: SIP-62 - For comprehension improvements | Scala Documentation :

This change may change the semantics of some programs. It may remove some map calls in the desugared code, which may change the program semantics (if the map implementation was side-effecting).

I think we should not fork the language by offering support for multiple desugarings.

The new desugaring is strictly better — or at least, I don’t see that any compelling argument has yet been offered otherwise in this thread.

5 Likes
    for
      n <- M(1)
      r <- M(n)
    yield r: Int

Anything but

2 Likes