Idiomatic Scala Support

[benjchristensen]

As of version 0.11.0 Scala support is provided through the use of implicits. Conversations on Twitter are bringing up other possible improvements. Let's use this issue to discuss.

[jmhofer]

A few comments from my point of view (since I first looked at this project, I always wanted to use it from Scala):

Now that the core is typesafe, imho we have the first workable Scala integration (via implicits). From here on, everthing depends on how much effort we want to put into this vs how convenient it will be for the users.

What we already have is that the most important method names match (map, flatMap, filter). Matt also made Scala for-comprehensions work with observables.

There's a lot of room for improvements, though. It seems that due to Scala-Java interop, there are problems with type inference (I even ran into a Scala compiler crash somewhere), and sometimes asJava collection conversions are necessary. Also, Scala/Akka futures are not yet addressed at all. An Akka-specific scheduler would probably be awesome to have, too.

For full Scala user convenience, we'd probably have to completely wrap rx.Observable into its own rx.scala.Observable. This would mean a lot more maintenance effort for the Scala integration, of course (I don't think we can do this automatically via code generation or macros, but I'm no expert of all that). But still, I think it's well worth it to create a dedicated Scala API.

[xeno-by]

1. Implicit-based integration with Java-based observables has a serious flaw as it doesn't support parameter type inference for lambdas. The situation isn't going to change until https://issues.scala-lang.org/browse/SI-6221 is fixed. It's definitely fixed in 2.11, and at EPFL we have a compiler plugin that backports the fix to 2.10. Probably Typesafe folks can be convinced to do the backport in one of 2.10.x releases (/cc @adriaanm @gkossakowski @retronym @JamesIry).

2. Some features that might be desirable in rx.scala.Observable: a) native function types, b) native collections, c) method names familiar to folks that use native collections, d) making parameter of defer by-name, e) creation of Observables via apply rather than via from.

3. I also don't think that macros can help with rx.scala.Observable, because what you're probably after isn't just a copy/paste of a Java API, but rather a redesign that takes Scala features into account. But if there are some things that can be autogenerated, I'll be happy to answer questions about how macros work and how they can be used.

[aloiscochard]

IMHO, after looking at the code, using implicits convertions is not a good idea.

I think you have better to start with a complete wrapper of the API, then you'll be able to apply scala idiom (basically writing a DSL) instead of having a Java API with some facilities for converting type.

And that way you don't have to wait for https://issues.scala-lang.org/browse/SI-6221 ... which sounds anyway like a clumsy integration of Java API to me.

[benjchristensen]

Here is some background on why the current design was chosen rather than having each language with a separate version of Observable:

---

The approach of having language specific packages/classes was pursued but did not work well because Rx is a composable library. It means that every time an Observable is used it needs to be re-wrapped or un-wrapped by whichever language is using it.

For example ...

From Java a library is exposed that has a method like this:

rx.Observable getData()

From Groovy a library is exposed with a method like:

rx.groovy.GroovyObservable getOtherData()

Then from Scala you need to wrap them again:

rx.scala.ScalaObservable.from(getOtherData())

This means we have an rx.Observable wrapped as rx.groovy.GroovyObservable wrapped as rx.scala.ScalaObservable.

To compose the two we would have:

rx.scala.ScalaObservable.zip(rx.scala.ScalaObservable.from(
      getOtherData()), 
      rx.scala.ScalaObservable.from(getData()),
       ... scala closure here ...);

Now what does ScalaObservable return from its operators? ScalaObservable or Observable?

Should the above zip operator return rx.scala.ScalaObservable or rx.Observable? What happens if this library is consumed from another language?

If Observable each step along the way it must be wrapped yet again. If ScalaObservable it has now changed all of the return types of rx.Observable to a subtype.

In short, for interop between languages it very quickly becomes a mess and our primary polyglot goal was that rx.Observable was usable across all libraries as the single type and because the whole point of Rx is chained composition it's not as simple as just a single decoration at the beginning. It affects every single method in an API and step of the chaining.

For this reason we chose the current language-adaptor model so rx.Observable can remain the sole public interface across languages.

[xeno-by]

Thanks for providing the design notes!

I wonder though how often people tend to mix languages in their projects. Usability improvements in switching from a Java-based least common denominator to an idiomatic API might be significant. Does being polyglot overweigh these improvements?

[jmhofer]

I'm not sure if wrapping Observable for Scala (not in a subclass though) would really hurt so much. Imho it's worth a try to find out how it feels. There will be a lot of wrapping and unwrapping behind the scene though, I'm afraid.

[benjchristensen]

At Netflix we are using Clojure, Groovy, Java and Scala and I know of apps running code from at least 3 of those 4 in the same JVM instance. I imagine it's not common in most environment for this type of diversity, but it is something we have wanted to support as seamlessly as possible. This is because we have wanted the rx.Observable to act as the interface we can expose across module boundaries as it naturally ends up at the edge of the API in methods such as Observable<T> getDataFromService(args). This is part of what drove us to target the JVM with RxJava and not any specific language.

That said, an idiomatic solution that works best for pure Scala apps is more important. If we can find a solution that can retain the use of rx.Observable without a completely separate wrapper then great, otherwise let's have a ScalaObservable for pure Scala apps and a way of going back and forth across language boundaries when it's needed. Perhaps the implicits that exist right now can solve the immediate interop needs for easy interaction from Scala to Java, but the option to convert to ScalaObservable would also be there.

Another piece of information to guide this ... the Rx.Net version in C# is defined by simple interfaces for Observer and Observable without any of the operator interfaces on them. All of the operator methods (static and instance level) are added via extension methods. It's quite clean and makes the interfaces more flexible (anyone can easily implement them and the extension methods are 'just there' without inheritance involved).

The reason RxJava has Observable as a concrete class is because extension methods don't exist in Java so we don't have a choice but to have them as concrete methods to enable the fluent chaining pattern.

In Scala however we do have extension methods (implicits), macros etc that theoretically can allow the rx.Observable to be made into whatever it needs to be for idiomatic Scala usage, similar to how C# implements Rx.Net using extension methods. I imagine a possible issue is if existing methods on rx.Observable are in the way and cause problems in achieving idiomatic Scala functionality, if that's the case I'd like to understand what those issues are and if they can be resolved and if not if they really are deal breakers.

At this point I become not so helpful as I am not skilled enough in Scala to have a valid opinion or guide the conversation much further. I do have some questions though for the Scala experts here:

1. What would make rx.Observable idiomatic for Scala?

I'd like to see unit tests or sample code demonstrating expected behavior and usage so that we're all working towards the same goal and know when we've achieved success.

2. What about the current rx.Observable + Scala implicits is not working well?

3. What requirements of (1) can not be achieved with implicits and macros and requires a separate concrete class as a wrapper?

Thank you everyone for your involvement, I really do want us to find the ideal solution for Scala and then as a secondary priority make it work well across the JVM for polyglot applications.

[xeno-by]

Thank you for your swift feedback! It's a pleasure to help.

1. I'm not an Rx expert, and in fact this is the first time I see any Rx code apart from random snippets in web articles, so please don't treat this list as definitive. Comprehensive analysis would require quite some time, and I'm not sure I have it right now.

a) Observable.from could be replaced by Observable.apply, so that one can write Observable(1, 2, 3) instead of Observable.from(1, 2, 3).

b) It shouldn't be necessary to write asJava, e.g. as in https://github.com/Netflix/RxJava/blob/master/language-adaptors/rxjava-scala/src/main/scala/rx/lang/scala/RxImplicits.scala#L349, and asScala (not sure whether it's necessary now).

c) Use of mutable collections in situations like https://github.com/Netflix/RxJava/blob/master/language-adaptors/rxjava-scala/src/main/scala/rx/lang/scala/RxImplicits.scala#L350 is really inconsistent with Scala's pursuit of immutability. I wonder whether it'd be possible to write such code in a functional way.

d) Instead of taking a no-arg function, Observable.defer could take a by-name parameter, which would obviate the need of creating an explicit closure: https://github.com/Netflix/RxJava/blob/master/language-adaptors/rxjava-scala/src/main/scala/rx/lang/scala/RxImplicits.scala#L232.

e) For a former C# developer like me, method names like lastOrDefault sound familiar, but for the Scala croud out there - not so much. How about changing the names and maybe even signatures, so that they mirror method names of native collections? E.g. lastOrDefault could become lastOption (also note the change from default values to options).

f) I'm also not sure whether extension methods are even necessary. From what I would guess, in C# they are forced to use them, because interfaces can't define methods with implementations (at least, that was the story with IEnumerable and Enumerable). In Scala, we can do that, so why not just put all the combinators in Observable?

g) mapMany in https://github.com/Netflix/RxJava/blob/master/language-adaptors/rxjava-scala/src/main/scala/rx/lang/scala/RxImplicits.scala#L351 is essentially flatMap, right?

h) No idea how this name could be stated more succinctly, but toBlockingObservable feels a bit verbose. When I moved from C# to Scala, in people's code I felt an overall tendency to compress everything, including names. This is kind of a vague observation, so feel free to ignore it.

2. Function literals don't support inference for parameter types unless they are used in a context that requires a Scala native function type. In particular, inference won't work if function literals are supposed to be converted to something implicitly.

E.g. in https://github.com/Netflix/RxJava/blob/master/language-adaptors/rxjava-scala/src/main/scala/rx/lang/scala/RxImplicits.scala#L341 one can't drop type annotations for lambda parameters. I tried removing them and got a compilation error. (Btw why are you using 2.10.1, when 2.10.2 is already available for quite long? Did you have any problems with it?)

That's an instance of https://issues.scala-lang.org/browse/SI-6221, which has been fixed in 2.11 and probably could be backported to 2.10. If you're in need of an immediate solution, we at EPFL have a compiler plugin for 2.10 that provides a backport.

[vjovanov]

Excellent observations about the API @xeno-by. Currently, we are preparing exercises for the Coursera class Principles of Reactive Programming. For this course we would like to make an idiomatic Scala API. @samuelgruetter is working on that.

The cross language compatibility pointed out by @benjchristensen is a reasonable concern. However, I think there is a lot of monolingual projects out there that would greatly benefit from the idiomatic Scala library. It seems to me that adding a Scala wrapper can be only a plus in those cases.

For polyglot projects we could also take the Java rx.Observable as a common ground (being a return type of each operation and in API signatures). Then all of the Scala support can be added by an implicit conversion (rx.Observable => rx.ScalaObservable).

As a long-shot maybe the interface for multi and mono lingual projects can be unified so that we do not repeat our selves. IMHO it is worth a try.

[jmhofer]

Thanks for all the observations! I'll try to address some of the ones from @xeno-by.

1a), 1e) and 1g) are just aliasing problems, imho. We already have flatMap as alias for mapMany, and the other aliases could easily be added, either directly in rx.Observable, or in a Scala wrapper, if the namespace gets too big otherwise (not my preferred solution). Methods returning options will have to live in a Scala wrapper, however.

I didn't take a look at 1b) and collection conversions yet. This really shouldn't be necessary often, if at all.

1c) You're looking at test code here which tests the callback "at the end of the world". This is not really normal usage of the API. The whole RxJava codebase is refreshingly (though not completely) free of mutability, if you ask me. Also, the Rx API is actually there to allow you to handle all your events in a functional, immutable way. - In short, I don't see a problem here.

1d) is Scala-specific. By-name parameters are a great feature for methods like defer, I agree. We should make use of them in the Scala wrapper.

I'm not worried about 1f) either.

1h) I like the verbose name in this specific case. Imho, it's the same as asInstanceOf in Scala: Maybe I'm evil, but I like to punish people who do things they probably shouldn't do. :) Ok, there will of course be valid use cases for blocking observables, but there shouldn't be too many of them, hopefully.

2. is quite a big problem, if you ask me. A backport of the fix to Scala 2.10 would be great. We can probably get around this with a dedicated Scala wrapper, though.

And the reason for 2.10.1 instead of 2.10.2 is currently just a technical problem with the build system. I, too, hope that this will be fixed soon.

[abersnaze]

1c) Even that can be made functional if you replace the subscribe() with .toList().toBlockingObservable().single();

[jmhofer]

@abersnaze You're right, good point.

I started a little experiment with a more idiomatic Scala wrapper. It's just humble beginnings currently, though. You can find it here.

Comments and forks are welcome, of course. I started from the super-extends pull request in order to check how this whole thing feels in Scala.

[benjchristensen]

Scala 2.10.2 support was added in version 0.11.2 (https://github.com/Netflix/RxJava/releases/tag/0.11.2).

[adriaanm]

1. Implicit-based integration with Java-based observables has a serious flaw as it doesn't support parameter type inference for lambdas. The situation isn't going to change until https://issues.scala-lang.org/browse/SI-6221 is fixed. It's definitely fixed in 2.11, and at EPFL we have a compiler plugin that backports the fix to 2.10. Probably Typesafe folks can be convinced to do the backport in one of 2.10.x releases (/cc @adriaanm @gkossakowski @retronym @JamesIry).

Sorry, we don't change type inference in minor versions unless there's a critical bug.
The risk of regression in source compatibility is too high.

[benjchristensen]

The cross language compatibility pointed out by @benjchristensen is a reasonable concern. However, I think there is a lot of monolingual projects out there that would greatly benefit from the idiomatic Scala library. It seems to me that adding a Scala wrapper can be only a plus in those cases.

I completely agree if a wrapper is the only way to achieve idiomatic support. Monolingual projects should take first priority as that is the common case.