Replies: 8 comments 6 replies
-
|
@dbrattli That's fair. My 2 cents (to take with a large grain of salt): IMO each language requires some customization of its Fable F# Core library implementation in some cases. For that reason I also had to customize (port or re-implement) most F# collections for Rust. Yes, that duplicates some Fable code and increases the feature maintenance cost for code that is not shared between languages, but there are some positives too, it lowers dependencies and can be tailored to perform better without impacting other languages. So far we've managed to keep most of Fable code self-contained, and tried hard to not add too many external dependencies that we do not control. My personal non-binding view on including large external libraries as dependencies is that it's fine in cases where it can be justified by better performance, but I'll let @alfonsogarciacaro comment on that. Obviously since it's you doing the work, the decision is yours. I'm sure you've considered just tailoring the existing F# collection implementation for Python needs, since it's probably most of the way there, if the array creation is customized, which was the main issue for Rust too. This may let you fix potential issues easier, if all the code is under your control. A potential benefit of staying with the existing Fable F# collection code (customized for Python) would be the theoretical possibility to some day maybe break out the custom language-specific code and share the common code between all languages. Going completely with an external implementation for Python for all collections will make this less possible (again, personally I'm all for it if the performance is there, case in point is Fable using the native collections for each language to the extent possible). |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the comment @ncave, and for pointing out the Rust specific implementations for core types such as List, Seq, Array, etc. That could be a good solution for Python as well, and as you say make it easier to share common code between the languages in future. I'll take a closer look at that work to see if I could do something similar for Python. Another point I see with the Python port going forwards is that a common use-case will be to use F# compiled code from Python and not the other way around. (With Fable JS it's probably more common to use JS code from F#). Thus it's important that the abstractions we have are Python friendly. That means that we need better control over the produced code to generate Python magic methods etc so that they behave the way other Python code would expect. But that should be possible to generate if I have my own copies of the F# code for these modules. I feel I'm right at the edge of generating Expression from Fable, or Fable Library from Expression. So I need to have a closer look at what you have done for Rust to see if that would make more sense for Python as well. |
Beta Was this translation helpful? Give feedback.
-
|
Hi there! Good to see that discussions for Fable Snake Island are coming back! :) I'll try to leave some notes in case they are helpful: Specialized arrays Funny enough, Don Syme advised against this (having different representations of the array type) in the first steps of Fable. And in a sense we tried to remove them in Fable 3 by making TypedArrays opt-in instead of opt-out, but we couldn't because this broke some libraries. We've managed to make it work but this is mainly due to the fact TypedArrrays are largely compatible with standard arrays in JS and because of some runtime checks and injections in fable-library calls (array constructor). TypedArrays help with microbenchmarks and in code parsing with the REPL, but it's difficult to know how much they help in normal projects. In any case, if it makes sense to use different array representations in Python I'd say go for it but chances are the ad-hoc fixes we've in place for JS won't always fit for Python. Which leads us to... Splitting Fable.Library F# code for each language The ideal situation would be that 90% of Fable code were common and with some magical switches we could compile to multiple languages, but this is probably not possible and I believe the fact that we just copy and adapted big chunks of Fable (vs carefully trying to refactor most code into common utilities) has made it possible to build the prototypes for other languages quickly. It should be ok to also specialize the Fable.Library F# code (or replace it with native code) for each language, and later try to refactor the common parts (as we're doing with Replacements). This would require some coordination to fix common errors that we may find in a particular language. Using external libraries I'm not opposed to using external libraries, in fact we already do it for longs and decimals in JS. Keeping control of Fable generated code was a factor in avoiding dependencies, but mainly it was to keep JS code as small as possible as this important for web apps. It may not be as important in other languages like Python or Rust. But I think external dependencies contribute to reopening an older debate. Distribute Fable.Library as a native package? In Fable 1, fable-library was distributed as an npm package. We changed that in Fable 2 because it was complicated for users to keep the fable-compiler and fable-library versions in sync (and Fable.Core which at the time changed more often). Embedding fable-library files make this transparent to users. However using a native package would have some advantages:
Again, the main drawback is having to sync the versions, we'll need a robust mechanism for this (it will also force us to publish one fable tool for each language separately). And we'll also likely need to decide whether:
What's your opinion? Do you think it's worth give it a new try at publishing fable-library as a package? BTW, it's great to know about you @ncave! Github notifications are the only way we have to communicate and you were inactive for a while, so I was a bit worried. It's indeed difficult times to think about open source in any case. |
Beta Was this translation helpful? Give feedback.
-
|
@alfonsogarciacaro IMO the current |
Beta Was this translation helpful? Give feedback.
-
|
@alfonsogarciacaro I think for Python the preferred way would be to distribute fable-library as a native package on PyPI, or make it optional to use bundled vs external package. This is something I might look into, now that I have more experience from modern Python dependency and package tools such as Poetry (think Paket). In the same way it might also be interesting to distribute Python compiled modules on PyPI than to bundle the source in NuGet's. But versioning and diamond dependencies to fable-library may make this difficult, so bundling the source is perhaps the best option in the short term. Being able to produce high-quality Python libraries from F# with seamless Python interop will probably be my main focus going forwards. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I got an IDEA! 💡 Though I'm not sure if it will actually improve things yet 😸 Since the beginning there's been a dichotomy between Fable.Core Nuget package and fable-library (originally fable-core) precompiled native files. What if we include the precompiled files in the Fable.Core package? This is already possible with Fable Nuget packages and will solve the dichotomy. It may have other advantages too:
Do you think it would work/be helpful? If we go this road, we'll likely have to solve the following:
|
Beta Was this translation helpful? Give feedback.
-
|
Would this also allow published libraries to include a single For example, Nacara is published as 2 libraries:
and each of these NPM packages has it's own
Meaning that when we try to compare the instance it does work. |
Beta Was this translation helpful? Give feedback.
-
|
@MangelMaxime If we go for the "mirror" route this could actually be possible today. The only thing we need to do is publish an npm package with the contents of |
Beta Was this translation helpful? Give feedback.
-
|
It could be really interesting 😇 |
Beta Was this translation helpful? Give feedback.
-
|
@alfonsogarciacaro This might help, and I like the idea of having a single fable-library included in Fable.Core.XYZ, but I don't have the full context yet to say about the possible versioning problems. For Python I have released |
Beta Was this translation helpful? Give feedback.
-
|
Btw did some quick benchmarking of Fable List.fs vs Expression FrozenList. FrozenList is built on Python tuples which means that they are immutable and backed by a C-impl.
(F# not optimized for append, but added it since I tested it) So for Python we really want to get as close to the native Python functions (C-impl) as we can, and do as little Python interpreter code as possible. Thus leveraging the Expression library could speed up the code significantly while producing code that looks more naturally like Python (less F# specific types i.e FSharpList). Expression is an external dependency, but we have control since I'm maintaining that as well. I might give it a try. |
Beta Was this translation helpful? Give feedback.
-
|
@dbrattli Sounds great, obviously native-based implementation should perform much better than a purely Python-based one. The only comment I would make is that, based on my own struggles with list implementation benchmarks, simple test cases don't always give you the best performance picture. It may be better to use more complicated test cases, for example using the list as a stack and hitting it hard, i.e. using the functional list for what it is good for (which, admittedly, is not much from performance point of view, as it is almost always better to use an array than a linked list, where possible, but that's beside the point). That said, it's very plausible that a native-based implementation will always beat a Python-based one no matter what. |
Beta Was this translation helpful? Give feedback.
-
|
@ncave Thanks for the input. I'll experiment some more. The best (long term) approach would probably be to compile List.fs et al to Rust using Fable and then generate a Python extension 😁 |
Beta Was this translation helpful? Give feedback.
-
|
@dbrattli Sure, that's an option if you want. The Rust-compilable F# Core collections are almost feature-complete, there is still some work left but not much. They're passing most of their tests (the commented tests are still todo). |
Beta Was this translation helpful? Give feedback.
-
For Python we currently have an issue with the handling of arrays. Python have arrays on numeric types only which means that for any kind of object we need to use lists instead. In addition, the core built-in types for manipulating binary data are bytes and bytearray. This means that F# arrays for Python currently have to be transpiled to either array, list or bytes. This works, sort of. The problem is with generic functions that needs to create new arrays. Many List.fs / Seq.fs functions in Fable library is backed by arrays. In addition they are implemented in F# which means that it's hard to add Python specific behavior. I don't really know how to solve this issue and have been blocked for some time. Currently considering dropping Seq.fs, List.fs etc in favor of using e.g Expression since that would give me more control and would also mean that List/Seq would not have to be backed by arrays for Python. But checking here first to get some input before I start a major rewrite of the Python library. Any input here would be appreciated @alfonsogarciacaro
Beta Was this translation helpful? Give feedback.
All reactions