TakeWhile version 7

[marcosemplici]

Hi,
I love Loophp Collection!
However I would like to ask you why TakeWhile has changed drastically.
It used to be lazy, now it uses cache if I'm not mistaken and this thing can impact on performance?
In my Business Logic I iterates tens of thousands...

best regards

[drupol]

Ciao Marco,

Thanks for using loophp/collection ! I'm glad you like it :)

I'll investigate a little bit and try to remember why I changed the implementation.

IIRC, I think I changed it to reuse existing stuff instead of making something completely custom.
Sometimes it's a good idea, sometimes it's not.

That said, the implementation is based on Every which is definitely lazy. The only thing that has changed is that a Cache decorator was added.

Anyhow, I will definitely investigate.

What would be nice is that if you could provide a code sample and the 2 versions numbers of loophp/collection (before and after).

Thanks!

[marcosemplici]

Okay! I will give you the two examples!

However my perplexity was: is the new version of takeWhile based on Every and therefore scrolling through the whole collection?

Thanks!

[drupol]

However my perplexity was: is the new version of takeWhile based on Every and therefore scrolling through the whole collection?

No. The current implementation of Every yields each element of the iterable(or collection) until a predicate returns a boolean false value.

In the meantime, I made a better version of TakeWhile which does not require Every, I'll polish it tonight and post a pull request for you to review as well tonight.

[marcosemplici]

Ok! Thanks!

But Every yield only one element, am I missing something? true o false
Then it can yield true only when it has finished the whole collection.

I'm sorry to "bother" you!

[drupol]

No, if you look at the source of the Every operation, you can see that it will break the loop once the predicate is false., so no, it won't loop the whole collection all the time.

And no worries, you don't bother at all !

[marcosemplici]

This is snippet:

$collection = Collection::fromIterable(
            (static function (): \Generator {
                for ($i = 0; $i < 1_000_000; $i++) {
                    yield $i;
                }
            })())
            ->takeWhile(static function (int $value): bool {
                echo round(memory_get_usage(true) / 1048576, 2);
                return $value < 999_900;
            });
        $collection->count();

With version 6.0.3 the memory is stable, instead with version 7 the memory increases

[stefanofago73]

HI,

this is an interesting discussion... Thanx Both!
Only to understand if I've seen the problem proposed by Marco, in this code:

https://github.com/loophp/collection/blob/7.1.0/src/Operation/TakeWhile.php

We see this costruction:

$iteratorAggregate = new CachingIteratorAggregate((new IterableIteratorAggregate($iterable))->getIterator());

This kind of decoration, looking at loophp/iterators, at the end, is related to a CachedIterator with FULLCACHE:

new CachingIterator(
            $iterator,
            CachingIterator::FULL_CACHE
        );

[ https://www.php.net/cachingiterator ]:

...Since the data will be loaded into the memory, the CachingIterator is able to check whether the given iterator has a next element...

Can this choice be a problem with collections made of thousand of elements?
Hope I've understood the problem...

Thank for Your work!

[drupol]

I just updated the Init and TakeWhile operations based on your inputs (see 09eb135 and 9d4bee7)

TakeWhile do not use a CachingIterator now and it is now more efficient.

[drupol]

I just tagged 7.1.1, it includes the fix :)

[marcosemplici]

Great!
Unfortunately Since, Until and DropWhile operator are also affected by the same problem!
Thanks!

[drupol]

Thanks, I just fixed this in version 7.1.2.

[drupol]

Just FYI, I restored an older version of these operations which doesn't have any memory issues.

Maybe one day I'll find a better algorithm for these, but in the meantime, this memory issue is fixed!

[stefanofago73]

...Maybe, later, You can reinsert the cache considering the DSL node with some more configurations...
...Maybe it could be something like:

`static fn (callable ...$callbacks, CachePolicy $policy): Closure => ...`

to reach something like:

->takeWhile($isSmallerThanThree, noCache())
->takeWhile($isSmallerThanThree, cacheSize(10))

However... I stop!
...Thanks for Your time and for a nice open source like Lookphp, really nice idea and an amazing work!

[drupol]

Hey,

Thanks for the idea but I don't think restoring the cache is needed since it works without it.
Do you have a use case where this would be useful?

Thx!

[stefanofago73]

Sorry for late reply...

...Before setting out some considerations, it would be interesting to know the path you wanted to take with the introduction of caches, on which we have intervened: perhaps there are similar ideas/intuitions?

I don't know all the Loophp APIs (not a loophp expert), so what I'm going to explain should be read as a set of general considerations.

In addition to offering a monadic approach to collections, Loophp has the advantage of promoting a "stream" development of data (iterators and generators) which generally defines a constant footprint in memory use and a strong composability of the processing steps: everything is almost entirely stateless. In streams, however, it may be necessary, in some cases, to pre-cache the data to filter or process it and, if something does not conform to what was expected, it is possible to go back (pushback-stream). Another issue is to operate not one element at a time but in micro-batch: this approach allows you to process "near" data sets (based on what criteria?) directly in memory before continuing with the iteration. Another argument could be to operate on time intervals in which the data accumulate and then be processed in one go and become a new element (coalescence)

I hope I have answered your questions and provided some useful ideas, but they are just considerations anyway. Thank you again for everything lookphp provides us today and I wish you good work!

[drupol]

...Before setting out some considerations, it would be interesting to know the path you wanted to take with the introduction of caches, on which we have intervened: perhaps there are similar ideas/intuitions?

While making those four operations, I noticed that there was something that could be abstracted and set common. I didn't succeeded but I succeeded to build those operations using the Every and Limit operations.

Compare by yourself the body of those operations (at commit 61f15cd), you'll see by yourself that they are almost the same:

• https://github.com/loophp/collection/blob/61f15cd053457ebcdc0ecc9135525b3f7cb27941/src/Operation/DropWhile.php
• https://github.com/loophp/collection/blob/61f15cd053457ebcdc0ecc9135525b3f7cb27941/src/Operation/Since.php
• https://github.com/loophp/collection/blob/61f15cd053457ebcdc0ecc9135525b3f7cb27941/src/Operation/TakeWhile.php
• https://github.com/loophp/collection/blob/61f15cd053457ebcdc0ecc9135525b3f7cb27941/src/Operation/Until.php

And for some reason, I added the Caching operator, but I don't remember exactly the reason... I'm scratching my head, but no-can-do !

I don't know all the Loophp APIs (not a loophp expert), so what I'm going to explain should be read as a set of general considerations.

In addition to offering a monadic approach to collections, Loophp has the advantage of promoting a "stream" development of data (iterators and generators) which generally defines a constant footprint in memory use and a strong composability of the processing steps: everything is almost entirely stateless. In streams, however, it may be necessary, in some cases, to pre-cache the data to filter or process it and, if something does not conform to what was expected, it is possible to go back (pushback-stream). Another issue is to operate not one element at a time but in micro-batch: this approach allows you to process "near" data sets (based on what criteria?) directly in memory before continuing with the iteration. Another argument could be to operate on time intervals in which the data accumulate and then be processed in one go and become a new element (coalescence)

This is extremely well summarized, I couldn't do better :)

I hope I have answered your questions and provided some useful ideas, but they are just considerations anyway. Thank you again for everything lookphp provides us today and I wish you good work!

You're very welcome! I'm now curious, are you using it everyday? For which purpose? Tell me more!

[stefanofago73]

I apologize for the late reply: during the weekend I try to think of something else... (even if I don't always succeed XD XD XD )
...I can't talk about elements of my company but the project certainly offers some interesting ideas...
Actually, I would like to understand, also to develop an internal proposition, the enterprise users of Loophp.
Having said that, it is easy to identify Loophp as an infrastructure for the definition of smart processing and transformations
on collections (in the small) and definition of processes of different nature (in the large)
I personally see some aspects of Lookphp:

• could have a good appeal, with a "specific" version in the context of using PHP in serverless
• it would be useful to understand if it is appropriate to provide a test suite or guidelines for optimization, which if it cannot be about speed, could lead to an increase in throughput
• it can become a good project to give birth to CEP (Complex Event Processing) for PHP; this vision somewhat follows the trend of modularization of the project: Iterators, Collections, Events!

Said this...
I cannot guarantee collaborations but your work is very interesting and we hope to see an ever greater diffusion and collaborate as soon as possible

Thank you!

[drupol]

Ok ok.

If you'd like to discuss lively, I'm available for a meeting with your team.

Have a good day :)