markAsDirty not working?

[DwayneSmurdon]

Hello,

We are using ebean 13.1.0 and we have a large reference data set, with over 150 tables and growing. We do not want to write getters/setters for each field, even with the help of lombok. We really only write to these tables during testing or data cleanup steps. So, rather that use myModel.setSomeField(toThisValue), we simply use:

myModel.setSomeField = toThisValue;
Of course, this will fail to update the db with an myModel.update() call.

So, we first call myModel.markAsDirty() but the update still fails.

Shouldn't that trigger it to update the entire model or at least check which fields have different values and then update?

How can we use = instead of setXXX and still get an update working as intended?

Any help would be appreciated.

Thank you,

-Dwayne

[rbygrave]

I see this, sorry, haven't got to it yet. In short we need to mark which properties are changed and I need to explain why etc. Should get to this soon, hopefully tomorrow. Cheers, Rob.

[rbygrave]

Background

We have per property changed state. Why?

To optimise SQL UPDATE we only desire to include in the UPDATE the properties that have actually changed. For example, if an entity had 100 properties and we only changed 1 of those we don't really want to include the other 99 in the UPDATE. Another example, if an entity had 5 properties but some of those were VARCHAR(5000) we similarly only wish to include only the properties that were changed. Simply including all properties in an update is not something we want to do frequently or by default.

On a per property basis we can also hold "Old Values". These are used to support ChangeLog etc. When we do a "stateless update" we don't have "Old Values" and this is ok, we just need to know and understand that. Sometimes we are either happy to or must perform a "stateless update" and when we do there are no "Old Values" (to support ChangeLog etc).

With respect to setter methods / mutator methods / setting fields. What is generally happening is that Ebean enhancement is changing PUTFIELD calls to instead call Ebean internal setter methods. The ebean internal setter method checks to see if the value has changed and if so sets changed state on the property, keeps the old value for the property, and sets the overall dirty state on the bean. The overall dirty state on the bean can be set when for example an embedded bean becomes dirty so there are some details on overall dirty state.

---

Shouldn't that trigger it to update the entire model

We still need the detail on which property is changed.

or at least check which fields have different values and then update?

In the absence of anything else, when using a PUTFIELD there is no interception ocuring per say and hence no recognition that the field changed and no storing of an 'Old value'. To say that differently, Ebean only creates the "Old values" on demand (via the interception that is put on the setters/mutator methods). This means that read only cases for Ebean are much cheaper than some other ORMs that instead create and hold the "Old values" regardless.

I'd suggest there are 4 ways of addressing the use case of using PUTFIELD but still getting all the expected dirty state.

• Having a method to mark all property as changed (The existing EntityBeanIntercept::setNewBeanForUpdate() does this)
• Explicitly mark the properties as changed (We have EntityBeanIntercept::setChangedProperty() but it's designed for internal use using propertyIndex so not nice for public use)
• A change such that Ebean enhancement changes these PUTFIELD calls into the appropriate ebean internal setter interception. (This means enhancing the test classes etc that are using the PUTFIELD calls, noting that we do something very similar for query beans)
• Some change to the query such that "Old Values" are created immediately plus some method to do check current value to old value and automatically determine the per property changed state

[rbygrave]

A change such that Ebean enhancement changes these PUTFIELD calls into the appropriate ebean internal setter interception

In case people are not aware there are 3 cases I know of where projects have "simulated property access" via public fields.

1. Play Framework
2. Ebean Query beans
3. Panache / Quarkus

TLDR: We could look to have a feature toggle to support this. Implementation wise we'd do it the the same way as the query bean enhancement. The restrictions / expectations would be:

• All GETFIELD/PUTFIELD calls on types in the package that contains entities are expected to be entities and persistent fields (and we probably wouldn't check those assumptions during enhancement). So field access on non entity classes in the same package as entity classes would be incorrectly changed but I think that would be an ok limitation.

Is this a crazy idea? Well maybe but we see this sort of case a bit and maybe it's worth trying it out. I've been in a couple of conversations now which end with "Java is not getting properties" and for this case where we "kinda get buried in getter/setter pairs for low returns" I'm thinking it's worth trying. The "150 reference tables" case is kind of the perfect case that justifies this sort of approach.

We can make this something that people explicitly opt in on via an entry in ebean.mf (in the same way to how we toggle enhancement features now).

Feedback welcome. Cheers, Rob.

[rPraml]

• A change such that Ebean enhancement changes these PUTFIELD calls into the appropriate ebean internal setter interception. (This means enhancing the test classes etc that are using the PUTFIELD calls, noting that we do something very similar for query beans)

This would probably slow down enhancement, but should be feasible.

• Some change to the query such that "Old Values" are created immediately plus some method to do check current value to old value and automatically determine the per property changed state

This would have the advantage, that it would work also with reflection frameworks.

But how would we handle lazy loads in this case?

My next idea is:
If we have a new entity, we can/must treat each property as changed, so this should not be a Problem
If we load an entity from DB, we could that do in a special mode that loads all properties and consider also all properties as dirty on save. We must pay attention, that we do not load a too large object graph in memory

Maybe something like DB.find(...).select("*").fetch(...).disableLazyLoad() can do the trick (=all selected/fetched properties are included in updates)

Can we reuse the disableLazyLoad flag in EBI?
Or should we use that in conjunction with an other method? ( DB.find(..).select(..).forFullUpdate() ?)
This marks all loaded properties as dirty and could also copy the "old" values on load, so that changelog would work.

This would mean that @DwayneSmurdon has to modify the queries (unless it's not a new bean) and the javadoc should mention that this method may be slower than accessing entities with getter/setters

Roland

[rbygrave]

but should be feasible.

Yeah, I have a working prototype actually.

also with reflection frameworks.

Not really in that reflection frameworks are still going to be doing PUTFIELD / GETFIELD so no that won't work in the sense that those will bypass interception.

handle lazy loads in this case?

Lazy loading, partial objects etc will all work.

My next idea is:

Yeah maybe. This field access approach is actually pretty clean though as no explicit markAsDirty() is required and everything like partial objects and lazy loading and disableLazyLoading etc will all just work as if there were getters/setters. It will also be the least work implementation wise.

What won't work (will bypass interception) is reflection based frameworks like Jackson.

[DwayneSmurdon]

Thanks for looking into this! Here is some very generalized feedback:

As a user:

• I do not want to have to modify my query to select particular fields for update as a sole way to achieve this behavior.
• the optimization I sometimes/often require is measured in lines of code I must write and maintain, rather than memory, network, or cpu optimizations.
• "opting in to certain behavior" via a config file (like ebean.mf) may work well for those needing a more pervasive change in "default" behavior, having a Java way to opt in, such as myBean.checkForChanges().update() or something similar is extremely desirable.
• I sometimes want to override whatever might be in ebean.mf as needed via Java code.
• I may wish to communicate my desire to override at any one of the following times: query time (i.e. when I retrieve from db), when I wish to modify a property (or right after), or at persistence time.
• I understand that if I use '=' to set a property (opposed to setXXX() method), I must either communicate to you which property states I wish you to inspect/mark as dirty, if needed, or ask you to check all properties.
• I may also wish to specify that all fields should be updated, regardless of dirty state; which fields to perform a state/dirty check on, to perform a state/dirty check on all fields.

Options for state change time Java options:
myBean.setProperty("someField", toThisValue).update() // no compile time type checking though.
myBean.myProp = thisValue; myBean.markAsDirty("myProp").update(); // only updates myProp, if value has changed. Or, perhaps only does another specific check on this property, all other fields that are marked dirty, presumably from a call to a setXXX() method, are also updated.

Options for persistence change time Java options:

checkAllFieldsForDirty = true; myBean.myProp = thisValue; myBean.update(checkAllFieldsForDirty) // check all fields
myBean.myProp = thisValue; myBean.update("myProp"); // only updates myProp, if value has changed, Or, perhaps only does another specific check on this property, all other fields that are marked dirty, presumably from a call to a setXXX() method, are also updated.
myBean.myProp = thisValue; myBean.update(myListOfProps); // same as above, but I can specify multiple properties.

If I had to choose 1 option now, it is:

checkAllFieldsForDirty = true; myBean.myProp = thisValue; myBean.update(checkAllFieldsForDirty) // check all fields

Because I don't have to specify property names to check in my code. Specifying property names as text Strings is fragile. If I refactor my field names, my code breaks.

My comments are directed more towards existing, modifiable beans, not new beans. The method/param names are for illustrative purposes; I don't know the library well enough to suggest real names.

By the way, I have assumed that you can't, at compile time, "intercept" a call like myBean.myProp = someValue. But after thinking about it, you probably can, right? If so, then you can probably do this without the user having to explicitly name the properties to inspect for changed state. That is ideal. I still want a Java way, in addition to a config (Ebean.mf) way to communicate to the library that I wish for it to intercept my use of '='.

Does this feedback help at all?

[rbygrave]

I have assumed that you can't, at compile time, "intercept" a call like myBean.myProp = someValue. But after thinking about it, you probably can, right?

Yes we can.

If so, then you can probably do this without the user having to explicitly name the properties to inspect for changed state.

Correct.

That is ideal.

I think so too because everything like dirty checking and lazy loading etc all "just works" making it easier to use. The downside is that its slightly magical - but if we document and explain it well enough we can manage that aspect.

This is what we have done as: #2858 - ENH: Add experimental entity-field-access ... entities with public fields (no getters/setters/accessors required)

Looking to release this today, so if you want to give it a try and provide feedback that would be great. It usually takes a couple of days for the IntelliJ plugin to be officially released.

Cheers, Rob.