Update call back func

[pradeepranwa1]

Update call back function which is set is not triggered, even code does not exist which uses the function i set

[casbin-bot]

@techoner @Nekotoxin

[hsluoyz]

@pradeepranwa1 can you provide more details?

[hsluoyz]

@leeqvip

[pradeepranwa1]

I am using mutiple gunicorn workers

INSTALLED_APPS = [
'casbin_adapter.apps.CasbinAdapterConfig'
]
CASBIN_MODEL = os.path.join("finbox_dashboard/casbin_middleware/authz_model.conf")

watcher = PostgresqlWatcher(host=BANK_CONNECT_APIS_PG_HOST_URL, port=BANK_CONNECT_APIS_PG_PORT,
user=BANK_CONNECT_APIS_PG_USER, password=BANK_CONNECT_APIS_PG_PASSWORD, dbname=BANK_CONNECT_APIS_PG_DBNAME, enforcer=enforcer)
watcher.set_update_callback(enforcer.load_policy)
enforcer.set_watcher(watcher)

using this setup django migration and casbin is working but watcher is not working

for watcher listen/notify service is created in db on any change in casbin policy worker notifies db and other workers are polling that change, but i observed a single worker(single instance of enforcer) fetches all triggers and apart from that set_update_callback is never called

[pradeepranwa1]

I updated the watcher code and added enforcer.load_policy() as mentioned below, but getting
django.db.utils.InterfaceError: connection already closed in the enforcer.load_policy

from casbin_adapter.enforcer import enforcer
POSTGRESQL_CHANNEL_NAME = "casbin_role_watcher"

def casbin_subscription(
    process_conn: Pipe,
    host: str,
    user: str,
    password: str,
    port: Optional[int] = 5432,
    dbname: Optional[str] = "postgres",
    delay: Optional[int] = 2,
    channel_name: Optional[str] = POSTGRESQL_CHANNEL_NAME,
    sslmode: Optional[str] = None,
    sslrootcert: Optional[str] = None,
    sslcert: Optional[str] = None,
    sslkey: Optional[str] = None
):
    # delay connecting to postgresql (postgresql connection failure)
    time.sleep(delay)
    conn = connect(
        host=host,
        port=port,
        user=user,
        password=password,
        dbname=dbname,
        sslmode=sslmode,
        sslrootcert=sslrootcert,
        sslcert=sslcert,
        sslkey=sslkey
    )
    # Can only receive notifications when not in transaction, set this for easier usage
    conn.set_isolation_level(extensions.ISOLATION_LEVEL_AUTOCOMMIT)
    curs = conn.cursor()
    curs.execute(f"LISTEN {channel_name};")
    print("Waiting for casbin policy update")
    while True and not curs.closed:
        if not select([conn], [], [], 5) == ([], [], []):
            print("Casbin policy update identified..")
            conn.poll()
            while conn.notifies:
                notify = conn.notifies.pop(0)
                print(f"Notify: {notify.payload}")
                process_conn.send(notify.payload)
                print("while update", enforcer)
                enforcer.load_policy()

traces

psycopg2.InterfaceError: connection already closed
bank_connect_apis  | 
bank_connect_apis  | The above exception was the direct cause of the following exception:
bank_connect_apis  | 
bank_connect_apis  | Traceback (most recent call last):
bank_connect_apis  |   File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
bank_connect_apis  |     self.run()
bank_connect_apis  |   File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run
bank_connect_apis  |     self._target(*self._args, **self._kwargs)
bank_connect_apis  |   File "/bank_connect_apis/finbox_dashboard/casbin_middleware/casbin_watcher.py", line 53, in casbin_subscription
bank_connect_apis  |     enforcer.load_policy()
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/casbin/core_enforcer.py", line 228, in load_policy
bank_connect_apis  |     raise e
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/casbin/core_enforcer.py", line 207, in load_policy
bank_connect_apis  |     self.adapter.load_policy(new_model)
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/casbin_adapter/adapter.py", line 22, in load_policy
bank_connect_apis  |     for line in lines:
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 394, in __iter__
bank_connect_apis  |     self._fetch_all()
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1867, in _fetch_all
bank_connect_apis  |     self._result_cache = list(self._iterable_class(self))
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 87, in __iter__
bank_connect_apis  |     results = compiler.execute_sql(
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1396, in execute_sql
bank_connect_apis  |     cursor = self.connection.cursor()
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
bank_connect_apis  |     return func(*args, **kwargs)
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 323, in cursor
bank_connect_apis  |     return self._cursor()
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 300, in _cursor
bank_connect_apis  |     with self.wrap_database_errors:
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
bank_connect_apis  |     raise dj_exc_value.with_traceback(traceback) from exc_value
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py", line 301, in _cursor
bank_connect_apis  |     return self._prepare_cursor(self.create_cursor(name))
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py", line 26, in inner
bank_connect_apis  |     return func(*args, **kwargs)
bank_connect_apis  |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 269, in create_cursor
bank_connect_apis  |     cursor = self.connection.cursor()
bank_connect_apis  | django.db.utils.InterfaceError: connection already closed

[hsluoyz]

@trbtm can you take a look at this?

[pradeepranwa1]

@hsluoyz #27
pr for above issue

[trbtm]

@hsluoyz, @pradeepranwa1

I looked into the MR and made some comments there. Generally there are several issues right now (some of which were already mentioned by @pradeepranwa1)

• PostgresqlWatcher.should_reload is never called by Enforcer.enforce although the README kind of implies this. I searched the whole pycasbin code base and should_reload is mentioned nowhere. There are two options to mitigate this
  1. adjust pycasbin
  2. Fix the README (my preference)
• PostgresqlWatcher.update_callback is not called by should_reload. IMHO there are these options
  1. remove it from PostgresqlWatcher and the user has to do that by himself
  2. rename should_reload to something like reload_on_update to reflect the call of update_callback and make the callback handler part of the PostgresqlWatcher.__init__
  3. Add some sort of timer, that polls the pipe e.g. every second and then executes the update handler (my preference)
• PostgresqlWatcher.should_reload returns None if the polling condition is not met and no EOFError occured. It should return False in this case (easy fix)
• Unrelated to this issue: The unit tests are not really testing anything but if some properties were set correctly. Kind of misses the point.

I am willing to work on fixing all of these issues. @hsluoyz any objects to using pytest instead of stdlib unittest?

[trbtm]

#29 PR for the points mentioned.