From d9eeaf6a0b3cf6ef592e0f6e7e4ea4597ebc6be6 Mon Sep 17 00:00:00 2001 From: Nuutti Sten Date: Thu, 23 Dec 2021 12:37:07 +0000 Subject: [PATCH] created html pages --- README.md | 165 +- docs/_data/sidebars/home_sidebar.yml | 16 +- docs/customclientdata.html | 1955 +++++++++++++++++++ docs/federatedcustomerjourney.html | 1391 +++++++++++++ docs/index.html | 223 +++ docs/results/accuracy.png | Bin 0 -> 35692 bytes docs/sidebar.json | 7 + docs/visuals/federated_learning_process.png | Bin 0 -> 20765 bytes 8 files changed, 3745 insertions(+), 12 deletions(-) create mode 100644 docs/customclientdata.html create mode 100644 docs/federatedcustomerjourney.html create mode 100644 docs/index.html create mode 100644 docs/results/accuracy.png create mode 100644 docs/sidebar.json create mode 100644 docs/visuals/federated_learning_process.png diff --git a/README.md b/README.md index e7ce4b1..62d7584 100644 --- a/README.md +++ b/README.md @@ -1 +1,164 @@ -The readme of your project will be autogenerated by nbdev at nbdev_build_docs based on index.ipynb +# Simulating Customer Journey Prediction in a Federated Learning Setup + + + +```python +%load_ext lab_black +# nb_black if running in jupyter +``` + +## About + +In this study I show that customer paths can be predicted in a federated setup. +The purpose of the work is to show, that this is possible using real customer journey data. +The purpose is not yet to provide optimal solutions. +The goal of the work was to observe machine learning in the given setup. +With these results I hope to encourage further research and innovation on federated learning for privacy preserving personalized digital services. + +Federated learning (FL) is a term used for machine learning in de-centralized +setups where data is distibuted across edge devices [[1](#mcmahan2016communication)]. +In a FL setup a shared model is learned by iteratively aggegating locally trained models. +A FL setup consists of a server updating a global model averaged from aggregate distributed model parameters, +and multiple clients (edge devices, for example mobile phones) that fit the global model to their data. + +![federated learning process](visuals/federated_learning_process.png "Figure 1") + +FL can be used for improved privacy, because instead of data, +only the aggregated model parameters are shared. The raw data remains in the edge devices. +The federated service provider does not rerquire a direct access to the data. +However, the model update information still contains indirect information of the client data, so some privacy risks still prevail [[2](#truong2021privacy)]. + +Because each local dataset is hosted by an edge device and represents a single client, FL setups typically follow a few assumpions: +the data is strongly non-IID (it represents a single client), +unbalanced (the users use the service differently), +massively distributed (there are more clients than data points per client), +and finally limited communication (federated updates require the device to be at rest, plugged to charger and connected to Wi-Fi). + +A customer journey describes the service touchpoints and transitions between them as a customer goes through a service process. +We can model these as states of a stochastic process, and use machine learning to predict the next state based on the previous ones, and other information about the customer. +Knowing the most likely next states would allow a service provider to tailor the service for the individual customers. + +Typically, each customer may only have few data points of their own, e.g. few clicks on a web page since they have entered the service. +This means, that a localized machine learning model using the data of a single customer is not an option. +However, customers may be unwilling to share their data for centralized processing, as they are for example with website 'cookies' [[3](#mueller2018ignore)]. + +With federated learning, the customers could benefit from each other without directly sharing their data. +This makes customer journey prediction is an interesting field of application for federated learning, +especially in the public sector & government context. + +For the public sector, this opens up new means for creating novel digital services. +Cities or governments can provide the citizens personal AI assistants that give them personalized recommendation and advice in various areas of life, +including education, healthcare and career. +These kind of services would require very sensitive data, that the citizens may be unwilling to trust the government with, and that local regulation may prohibit the officials from collecting. +Federated learning can help provide these services, without the need for data share. + +The results may also be applied to other next state prediction problems not involving human customers, +but artificial clients insted. One such example could be predicting faults on IoT devices. + +## The simulation study + +In this work I show how custom data can be used with TFF. I use the customer journey dataset by Bernard & Andritsos [[4](#bernard2019customer)]. +I use 21k data points from 3000 customers (divided into 2100 train and 900 validation clients). There are 16 state labels (including an abstract state describing end of service), 2 customer background features (age and income) and the order of the action. +In the dataset, most customers only had 3-5 events recorded and the max observed was 10. +I create a simple federated learning setup for next state prediction on this dataset. +simulate the federated learning with TFF and compare the results against centralized computation baseline using identical NN model. + +The code is presented in the wiki tabs CustomClientData and FederatedCustomerJourney and in the notebooks 00_data.ipynb and 01_model.ipynb that can be found from the root of the repository. + +### Results + +![accuracy plot](results/accuracy.png "Figure 2") +Train and validation accuracy in comparison to baselines. X-axis shows the federate update iterations, and y axis shows the accuracy. The federated updates are not directly comparable to progression of centralized computation, for which we only plot the result after 20 epochs. + +We observe the federated learning achieves clearly better stats than random guessing, but does not perform as well as the centralized baseline. +Complete reproducibility proved difficult to achieve using TFF, so the results may vary depending on the run. + +View the tabs / notebooks for further info. + +It should be noted that the results were achieved in an optimistic scenario. For example the issues of changing client pool, client availablity, update scheduling and malicious behaviour were not considered in practice. + + + + +## Contents + +The repository core structure is the following: + + data/ # A folder where data is loaded (empty, run 00_data.ipynb to load data) + docs/ # HTML doc files are generated here + ml_federated_customer_journey/ # python module, functions and classes from notebooks are exported here + 00_customclientdata.ipynb # Load and clean data, define related functions + 01_federatedcustomerjourney.ipynb # federated simulation + index.ipynb # this notebook, the repo README and wiki main page are generated from this + settings.ini # project metadata for nbdev tool + + +## How to Install and Run + +To install and run the code: + + git clone git@github.com:City-of-Helsinki/ml_federated_customer_path.git + cd ml_federated_customer_path + # (create and activate virtual environment of your choice) + pip install requirements.txt + nbdev_install_git_hooks + + # run 00_customclientdata.ipynb to load and clean data + # run 01_federatedcustomerjourney.ipynb to run the federated learning simulation + + # to update ml_federated_customer_journey module and docs, call: + nbdev_build_lib && nbdev_build_docs + +Please note that the exact results are not reproducible due to inheritant reproducibility issues of TFF. + +Feel free to try out different setups! + + +## Contributing + +Drop the authors message if you are interested in further research on the topic! (firstname.lastname(at)hel.fi) + +See [here](https://github.com/City-of-Helsinki/ml_federated_customer_journey/blob/master/CONTRIBUTING.md) on how to contribute to the code. + + +## References + +1) McMahan et al. Communication-Efficient Learning of Deep Networks from Decentralized Data. 2021. Google Inc. https://arxiv.org/pdf/1602.05629.pdf + +2) Truong et al. Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective. 2021. https://arxiv.org/abs/2011.05411 + +3) Mueller, R. 76% of Users Ignore Cookie Banners: the User Behaviour After 30 Days of GDPR. 2018. Amazee Metrics (online journal). https://www.amazeemetrics.com/en/blog/76-ignore-cookie-banners-the-user-behavior-after-30-days-of-gdpr/ + +4) Bernard, G., & Andritsos, P. (2019). Contextual and behavioral customer journey discovery using a genetic approach. In 23rd European Conference on Advances in Databases and Information Systems (ADBIS), pages 251–266, Cham. Springer. +Dataset available at: https://customer-journey.me/datasets/ + +This project was built using [nbdev](https://nbdev.fast.ai/) on top of the city of Helsinki [ml_project_template](https://github.com/City-of-Helsinki/ml_project_template). + +## How to Cite + +To cite this work, use: + +```python +%%script False + +@misc{ + sten2021simulating, + title = {Simulating Customer Journey Prediction in a Federated Learning Setup}, + author = {Nuutti A Sten}, + month = {12}, + year = {2021}, + howpublished = {City of Helsinki}, + doi = {ADD DOI HERE}, +} +``` + + Couldn't find program: 'False' + + +## Copyright +{% include note.html content='Edit the year and author below according to your project!' %} +Copyright 2021 City-of-Helsinki. Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this project's files except in compliance with the License. +A copy of the License is provided in the LICENSE file in this repository. + +The Helsinki logo is a registered trademark owned by the city of Helsinki. diff --git a/docs/_data/sidebars/home_sidebar.yml b/docs/_data/sidebars/home_sidebar.yml index 0f0c297..7710cc2 100644 --- a/docs/_data/sidebars/home_sidebar.yml +++ b/docs/_data/sidebars/home_sidebar.yml @@ -10,18 +10,12 @@ entries: title: Overview url: / - output: web,pdf - title: Data - url: data.html + title: CustomClientData + url: customclientdata.html - output: web,pdf - title: Model - url: model.html - - output: web,pdf - title: Loss - url: loss.html - - output: web,pdf - title: Workflow - url: workflow.html + title: FederatedCustomerJourney + url: federatedcustomerjourney.html output: web - title: ml_project_template + title: ml_federated_customer_journey output: web title: Sidebar diff --git a/docs/customclientdata.html b/docs/customclientdata.html new file mode 100644 index 0000000..5b24a30 --- /dev/null +++ b/docs/customclientdata.html @@ -0,0 +1,1955 @@ +--- + +title: CustomClientData + + +keywords: fastai +sidebar: home_sidebar + +summary: "Create custom TFF ClientData from Pandas Dataframe" +description: "Create custom TFF ClientData from Pandas Dataframe" +nb_path: "00_customclientdata.ipynb" +--- + + +
+ + {% raw %} + +
+ +
+
+ +
+ +
+
2021-12-23 12:36:29.882993: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
+
+
+
+ +
+
+ +
+ {% endraw %} + +
+
+

input: none / raw data

+

output: function for creating custom tff ClientData

+

description:

+

In this notebook we generate customer path data, and transform it into tensorflow federated ClientData format. +You can edit the code to load and clean and transform your own client path data.

+ +
+
+
+
+
+

Import relevant modules

+
+
+
+
+
+

Uncomment the following cell to run with the newest version of tff

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
# can causes a duplicate tensorboard install, leading to errors.
+#!pip uninstall --yes tensorboard tb-nightly
+
+#!pip install --quiet --upgrade tensorflow-federated-nightly
+#!pip install --quiet --upgrade nest-asyncio
+#!pip install --quiet --upgrade tb-nightly  # or tensorboard, but not both
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
from pyarrow import feather
+from scipy.special import softmax
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Define notebook parameters

+
+
+
+ {% raw %} + +
+
+ +
+
+
seed = 0
+
+SHUFFLE_BUFFER = 100
+NUM_EPOCHS = 1
+BATCH_SIZE = 32
+
+n_customers = 10000  # number of customers (paths, assuming only one path per customer)
+max_path_length = 100  # limit simulation length
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Define any immediate derivative operations from the parameters:

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
np.random.seed(seed)
+tf.random.set_seed(seed)
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

(or alternatively load your own data and turn it into an applicable format)

Try for example: https://cseweb.ucsd.edu/~jmcauley/datasets.html

+

or https://archive.ics.uci.edu/ml/datasets/Entree+Chicago+Recommendation+Data

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
from pathlib import Path
+import requests, zipfile, io
+
+p = Path().cwd() / "data" / "raw_data"
+if not (p / "customer-journey-unil-ch-datasets").exists():  # check if already loaded
+    r = requests.get(
+        "http://customer-journey.me/wp-content/uploads/2018/02/customer-journey-unil-ch-datasets.zip"
+    )
+    z = zipfile.ZipFile(io.BytesIO(r.content))
+    z.extractall(p)
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
filepaths = (
+    p
+    / "customer-journey-unil-ch-datasets"
+    / "csv"
+    / "configuration6"
+    / "excluding-solution"
+).glob("*.csv")
+
+
+def read_csv_list_to_df(filepaths):
+    max_trace_id = 0
+    df_list = []
+    for f in filepaths:
+        df = pd.read_csv(f)
+        df.trace_id += max_trace_id  # user id count begins from 0 at each file
+        max_trace_id = df.trace_id.max()
+        df_list.append(df)
+
+    return pd.concat(df_list)
+
+
+df = read_csv_list_to_df(filepaths)
+df
+
+ +
+
+
+ +
+
+ +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
trace_idactivitiesowneremployedageincome
00activity_09noyes20-39yohigh
10activity_04noyes20-39yohigh
20activity_02noyes20-39yohigh
31activity_09noyes60-79yohigh
41activity_04noyes60-79yohigh
.....................
42984994activity_07yesno40-59yomiddle
42994994activity_08yesno40-59yomiddle
43004994activity_09yesno40-59yomiddle
43014995activity_10yesyes60-79yomiddle
43024995activity_06yesyes60-79yomiddle
+

21521 rows × 6 columns

+
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
df.nunique()
+
+ +
+
+
+ +
+
+ +
+ + + +
+
trace_id      4996
+activities      15
+owner            2
+employed         2
+age              5
+income           3
+dtype: int64
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
df.activities.unique()
+
+ +
+
+
+ +
+
+ +
+ + + +
+
array(['activity_09', 'activity_04', 'activity_02', 'activity_10',
+       'activity_06', 'activity_03', 'activity_07', 'activity_08',
+       'activity_01', 'activity_05', 'noise_2', 'noise_0', 'noise_4',
+       'noise_3', 'noise_1'], dtype=object)
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
df.age.unique()
+
+ +
+
+
+ +
+
+ +
+ + + +
+
array(['20-39yo', '60-79yo', '0-19yo', '40-59yo', '80yo+'], dtype=object)
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
df.income.unique()
+
+ +
+
+
+ +
+
+ +
+ + + +
+
array(['high', 'middle', 'low'], dtype=object)
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
df.trace_id = df.trace_id.astype("int")
+df.activities = df.activities.astype("category")
+df.owner = df.owner.map(lambda x: 1 if "yes" else 0).astype("uint8")
+df.employed = df.employed.map(lambda x: 1 if "yes" else 0).astype("uint8")
+df.age = df.age.map(
+    {"0-19yo": 0, "20-39yo": 1, "40-59yo": 2, "60-79yo": 3, "80yo+": 4}
+).astype("uint8")
+df.income = df.income.map({"low": 0, "middle": 1, "high": 2}).astype("uint8")
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
df.reset_index(inplace=True)
+df.rename({"index": "action_index"}, axis=1, inplace=True)
+df
+
+ +
+
+
+ +
+
+ +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
action_indextrace_idactivitiesowneremployedageincome
000activity_091112
110activity_041112
220activity_021112
331activity_091132
441activity_041132
........................
2151642984994activity_071121
2151742994994activity_081121
2151843004994activity_091121
2151943014995activity_101131
2152043024995activity_061131
+

21521 rows × 7 columns

+
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
# we need extra category for denoting that client activity has stopped
+activity_ended = f"activity_{df.activities.nunique()+1:d}"
+df.activities = df.activities.cat.add_categories([activity_ended])
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+
    +

  • order of events for customer

    +
    • +

  • next and previous event

    +
    • +
+ +
+
+
+ {% raw %} + +
+
+ +
+
+
df.columns
+
+ +
+
+
+ +
+
+ +
+ + + +
+
Index(['action_index', 'trace_id', 'activities', 'owner', 'employed', 'age',
+       'income'],
+      dtype='object')
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
prev_next_df = pd.DataFrame(
+    columns={
+        "client_id": int,
+        "action_index": "uint8",
+        "prev_activity": "category",
+        "owner": "uint8",
+        "employed": "uint8",
+        "age": "uint8",
+        "income": "uint8",
+        "next_activity": "category",
+    }
+)
+
+for client_id, client_data in df.groupby("trace_id"):
+    client_data.action_index -= client_data.action_index.min()
+    client_data.action_index = client_data.action_index.astype(int)
+    # rename columns
+    buf_df = client_data.rename(
+        {"trace_id": "client_id", "activities": "prev_activity"}, axis=1
+    )
+
+    # add new column for next activity
+    buf_df["next_activity"] = client_data.activities.shift(
+        periods=-1, fill_value=activity_ended
+    ).astype("category")
+    # add buffer to prev_next_df
+    prev_next_df = pd.concat((prev_next_df, buf_df), axis=0, ignore_index=True)
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
prev_next_df
+
+ +
+
+
+ +
+
+ +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
client_idaction_indexprev_activityowneremployedageincomenext_activity
000activity_091112activity_04
101activity_041112activity_02
202activity_021112activity_16
310activity_091132activity_04
411activity_041132activity_02
...........................
2151649942activity_071121activity_08
2151749943activity_081121activity_09
2151849944activity_091121activity_16
2151949950activity_101131activity_06
2152049951activity_061131activity_16
+

21521 rows × 8 columns

+
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
grouped_by_action_count = prev_next_df.groupby("client_id").max()[["action_index"]]
+clients_to_be_dropped = (
+    grouped_by_action_count[grouped_by_action_count.action_index > 10].dropna().index
+)
+# clients_to_be_dropped.to_numpy()
+print(clients_to_be_dropped)
+outliers = (
+    prev_next_df.apply(
+        lambda row: 1 if row.client_id in clients_to_be_dropped else np.nan, axis=1
+    )
+    .dropna()
+    .index
+)
+print(outliers)
+prev_next_df.drop(outliers, inplace=True)
+
+ +
+
+
+ +
+
+ +
+ +
+
Int64Index([999, 1998, 2997, 3996], dtype='int64', name='client_id')
+Int64Index([ 4295,  4296,  4297,  4298,  4299,  4300,  4301,  4302,  8592,
+             8593,  8594,  8595,  8596,  8597,  8598,  8599,  8600,  8601,
+            12881, 12882, 12883, 12884, 12885, 12886, 12887, 12888, 17213,
+            17214, 17215, 17216, 17217, 17218, 17219],
+           dtype='int64')
+
+
+
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
prev_next_df.dropna(inplace=True)
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
prev_next_df.nunique()
+
+ +
+
+
+ +
+
+ +
+ + + +
+
client_id        4992
+action_index       11
+prev_activity      15
+owner               1
+employed            1
+age                 5
+income              3
+next_activity      16
+dtype: int64
+
+ +
+ +
+
+ +
+ {% endraw %} + +
+
+

Drop uninformative columns that only contain one value

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
prev_next_df.drop(["owner", "employed"], axis=1, inplace=True)
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
prev_next_df["action_count"] = prev_next_df.action_index + 1
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
fig, axs = plt.subplots(1, 3, figsize=(15, 4))
+
+for ax, column in zip(axs, ["action_count", "age", "income"]):
+    prev_next_df.groupby("client_id").max()[column].hist(ax=ax, density=True)
+    ax.set_ylabel("%")
+    ax.set_xlabel(column)
+    ax.spines["top"].set_visible(False)
+    ax.spines["right"].set_visible(False)
+    ax.grid(False)
+    [ax.axhline(tick, color="white", linewidth=2) for tick in ax.get_yticks()]
+
+plt.savefig("results/client_histograms.png")
+
+ +
+
+
+ +
+
+ +
+ + + +
+ +
+ +
+ +
+
+ +
+ {% endraw %} + +
+
+

Ok, so most clients have from 3 to 5 actions (including end of journey). considering background information, all groups are well represented.

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
# convert state feature into one hot
+onehot = pd.get_dummies(prev_next_df.prev_activity, prefix="prev")
+prev_next_df[onehot.columns] = onehot
+prev_next_df.drop("prev_activity", axis=1, inplace=True)
+
+# convert label categories into numerical format
+# (this is because for the moment TFF does not support multi-output models)
+prev_next_df[["next_activity"]] = prev_next_df[["next_activity"]].apply(
+    lambda x: x.cat.codes
+)
+prev_next_df = prev_next_df.astype(int)
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
prev_next_df.dtypes
+
+ +
+
+
+ +
+
+ +
+ + + +
+
client_id           int64
+action_index        int64
+age                 int64
+income              int64
+next_activity       int64
+action_count        int64
+prev_activity_01    int64
+prev_activity_02    int64
+prev_activity_03    int64
+prev_activity_04    int64
+prev_activity_05    int64
+prev_activity_06    int64
+prev_activity_07    int64
+prev_activity_08    int64
+prev_activity_09    int64
+prev_activity_10    int64
+prev_noise_0        int64
+prev_noise_1        int64
+prev_noise_2        int64
+prev_noise_3        int64
+prev_noise_4        int64
+prev_activity_16    int64
+dtype: object
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
prev_next_df
+
+ +
+
+
+ +
+
+ +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
client_idaction_indexageincomenext_activityaction_countprev_activity_01prev_activity_02prev_activity_03prev_activity_04...prev_activity_07prev_activity_08prev_activity_09prev_activity_10prev_noise_0prev_noise_1prev_noise_2prev_noise_3prev_noise_4prev_activity_16
00012310000...0010000000
10112120001...0000000000
202121530100...0000000000
31032310000...0010000000
41132120001...0000000000
..................................................................
215164994221730000...1000000000
215174994321840000...0100000000
2151849944211550000...0010000000
215194995031510000...0001000000
2152049951311520000...0000000000
+

21488 rows × 22 columns

+
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
# We need to convert the data into untidy nested format for TFF
+# so that x is a vector and y is a scalar
+cxy_df = pd.DataFrame(columns=["client_id", "x", "y"])
+cxy_df.client_id = prev_next_df.client_id
+cxy_df.x = prev_next_df[
+    prev_next_df.drop(["client_id", "next_activity"], axis=1).columns
+].apply(lambda row: row.to_numpy(), axis=1)
+cxy_df.y = prev_next_df.next_activity
+
+cxy_df
+
+ +
+
+
+ +
+
+ +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
client_idxy
00[0, 1, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, ...3
10[1, 1, 2, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, ...1
20[2, 1, 2, 3, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...15
31[0, 3, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, ...3
41[1, 3, 2, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, ...1
............
215164994[2, 2, 1, 3, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, ...7
215174994[3, 2, 1, 4, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, ...8
215184994[4, 2, 1, 5, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, ...15
215194995[0, 3, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, ...5
215204995[1, 3, 1, 2, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, ...15
+

21488 rows × 3 columns

+
+
+ +
+ +
+
+ +
+ {% endraw %} + +
+
+

Save data for further use

+
+
+
+ {% raw %} + +
+
+ +
+
+
feather.write_feather(cxy_df, "data/preprocessed_data/data.f")
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Create function to convert df into tff ClientData

Following this discussion: https://stackoverflow.com/questions/58965488/how-to-create-federated-dataset-from-a-csv-file

+ +
+
+
+ {% raw %} + +
+ +
+
+ +
+ + +
+

create_tff_client_data_from_df[source]

create_tff_client_data_from_df(df, client_id_col='client_id', sample_size=1.0, shuffle_buffer=100, batch_size=32, num_epochs=20, prefetch_buffer=100, shuffle_seed=42)

+
+

turn pd dataframe into tff client dataset

+ +
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+ +
+ {% endraw %} + +
+
+

Test (well, at least it should not crash)

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
tff_data = create_tff_client_data_from_df(cxy_df)
+
+ +
+
+
+ +
+
+ +
+ +
+
2021-12-23 11:41:39.523696: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
+2021-12-23 11:41:39.783200: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
+2021-12-23 11:41:39.783253: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ml-nuutti-1a-m-mem): /proc/driver/nvidia/version does not exist
+2021-12-23 11:41:39.785298: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
+To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+
+
+
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
tff_data.create_tf_dataset_for_client(tff_data.client_ids[0])
+
+ +
+
+
+ +
+
+ +
+ + + +
+
<PrefetchDataset shapes: OrderedDict([(x, (None, 20)), (y, (None,))]), types: OrderedDict([(x, tf.int64), (y, tf.int32)])>
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
train_data, test_data = tff.simulation.datasets.ClientData.train_test_client_split(
+    tff_data, 500
+)
+
+ +
+
+
+ +
+
+ +
+ +
+
2021-12-23 11:41:39.966178: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
+2021-12-23 11:41:39.966611: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2095190000 Hz
+
+
+
+ +
+
+ +
+ {% endraw %} + +
+ + diff --git a/docs/federatedcustomerjourney.html b/docs/federatedcustomerjourney.html new file mode 100644 index 0000000..a8349b1 --- /dev/null +++ b/docs/federatedcustomerjourney.html @@ -0,0 +1,1391 @@ +--- + +title: FederatedCustomerJourney + + +keywords: fastai +sidebar: home_sidebar + + + +nb_path: "01_federatedcustomerjourney.ipynb" +--- + + +
+ + {% raw %} + +
+ +
+ {% endraw %} + +
+
+

input: data + tff conversion function from CustomClientData

+

output: TFF model for predicting customer paths

+

description:

+

Simulating federated learning on predicting customer paths.

+ +
+
+
+
+
+

Import relevant modules

+
+
+
+ {% raw %} + +
+
+ +
+
+
# uncomment this cell to get the newest version of tff
+
+# tensorflow_federated_nightly also bring in tf_nightly, which
+# can causes a duplicate tensorboard install, leading to errors.
+#!pip uninstall --yes tensorboard tb-nightly
+
+#!pip install --quiet --upgrade tensorflow-federated-nightly
+#!pip install --quiet --upgrade nest-asyncio
+#!pip install --quiet --upgrade tb-nightly  # or tensorboard, but not both
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
import collections
+import matplotlib.pyplot as plt
+import numpy as np
+import nest_asyncio
+
+nest_asyncio.apply()
+
+from pathlib import Path
+from pyarrow import feather
+import pandas as pd
+
+import tensorflow as tf
+import tensorflow_federated as tff
+
+
+from ml_federated_customer_journey.customclientdata import (
+    create_tff_client_data_from_df,
+)
+
+ +
+
+
+ +
+
+ +
+ +
+
2021-12-23 12:20:40.264164: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
+
+
+
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
train_test_client_split = tff.simulation.datasets.ClientData.train_test_client_split
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

You can also view the results using tensorboard:

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
%load_ext tensorboard
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
tff.federated_computation(lambda: "Hello, World!")()
+
+ +
+
+
+ +
+
+ +
+ +
+
2021-12-23 12:20:42.695475: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
+2021-12-23 12:20:42.929745: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
+2021-12-23 12:20:42.929813: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ml-nuutti-1a-m-mem): /proc/driver/nvidia/version does not exist
+2021-12-23 12:20:42.930471: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
+To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2021-12-23 12:20:42.956578: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
+2021-12-23 12:20:42.964904: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2095190000 Hz
+
+
+
+ +
+ + + +
+
b'Hello, World!'
+
+ +
+ +
+
+ +
+ {% endraw %} + +
+
+

Define notebook parameters

You can easily try different setups by running the notebook with papermill using different parameters.

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
seed = 0
+data_filepath_parts = ("data", "preprocessed_data", "data.f")  # for pathlib
+test_split = 0.2
+
+NUM_EPOCHS = 20
+BATCH_SIZE = 32
+SHUFFLE_BUFFER = 100
+FEDERATED_UPDATES = 50
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Make immediate derivations from the parameters:

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
np.random.seed(seed)
+tf.random.set_seed(seed)
+data_filepath = Path.cwd() / Path(*data_filepath_parts)
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Load Data

+
+
+
+ {% raw %} + +
+
+ +
+
+
df = feather.read_feather(data_filepath)
+df.head()
+
+ +
+
+
+ +
+
+ +
+ + +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
client_idxy
00[0, 1, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, ...3
10[1, 1, 2, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, ...1
20[2, 1, 2, 3, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...15
31[0, 3, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, ...3
41[1, 3, 2, 2, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, ...1
+
+
+ +
+ +
+
+ +
+ {% endraw %} + +
+
+

How many data points

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
df.shape
+
+ +
+
+
+ +
+
+ +
+ + + +
+
(21488, 3)
+
+ +
+ +
+
+ +
+ {% endraw %} + +
+
+

How many features

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
NUM_FEATURES = df.x[0].shape[0]
+NUM_FEATURES
+
+ +
+
+
+ +
+
+ +
+ + + +
+
20
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
NUNIQUE_LABELS = df.y.nunique()
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Convert into tff ClientData (training + testing datasets):

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
client_data = create_tff_client_data_from_df(df, sample_size=1)
+train_data, test_data = train_test_client_split(
+    client_data, int(df.client_id.nunique() * test_split)
+)
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Test and train dataset size (number of clients in each set)

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
len(train_data.client_ids)
+
+ +
+
+
+ +
+
+ +
+ + + +
+
2141
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
len(test_data.client_ids)
+
+ +
+
+
+ +
+
+ +
+ + + +
+
998
+
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
ELEMENT_SPEC = train_data.element_type_structure
+ELEMENT_SPEC
+
+ +
+
+
+ +
+
+ +
+ + + +
+
OrderedDict([('x', TensorSpec(shape=(None, 20), dtype=tf.int64, name=None)),
+             ('y', TensorSpec(shape=(None,), dtype=tf.int32, name=None))])
+
+ +
+ +
+
+ +
+ {% endraw %} + +
+
+

Create Federeted ML Process with TFF

+
+
+
+ {% raw %} + +
+
+ +
+
+
def create_keras_model():
+    """
+    Return new keras model instance
+    """
+    visible = tf.keras.layers.Input(shape=(NUM_FEATURES,))
+
+    hidden1 = tf.keras.layers.Dense(
+        48,
+        activation=None,
+        name="l1relu",
+    )(visible)
+    output = tf.keras.layers.Dense(
+        NUNIQUE_LABELS + 1,
+        activation="softmax",
+        name="l3softmax",
+    )(hidden1)
+    model = tf.keras.Model(inputs=visible, outputs=output)
+    return model
+
+ +
+
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
def model_fn():
+    """
+    Create tff model (keras model + data format + loss & metrics)
+    """
+    # We _must_ create a new model here, and _not_ capture it from an external
+    # scope. TFF will call this within different graph contexts.
+    keras_model = create_keras_model()
+    return tff.learning.from_keras_model(
+        keras_model,
+        input_spec=collections.OrderedDict(
+            x=ELEMENT_SPEC["x"],
+            y=ELEMENT_SPEC["y"],
+        ),
+        loss=tf.keras.losses.SparseCategoricalCrossentropy(),
+        metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],
+    )
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Create federated averaging process:

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
iterative_process = tff.learning.build_federated_averaging_process(
+    model_fn,
+    client_optimizer_fn=lambda: tf.keras.optimizers.RMSprop(learning_rate=0.02),
+    server_optimizer_fn=lambda: tf.keras.optimizers.RMSprop(learning_rate=1.0),
+)
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Initialize federated averaging process:

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
state = iterative_process.initialize()
+
+ +
+
+
+ +
+
+ +
+ +
+
WARNING:tensorflow:From /anaconda/envs/customerjourney/lib/python3.8/site-packages/tensorflow_federated/python/core/impl/compiler/tensorflow_computation_transformations.py:59: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
+Instructions for updating:
+Use `tf.compat.v1.graph_util.extract_sub_graph`
+
+
+
+ +
+ +
+
WARNING:tensorflow:From /anaconda/envs/customerjourney/lib/python3.8/site-packages/tensorflow_federated/python/core/impl/compiler/tensorflow_computation_transformations.py:59: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
+Instructions for updating:
+Use `tf.compat.v1.graph_util.extract_sub_graph`
+
+
+
+ +
+
+ +
+ {% endraw %} + +
+
+

Create federated evaluation process (validation):

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
evaluation = tff.learning.build_federated_evaluation(model_fn)
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Function for loading client data in batches:

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
def batch_client_data(client_data, batch_size=BATCH_SIZE):
+    batch = [
+        client_data.create_tf_dataset_for_client(client_data.client_ids[idx])
+        for idx in np.random.choice(
+            np.arange(len(client_data.client_ids)), size=BATCH_SIZE
+        )
+    ]
+    return batch
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

Federated training & evaluation:

+
+
+
+ {% raw %} + +
+
+ +
+
+
%%time
+# first iteration
+state, train_metrics = iterative_process.next(state, batch_client_data(train_data))
+test_metrics = evaluation(state.model, batch_client_data(test_data))
+
+# see progress
+print("federated_update  {}, loss={:.2f}, accuracy={:.2f}".format(0, train_metrics['train']['loss'], train_metrics['train']['sparse_categorical_accuracy']))
+
+# save results
+metrics_df = pd.DataFrame(
+    {
+        "federated_update": [0],
+        "train_loss": [train_metrics["train"]["loss"]],
+        "train_accuracy": [train_metrics["train"]["sparse_categorical_accuracy"]],
+        "train_size": [train_metrics["stat"]["num_examples"]],
+        "test_loss": [test_metrics["loss"]],
+        "test_accuracy": [test_metrics["sparse_categorical_accuracy"]],
+    }
+)  # , 'test_loss': float, 'test_size':int})
+
+# run federated update cycles
+for i in range(FEDERATED_UPDATES):
+    # update, get train metrics
+    state, train_metrics = iterative_process.next(state, batch_client_data(train_data))
+    # evaluate
+    test_metrics = evaluation(state.model, batch_client_data(test_data))
+    # save results
+    metrics_df = pd.concat(
+        (
+            metrics_df,
+            pd.DataFrame(
+                {
+                    "federated_update": [i + 1],
+                    "train_loss": [train_metrics["train"]["loss"]],
+                    "train_accuracy": [
+                        train_metrics["train"]["sparse_categorical_accuracy"]
+                    ],
+                    "train_size": [train_metrics["stat"]["num_examples"]],
+                    "test_loss": [test_metrics["loss"]],
+                    "test_accuracy": [test_metrics["sparse_categorical_accuracy"]],
+                }
+            ),
+        ),
+        axis=0,
+    )
+    # see progress
+    print("federated_update  {}, loss={:.2f}, accuracy={:.2f}".format(i + 1, train_metrics['train']['loss'], train_metrics['train']['sparse_categorical_accuracy']))
+metrics_df.set_index("federated_update", drop=True, inplace=True)
+
+ +
+
+
+ +
+
+ +
+ +
+
2021-12-23 12:21:27.136377: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:27.656336: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:27.748344: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:33.631581: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:35.353653: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:35.392388: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:36.808093: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:36.908555: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:36.981553: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:38.516368: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:41.828356: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:43.524509: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:43.530324: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:45.048431: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:45.100342: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:48.485201: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:48.509787: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:51.612459: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:51.612554: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:54.912343: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:54.972350: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:21:57.984340: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:01.244655: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:01.255567: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:01.265025: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:01.276832: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:02.636345: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:02.729645: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:06.080092: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:06.196943: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:07.739155: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:07.764423: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:07.775073: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:09.325119: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:09.340700: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:10.832119: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:10.837680: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:12.590122: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:14.121172: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:14.140444: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:14.146665: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:17.100732: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:18.959558: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:18.995023: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:22.011549: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:22.020355: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:22.025446: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:22.047956: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:23.838793: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:23.841719: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:25.348562: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:25.391140: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:25.399491: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:25.399578: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:26.892338: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:26.903450: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:28.312364: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:28.376336: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:28.378961: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:28.413330: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:28.418083: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:28.438301: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:30.176286: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:30.210613: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:31.728869: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:31.791241: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:34.796362: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:34.870061: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:36.608354: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:36.609238: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:37.998275: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:38.128351: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:38.169969: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:39.537369: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:39.617985: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:42.934535: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:44.518924: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:44.536359: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:49.264395: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:50.889295: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:53.864860: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:55.628354: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:57.070208: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:58.596341: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:58.638823: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:22:58.639441: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:00.108343: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:00.110461: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:00.169729: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:01.936760: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:04.959681: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:04.966119: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:04.981693: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:06.482261: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:06.553552: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:08.320345: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:08.339202: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:11.368930: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:11.425565: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:11.428588: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:11.430548: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:12.919752: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:14.824729: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:16.393254: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:17.890446: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:17.902777: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:19.475934: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:25.901253: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:27.525006: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:32.094540: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:32.112477: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:32.144562: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:33.941747: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:35.410229: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:35.424451: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:35.479841: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:36.872498: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:36.971176: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:38.447056: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:38.525556: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:40.268701: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:41.746859: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:41.751637: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:44.912351: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:46.676215: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:49.799873: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:53.104417: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:53.140458: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:56.161489: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:57.488345: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:23:59.472940: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:24:01.010235: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:24:01.032257: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:24:02.489017: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:24:04.062348: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+2021-12-23 12:24:07.927882: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
+
+
+
+ +
+ +
+
federated_update  0, loss=2.30, accuracy=0.33
+federated_update  1, loss=14.91, accuracy=0.07
+federated_update  2, loss=12.05, accuracy=0.25
+federated_update  3, loss=11.01, accuracy=0.32
+federated_update  4, loss=14.65, accuracy=0.09
+federated_update  5, loss=9.02, accuracy=0.44
+federated_update  6, loss=12.66, accuracy=0.21
+federated_update  7, loss=9.11, accuracy=0.43
+federated_update  8, loss=11.26, accuracy=0.30
+federated_update  9, loss=11.19, accuracy=0.30
+federated_update  10, loss=10.50, accuracy=0.34
+federated_update  11, loss=11.39, accuracy=0.29
+federated_update  12, loss=8.87, accuracy=0.44
+federated_update  13, loss=9.65, accuracy=0.39
+federated_update  14, loss=9.11, accuracy=0.43
+federated_update  15, loss=8.27, accuracy=0.48
+federated_update  16, loss=13.42, accuracy=0.16
+federated_update  17, loss=6.81, accuracy=0.57
+federated_update  18, loss=10.34, accuracy=0.35
+federated_update  19, loss=9.33, accuracy=0.42
+federated_update  20, loss=9.08, accuracy=0.43
+federated_update  21, loss=8.00, accuracy=0.50
+federated_update  22, loss=11.33, accuracy=0.29
+federated_update  23, loss=10.83, accuracy=0.32
+federated_update  24, loss=11.91, accuracy=0.26
+federated_update  25, loss=9.72, accuracy=0.39
+federated_update  26, loss=11.46, accuracy=0.29
+federated_update  27, loss=7.77, accuracy=0.51
+federated_update  28, loss=10.68, accuracy=0.33
+federated_update  29, loss=8.59, accuracy=0.46
+federated_update  30, loss=10.04, accuracy=0.37
+federated_update  31, loss=9.78, accuracy=0.38
+federated_update  32, loss=9.94, accuracy=0.38
+federated_update  33, loss=12.65, accuracy=0.21
+federated_update  34, loss=7.63, accuracy=0.52
+federated_update  35, loss=11.50, accuracy=0.28
+federated_update  36, loss=8.16, accuracy=0.49
+federated_update  37, loss=10.84, accuracy=0.32
+federated_update  38, loss=7.17, accuracy=0.54
+federated_update  39, loss=12.02, accuracy=0.24
+federated_update  40, loss=9.21, accuracy=0.43
+federated_update  41, loss=10.20, accuracy=0.36
+federated_update  42, loss=9.85, accuracy=0.39
+federated_update  43, loss=10.54, accuracy=0.34
+federated_update  44, loss=9.02, accuracy=0.42
+federated_update  45, loss=10.84, accuracy=0.32
+federated_update  46, loss=9.34, accuracy=0.42
+federated_update  47, loss=12.42, accuracy=0.22
+federated_update  48, loss=9.63, accuracy=0.40
+federated_update  49, loss=10.28, accuracy=0.35
+federated_update  50, loss=10.26, accuracy=0.36
+CPU times: user 6min 14s, sys: 14.5 s, total: 6min 28s
+Wall time: 3min 1s
+
+
+
+ +
+
+ +
+ {% endraw %} + +
+
+

Create baseline (centralized model)

Compare federated learning to running same in a centralized setup. +The learning speed is not directly comparable, because with centralized model we only do epochs, +whereas with federated learning there are decentralized epochs and federated updates alternating.

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
%%time
+# create train and test data for centralized setup
+# (use the same train and test data, although it is passed to the models in different ways)
+train_data_centralized = (
+    train_data.create_tf_dataset_from_all_clients()
+    .map(lambda x: (x["x"][0], x["y"][0]))
+    .shuffle(SHUFFLE_BUFFER)
+    .batch(BATCH_SIZE)
+    .prefetch(tf.data.AUTOTUNE)
+)
+
+test_data_centralized = (
+    test_data.create_tf_dataset_from_all_clients()
+    .map(lambda x: (x["x"][0], x["y"][0]))
+    .shuffle(SHUFFLE_BUFFER)
+    .batch(BATCH_SIZE)
+    .prefetch(tf.data.AUTOTUNE)
+)
+
+# create centralized model
+
+centralized_model = create_keras_model()
+
+centralized_model.compile(
+    loss="sparse_categorical_crossentropy",
+    metrics=["sparse_categorical_accuracy"],
+    optimizer="RMSprop",
+)
+
+# fit and evaluate
+history = centralized_model.fit(
+    train_data_centralized,
+    validation_data=test_data_centralized,
+    epochs=NUM_EPOCHS,
+    batch_size=BATCH_SIZE,
+)
+
+# view results
+pd.DataFrame(history.history).plot()
+
+ +
+
+
+ +
+
+ +
+ +
+
WARNING:tensorflow:AutoGraph could not transform <function <lambda> at 0x7ff4d59c4f70> and will run it as-is.
+Cause: could not parse the source code of <function <lambda> at 0x7ff4d59c4f70>: no matching AST found
+To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
+WARNING: AutoGraph could not transform <function <lambda> at 0x7ff4d59c4f70> and will run it as-is.
+Cause: could not parse the source code of <function <lambda> at 0x7ff4d59c4f70>: no matching AST found
+To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
+WARNING:tensorflow:AutoGraph could not transform <function <lambda> at 0x7ff4d4001670> and will run it as-is.
+Cause: could not parse the source code of <function <lambda> at 0x7ff4d4001670>: no matching AST found
+To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
+WARNING: AutoGraph could not transform <function <lambda> at 0x7ff4d4001670> and will run it as-is.
+Cause: could not parse the source code of <function <lambda> at 0x7ff4d4001670>: no matching AST found
+To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
+Epoch 1/20
+210/210 [==============================] - 10s 46ms/step - loss: 1.8254 - sparse_categorical_accuracy: 0.4806 - val_loss: 1.4598 - val_sparse_categorical_accuracy: 0.6456
+Epoch 2/20
+210/210 [==============================] - 10s 46ms/step - loss: 1.4857 - sparse_categorical_accuracy: 0.6144 - val_loss: 1.2795 - val_sparse_categorical_accuracy: 0.6621
+Epoch 3/20
+210/210 [==============================] - 9s 45ms/step - loss: 1.5108 - sparse_categorical_accuracy: 0.5861 - val_loss: 1.3949 - val_sparse_categorical_accuracy: 0.6128
+Epoch 4/20
+210/210 [==============================] - 10s 46ms/step - loss: 1.3662 - sparse_categorical_accuracy: 0.6013 - val_loss: 1.2798 - val_sparse_categorical_accuracy: 0.6761
+Epoch 5/20
+210/210 [==============================] - 10s 45ms/step - loss: 1.2101 - sparse_categorical_accuracy: 0.6661 - val_loss: 1.1067 - val_sparse_categorical_accuracy: 0.6978
+Epoch 6/20
+210/210 [==============================] - 9s 44ms/step - loss: 1.3494 - sparse_categorical_accuracy: 0.6034 - val_loss: 1.3121 - val_sparse_categorical_accuracy: 0.6304
+Epoch 7/20
+210/210 [==============================] - 10s 46ms/step - loss: 1.2088 - sparse_categorical_accuracy: 0.6489 - val_loss: 1.0820 - val_sparse_categorical_accuracy: 0.6790
+Epoch 8/20
+210/210 [==============================] - 10s 46ms/step - loss: 1.1442 - sparse_categorical_accuracy: 0.6738 - val_loss: 1.1076 - val_sparse_categorical_accuracy: 0.6650
+Epoch 9/20
+210/210 [==============================] - 10s 47ms/step - loss: 1.2855 - sparse_categorical_accuracy: 0.6379 - val_loss: 1.1851 - val_sparse_categorical_accuracy: 0.6738
+Epoch 10/20
+210/210 [==============================] - 10s 45ms/step - loss: 1.4220 - sparse_categorical_accuracy: 0.5843 - val_loss: 1.3356 - val_sparse_categorical_accuracy: 0.6245
+Epoch 11/20
+210/210 [==============================] - 10s 46ms/step - loss: 1.3806 - sparse_categorical_accuracy: 0.6065 - val_loss: 1.3161 - val_sparse_categorical_accuracy: 0.6449
+Epoch 12/20
+210/210 [==============================] - 10s 46ms/step - loss: 1.2324 - sparse_categorical_accuracy: 0.6366 - val_loss: 1.1956 - val_sparse_categorical_accuracy: 0.6534
+Epoch 13/20
+210/210 [==============================] - 10s 46ms/step - loss: 1.3406 - sparse_categorical_accuracy: 0.6060 - val_loss: 1.2209 - val_sparse_categorical_accuracy: 0.6310
+Epoch 14/20
+210/210 [==============================] - 10s 45ms/step - loss: 1.3025 - sparse_categorical_accuracy: 0.6123 - val_loss: 1.2510 - val_sparse_categorical_accuracy: 0.6388
+Epoch 15/20
+210/210 [==============================] - 10s 46ms/step - loss: 1.2434 - sparse_categorical_accuracy: 0.6559 - val_loss: 1.2067 - val_sparse_categorical_accuracy: 0.6654
+Epoch 16/20
+210/210 [==============================] - 10s 46ms/step - loss: 1.4265 - sparse_categorical_accuracy: 0.6081 - val_loss: 1.3274 - val_sparse_categorical_accuracy: 0.6702
+Epoch 17/20
+210/210 [==============================] - 10s 45ms/step - loss: 1.2236 - sparse_categorical_accuracy: 0.6579 - val_loss: 1.1487 - val_sparse_categorical_accuracy: 0.6907
+Epoch 18/20
+210/210 [==============================] - 10s 45ms/step - loss: 0.9367 - sparse_categorical_accuracy: 0.7356 - val_loss: 0.8296 - val_sparse_categorical_accuracy: 0.7944
+Epoch 19/20
+210/210 [==============================] - 10s 45ms/step - loss: 1.3789 - sparse_categorical_accuracy: 0.5812 - val_loss: 1.3323 - val_sparse_categorical_accuracy: 0.6054
+Epoch 20/20
+210/210 [==============================] - 10s 45ms/step - loss: 1.3409 - sparse_categorical_accuracy: 0.6163 - val_loss: 1.2852 - val_sparse_categorical_accuracy: 0.6177
+CPU times: user 4min 18s, sys: 14.3 s, total: 4min 32s
+Wall time: 3min 48s
+
+
+
+ +
+ +
+
WARNING:tensorflow:AutoGraph could not transform <function <lambda> at 0x7ff4d59c4f70> and will run it as-is.
+Cause: could not parse the source code of <function <lambda> at 0x7ff4d59c4f70>: no matching AST found
+To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
+WARNING:tensorflow:AutoGraph could not transform <function <lambda> at 0x7ff4d4001670> and will run it as-is.
+Cause: could not parse the source code of <function <lambda> at 0x7ff4d4001670>: no matching AST found
+To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
+
+
+
+ +
+ + + +
+
<AxesSubplot:>
+
+ +
+ +
+ + + +
+ +
+ +
+ +
+
+ +
+ {% endraw %} + +
+
+

Plot results:

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
fig, ax = plt.subplots(1, constrained_layout=True)
+# federated results
+metrics_df.plot(ax=ax, y=["train_loss", "test_loss"], color=["royalblue", "red"])
+# centralized baseline
+ax.axhline(
+    history.history["val_loss"][-1],
+    linestyle="--",
+    color="black",
+    label="baseline (centralized computation)",
+)
+ax.set_ylabel("loss (sparse_categorical_crossentropy)")
+ax.spines["top"].set_visible(False)
+ax.spines["right"].set_visible(False)
+ax.legend()
+ax.set_ylim(ymin=0)
+
+ +
+
+
+ +
+
+ +
+ + + +
+
(0.0, 16.483655858039857)
+
+ +
+ +
+ + + +
+ +
+ +
+ +
+
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
fig, ax = plt.subplots(1, constrained_layout=True)
+# plot federated learning results
+metrics_df.plot(
+    ax=ax, y=["train_accuracy", "test_accuracy"], color=["royalblue", "red"]
+)
+# centralized baseline
+ax.axhline(
+    history.history["val_sparse_categorical_accuracy"][-1],
+    linestyle="--",
+    color="black",
+    label="baseline (centralized computation)",
+)
+# random guessing baseline
+ax.axhline(
+    1 / NUNIQUE_LABELS, linestyle="-.", color="grey", label="baseline (random quessing)"
+)
+ax.legend()
+ax.spines["top"].set_visible(False)
+ax.spines["right"].set_visible(False)
+ax.set_ylim(0, 1)
+plt.savefig(Path.cwd() / "results" / "accuracy.png")
+
+ +
+
+
+ +
+
+ +
+ + + +
+ +
+ +
+ +
+
+ +
+ {% endraw %} + +
+
+

With the current setup, we can reach test accuracy of 50% and slightly above (depending on the run). However, at times the model falls back to the level of random quessing.

+ +
+
+
+
+
+

Conclusions

+
+
+
+
+
+
    +
  • We can do federated learning of customer paths even with quite little data
    • +
  • It's doing significantly better than random guessing
    • +
  • However, cetralized model stil outperforms the federated computation. More work is required for optimization.
    • +
+ +
+
+
+
+ + diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..fac3a12 --- /dev/null +++ b/docs/index.html @@ -0,0 +1,223 @@ +--- + +title: Simulating Customer Journey Prediction in a Federated Learning Setup + + +keywords: fastai +sidebar: home_sidebar + + + +nb_path: "index.ipynb" +--- + + +
+ + {% raw %} + +
+ +
+ {% endraw %} + + {% raw %} + +
+
+ +
+
+
%load_ext lab_black
+# nb_black if running in jupyter
+
+ +
+
+
+ +
+ {% endraw %} + +
+
+

About

In this study I show that customer paths can be predicted in a federated setup. +The purpose of the work is to show, that this is possible using real customer journey data. +The purpose is not yet to provide optimal solutions. +The goal of the work was to observe machine learning in the given setup. +With these results I hope to encourage further research and innovation on federated learning for privacy preserving personalized digital services.

+

Federated learning (FL) is a term used for machine learning in de-centralized +setups where data is distibuted across edge devices [1]. +In a FL setup a shared model is learned by iteratively aggegating locally trained models. +A FL setup consists of a server updating a global model averaged from aggregate distributed model parameters, +and multiple clients (edge devices, for example mobile phones) that fit the global model to their data.

+

federated learning process

+

FL can be used for improved privacy, because instead of data, +only the aggregated model parameters are shared. The raw data remains in the edge devices. +The federated service provider does not rerquire a direct access to the data. +However, the model update information still contains indirect information of the client data, so some privacy risks still prevail [2].

+

Because each local dataset is hosted by an edge device and represents a single client, FL setups typically follow a few assumpions: +the data is strongly non-IID (it represents a single client), +unbalanced (the users use the service differently), +massively distributed (there are more clients than data points per client), +and finally limited communication (federated updates require the device to be at rest, plugged to charger and connected to Wi-Fi).

+

A customer journey describes the service touchpoints and transitions between them as a customer goes through a service process. +We can model these as states of a stochastic process, and use machine learning to predict the next state based on the previous ones, and other information about the customer. +Knowing the most likely next states would allow a service provider to tailor the service for the individual customers.

+

Typically, each customer may only have few data points of their own, e.g. few clicks on a web page since they have entered the service. +This means, that a localized machine learning model using the data of a single customer is not an option. +However, customers may be unwilling to share their data for centralized processing, as they are for example with website 'cookies' [3].

+

With federated learning, the customers could benefit from each other without directly sharing their data. +This makes customer journey prediction is an interesting field of application for federated learning, +especially in the public sector & government context.

+

For the public sector, this opens up new means for creating novel digital services. +Cities or governments can provide the citizens personal AI assistants that give them personalized recommendation and advice in various areas of life, +including education, healthcare and career. +These kind of services would require very sensitive data, that the citizens may be unwilling to trust the government with, and that local regulation may prohibit the officials from collecting. +Federated learning can help provide these services, without the need for data share.

+

The results may also be applied to other next state prediction problems not involving human customers, +but artificial clients insted. One such example could be predicting faults on IoT devices.

+

The simulation study

In this work I show how custom data can be used with TFF. I use the customer journey dataset by Bernard & Andritsos [4]. +I use 21k data points from 3000 customers (divided into 2100 train and 900 validation clients). There are 16 state labels (including an abstract state describing end of service), 2 customer background features (age and income) and the order of the action. +In the dataset, most customers only had 3-5 events recorded and the max observed was 10. +I create a simple federated learning setup for next state prediction on this dataset. +simulate the federated learning with TFF and compare the results against centralized computation baseline using identical NN model.

+

The code is presented in the wiki tabs CustomClientData and FederatedCustomerJourney and in the notebooks 00_data.ipynb and 01_model.ipynb that can be found from the root of the repository.

+

Results

accuracy plot +Train and validation accuracy in comparison to baselines. X-axis shows the federate update iterations, and y axis shows the accuracy. The federated updates are not directly comparable to progression of centralized computation, for which we only plot the result after 20 epochs.

+

We observe the federated learning achieves clearly better stats than random guessing, but does not perform as well as the centralized baseline. +Complete reproducibility proved difficult to achieve using TFF, so the results may vary depending on the run.

+

View the tabs / notebooks for further info.

+

It should be noted that the results were achieved in an optimistic scenario. For example the issues of changing client pool, client availablity, update scheduling and malicious behaviour were not considered in practice.

+ +
+
+
+
+
+

Contents

The repository core structure is the following:

+ +
data/   # A folder where data is loaded (empty, run 00_data.ipynb to load data)
+docs/   # HTML doc files are generated here
+ml_federated_customer_journey/ # python module, functions and classes from notebooks are exported here
+00_customclientdata.ipynb   # Load and clean data, define related functions
+01_federatedcustomerjourney.ipynb # federated simulation
+index.ipynb # this notebook, the repo README and wiki main page are generated from this
+settings.ini # project metadata for nbdev tool
+ +
+
+
+
+
+

How to Install and Run

To install and run the code:

+ +
git clone git@github.com:City-of-Helsinki/ml_federated_customer_path.git
+cd ml_federated_customer_path
+# (create and activate virtual environment of your choice)
+pip install requirements.txt
+nbdev_install_git_hooks
+
+# run 00_customclientdata.ipynb to load and clean data
+# run 01_federatedcustomerjourney.ipynb to run the federated learning simulation
+
+# to update ml_federated_customer_journey module and docs, call:
+nbdev_build_lib && nbdev_build_docs
+
+
+

Please note that the exact results are not reproducible due to inheritant reproducibility issues of TFF.

+

Feel free to try out different setups!

+ +
+
+
+
+
+

Contributing

Drop the authors message if you are interested in further research on the topic! (firstname.lastname(at)hel.fi)

+

See here on how to contribute to the code.

+ +
+
+
+
+
+

References

1) McMahan et al. Communication-Efficient Learning of Deep Networks from Decentralized Data. 2021. Google Inc. https://arxiv.org/pdf/1602.05629.pdf

+

2) Truong et al. Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective. 2021. https://arxiv.org/abs/2011.05411

+

3) Mueller, R. 76% of Users Ignore Cookie Banners: the User Behaviour After 30 Days of GDPR. 2018. Amazee Metrics (online journal). https://www.amazeemetrics.com/en/blog/76-ignore-cookie-banners-the-user-behavior-after-30-days-of-gdpr/

+

4) Bernard, G., & Andritsos, P. (2019). Contextual and behavioral customer journey discovery using a genetic approach. In 23rd European Conference on Advances in Databases and Information Systems (ADBIS), pages 251–266, Cham. Springer. +Dataset available at: https://customer-journey.me/datasets/

+

This project was built using nbdev on top of the city of Helsinki ml_project_template.

+ +
+
+
+
+
+

How to Cite

To cite this work, use:

+ +
+
+
+ {% raw %} + +
+
+ +
+
+
%%script False
+
+@misc{
+    sten2021simulating,
+    title = {Simulating Customer Journey Prediction in a Federated Learning Setup},
+    author = {Nuutti A Sten},
+    month = {12},
+    year = {2021},
+    howpublished = {City of Helsinki},
+    doi = {ADD DOI HERE},
+}
+
+ +
+
+
+ +
+
+ +
+ +
+
Couldn't find program: 'False'
+
+
+
+ +
+
+ +
+ {% endraw %} + +
+
+

{% include note.html content='Edit the year and author below according to your project!' %} +Copyright 2021 City-of-Helsinki. Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this project's files except in compliance with the License. +A copy of the License is provided in the LICENSE file in this repository.

+

The Helsinki logo is a registered trademark owned by the city of Helsinki.

+ +
+
+
+
+ + diff --git a/docs/results/accuracy.png b/docs/results/accuracy.png new file mode 100644 index 0000000000000000000000000000000000000000..6f78f82ff52caece31f65d63617edc690d12d870 GIT binary patch literal 35692 zcmd3Ng?s@bR!@j-QC?H-QC>`&7SeM`|i8D z|G+*UQ66UIaPD(o*SD^j;ExJYm`{nH!ok5|O23y-f`fxE1z#_ppn#uDyrwk+|L{A$ z({NO_F?MwMWN!o~|H;wT(#Fx!%;2T7k-dYNjWs6=7Yhf|OH)TjTL%GFR;&Md0*j5k z2`i)jss>mD&Gx;f0~{PW9_$MqE-B>&99+nZw1k+7Ysx{YONx%__2ZI>aWz!0oIY8m zJlXPn?l;^g%;#9RI3JM|6`wc%{F(XlC)HDihZtJn=XnTz0UvSSzocw3iC);Jomm`+ zIlEn(7Zmg|xmDmhnqn*&ny+)%8cY?0I;7dBNxi2ZG36o-ryVWDxPSxSNM`L|YPG_? z!TWKAmVmRmBp&ouJSf~x@8xrBbV}!^l#}R`#u#^O+_2}K(0)h0L@H(Lv`|2$eeq}w z_b~a`G5T*oE^h?dY#sRVudYM%vrZ=(6Ql57~98&XeC?Pura zzPZ09mY2XKA(0gmgI82k6oZZ+0o9()miv(WM;Yozvvl9ICWTQ@iU zz`#H;=vWmFcM|;1rl!=eFbrlrVtG~^@$Wx=j1-~1MNg3Ad}DetYU=)huZ4mc0^3jU z`$}Iku5XkV7Z*Y|+gZZi3q&Rnllifw@B!kkTuyt(-0hDk+IeOQWTGT@t~45WaXmmLs3_>sLTQas5-@u28~?&waR1c04>hc#URH zx9PRVrN@V!Fe1*^V5Rlw58WH~?4R0^^7HfIZ0mHwC`1?UHSlJ(x!l8zeXsXqo%R`y z_NNRZ7=t!9Lr1@e_&nZ2E9}-JCyTUuPB#Yl&jvYXYVEQFX|BPtygNHPl~q;j9+#FZ zwaW+`92|`iWrjUiXC57)y!_9tD0XQu4TtriKKFb2 zo{zUnTXEXfD*<>4%Myb9YH2|r>Cr;(R;o#2A{z1K%1SNEf(1Vt~EmNkS|9e zwV!iX$bZIdwF6spy_>`asnxtHDk`e|8-;i8J3?IvV`Ex4aQ*ij52^)fc6TQ|4N%TV zkAteI({bew@G@7$b&~2Wf;xW*K&`a~l*gci1_U5MALCi||BPnKYS!7)oSd8tri+R$ zyC)|kynsFysad=yTwU!7bpwmQ?p82e8sz$PLzp~k$hc#}9=?BKYG%f$>eBd*mRc&} z6&>9-Y|*>#tj$+nOw!zPdJM-fK2##-jF5f>Z80)F{`|P^>sOS~(b1sDNRz$EqC%~z z{Nbai@o@<~J#y!U1=*JhBG;cEE^AugmmLMnOih=6MJudk$LnS0e<)@^5omTDDwJv1h*_z|#$Z?e6j zruH|C;QzY1YGB=Pz*DkCz@ih>u>v_=@0<6&Js3>pEg3$7?&&`!0bH~nWbd)$2Fv`V?#nQ=%+NbT7lL4HsnuoIcown1LL*(=$~iWJENw=$Md-Ca|6 zB}|l&F|>!=BMeqyQBk-ob?dlL3j$^>2?vJ?X9{xTC#pK$%Zc0;aXJ;oLq8kiB{|0} zXUYRX<6&R0aiLgD7OqH%-aMo5xjXrY{ca5L%U#4gjesMDT-FHbM3E$#yeLZ-`)(msIMu{zK}k z`xD~uf0YP8p9+Rurd&xu5&(k$)KcO8Lyq@0b9rz==YQWG8Z;6Q*-C;39bVYiK1r}n z&qNFScc`5esr^gWujvo^MZf|Yys z4w){^l?r?r{9RjeT?-#f;ttKsq=KHnsg>vrjYv_*0}OH49L&heL(3iO``~?w7ZMr@ zTh-@!K;O4`lKCeA@o>5*g|Ek9-8TR(A+S3N`*(Cy3DzOTg9&W_R_2*C;k=;@OYBa& zDl#%M^XYEgVMK>pF8k9f7a9((8DrbrNq(E(L<5}mZlDQcxgBq@k-cWYwRjvg^+eIg zUaf|*epf0t9idVy)DTlu#lOD3ZoHV(_RVKc0NqCMcqJ9bti4%GKoPWbx5@h}H+RiZ zfd6_uTC>GlSc=pRDPG4OjkWO@0RV1JS=o=;T0U}*eRdgHSDK z8;26bv5hP&{eJ|&f7ApehkZ$ryS;)N8XDT4F7scD5R3r(?77=F5nU^+__GRdAN@7~ zG$vJ~@Z}4vZ~p!LGCVemVK`qJ8->rBKmL=*?8QJ?g4ILQ1$sW7=ZgM_j_v?Z8$Pf>#lyq1ZpR7$-Wybg z{irk@yThhji)3n22Pzrjm?_p(z7CYXv3Eryf=>iAhML-@kuJLlX%2?DUV|5geP@=qm+ZZpooagKtjD`aE&5>VVE>ZeJ?Ma z!3zZ>!-=~p&BkE(G-HyEynNVjw23Nu1rMO+=}gw1$I>nN2a#)l$uyP=^%GX9%sRE- z--QyeTTC+Ld`SP}k1ew8J%q2WOiawX zo2eFZ7_|6W=Vp$WHY=vMEV}K_R)qo*u>JillL!K<(4?P0XU0cFV3k;X5WVM8&Q&1G zeC9L^&R$^r?OzOBn9!8}DDrrBmO=hS67P9xXU6{nNB<>;{uf@6evOHXQ_|59g7oM& zd-9v4IuZOM1?ZBGPYbEKOrJDes)V7TVGAkt&7@X$g~0>BmxZk|p_5L$dC#*UFOffQ zz_%YhDk_ew-NVidbi;fB^bSx!Kj^z7y(Tpn9PZ$S5-_h`&eowx14Oqg&b_rCg+BbV3?ZdmR4s4V;h=l-lkiP4`Feo}zdchHu{3lEA9 z1qLUpUGw$>%%ooDBVlBMaWt}VwNUi&=)rhEwO<|n@b}zb0j*{q|FeEKzX0xce*P*@ zX^_kPnZY#SWN`A&KYDc*H(hFN4yO-Kml?WUt$c%jTr?esXWjars_-!Rux=}DU@(!B zo`GSttcM((ue!5S0?zv+RG$rS3#>MRo>9+V5ImlxKi0}Rcx?$T*-cq`0;!S6XGaC< z*Di7*esj~fxNegerdPl<%~w&T1Mvm^PsYxUUG#QlxUt)Q&Z8GCMnqpvUNU znGBw7oSpsAuo-YNP^O<28{J5STycD_u3Q1?k2Ms-EA@+A^8$Ft8}U@0Zgi`dTEkP| zYdcz^jufKGjw#R1rdd7%oLL*A)SsWv(6Uu;2*d>L_jJ$GVDKXgAeuyuTM?CH=q=36 z=Wli0HeydNru5^LWD?`!>E`Clx_Mj=-jI2`E+5?%!v`i+L8W=3!f?mPB%zs7cAU;^rH zJ1g?8TiAU3_0072>+Bf&weE2H_5if<{QQ24c|aA3cUNnXU;~E>4eI1}ozK87kD70- z-2k@q0W8o|Rizml8#6L7$(^q-iDY76uy5%pHyZft>Df5nQD z4FK!R$OvVm$c-tq?sPp@^gi#d4cmL!TSr@)5qh+AqkZ>=>79Yq0V|iKvPL+^+C_x!NJJMsp{6DNUv%3$WsWM{wYAPIHmkOM?eqW zTkm9!EV^7o&+J?bdGviiE%oU?-b@`%Edc^Q1tfAV$lBaOjaXBRRM{GUgC2-87l0!e zh~wQ1PucmJvXg_dQ;V|GowAciI9?%KlJk2QDgUUhDHO&EfRo`NK=8pJD^EH16bSTk zvoTsYU;w%SAh!8DoKd(O6$392d3)57zB87~>GR+TEa6Ozjeajw7&`AzrojZDpi6k* za$lOqup6V=dZGWHvIj0oR!IpLj>7wjl2F(2M?+c8%#p>;Xf_Oxw%wzg-pxSLO+!3fsL`>E2`lD^(}I> zjM#j4Dht?r2pEX#Wq<7HCJ$bifH>P4rUQC(d)pM~&?w&?ZQx0G7aXH+?X_BUk8gxBbsGw6u0b{(M13@9ph<2~-QvwM$-@ z0kWkQG%psyXTp-f4#zA~z;-{?v3Q*nbW#SCB1FQW``SaQa8{k@vrPgHS(e|?Z!87i zDErpJJlMmeZy1^fZ8rIihLhPX*N2KEVYA2iraB6CX+wWzr?m8Aq~( z(zrSTqf%x*k?-r(CbTqXUWF2F=XOb=4Oj!n9ixr@*v_7whCqSm;MQRyBO_ZwGh(6u zIVIg5HVj*|OzC+ruiLrHs;I==-d1Ta9S&QSX*7yB{IqDTz$UqZ8l26O?`!SsC3v+Y zgN}1S981I5W=KTb2Td8L<(vG5|n! zuNmIR*ciCX{RqW69hUz7em=lV*9%A9Qg7cP0zvmr$ZW9s*rAWiR#|v($3cYqJQKOh za2s5XQ2x6^+TMja?dIjWBi~OIX%n6{HSF0;{{8y~++(010n^6rIF$1$;bU0t)QP=jM!Rm{KfcrRXJX28MTR{Q>s}|H3{Klbn zg)t@wFbw-+=2C+ILHZ-Ook z04j#DpGGMJJ~lRXePhE89F{G>WjkOlk*vxbrgU8t)zzf{C&ILwc6AsK;)gAd-U>n| z5@0jH{^zKFk>!IOI66zMui03RT^lkX8<1$3nVDBztln)Y{0^|?f#V49cmy%a87P@} zfJh){!8_JEStuzg8ft2)F?={5`=D705sMTdLE1*84XDM@c%9w&$JG;`ONHpdq;(juxAQ$$v`X1#f#pnzG8vqinqPv>P8EZr=*W z1Z~0606-6S7bQNxVFr9Yn60`yiS$|D$xXlXC~okE`2Fv^?n5(X0!yA#H;@mf||m^OogLhY^!UvRIuY8zBP>9y*-bMSxge1Qs7m*$%a4Tu%W_Fk$R%+F!w&g5A)zvegu;0jdjSf*W?(oK=mz;*EkLD2c0&{Z&PKoq!huG$1uYND z@PMECW>S6*v!}o|)3dN_W_%;JZ`pb#McM`;9GE~;*Rex}dAtO}h)+O9z-xPhtjrE{ zIR|Jk;CqBN|D^6&GaFPx;5EuW2hZ)IP2z5XV8gz~oF5>7nYno;sDZUe(KJwI%D~$L zj;5(ywv7@L>#KMYg zFHmn{-XNS*l9{+%2mB0l#5KT(UK^j=IiOBp;(b8>;S>lo`X$dZd9+@j>gtS#h&6R| zn39u|jg*z6Q#g!5?sf}G*3*PNeodIi?oE}%12Og|f&HzX!w~oAhJ7Evefcz?s=W;# zIw&4qmHs2X(8v60! zSB*0K3t{(4bt{-a{z1fPYzuN5m=r1Dx%_vG3;Bj!A^18#j{4?TB*ev`w9=(}w!3Ym zN{Ndj1P2GF_&i)&59EMM60o;HnFc7KRUmI9pR7#fcXpl$!xRm;IpS#)PZn{3os57* zSsSoGvbmW&d8iu%I&MJM14G~eys7`xH9I{M(`R5$_0n6Ko6~Y~Xd?u6U&FBgD_k;l zULDwHE8f$}l_d5v7v5p`er;{d1K(DZbT-emS{Q&dHq0^tr9sQg91ft$qYBEl#H&IQ zYkN%U^IMH(W%-)SHG*41HUY%gsUYQy2I*BYEZzZuQar5S@EO&P!3IGvU|`*Jq1Jp+ zWP?mrF=2keP)ZWESj|SHV2TbL_P;i2=HlUr`u_b5NJ~($$=5-YvUBML1h{LO*QF*L z$g9Z!nZWTK)~>k%w`OW(1^O-!q`9#`z!$5h51k3~SkL#~T+DE8VwNWun?9s z-{gCEl*X|&I3F17j_1Lm6+_^zV7%Jo0gVFT1h56epbLAzQuvd#Ao!P-ky!;o3H3PC z<>KE70hAj06f&c*0dshOU|{J0Onzv8Uf&T20nEV=(zj&d+5@z zrb+5MH8lbd8rY*>fdp?*o_C=Wl#DGODJh60VdS_4AjSa5WozgL0R2DUT8BXx_(nzL zIjBG(fC8T)4}doRmyi>L27=c0i-~y-br^_aUK`{b-apwQQ@FxEBYW-0u0OA%N^gGLn1jHvFpgM{l4Z##EDQsQZ2!?UEPhP!x}33`s}BE9^0x%4F2%l|zEHa+g5u%$oA>Y| zNcuN`$K3?vu=eYUe1Hgm-0NIX&-3J6%N7`NV70<(D%qxa=R!wE zhxDlN3=6L1;i3fQlp529dwZ8R;`LFML6Qssu0U^?@cHv&&}XQ=u+4%^|Jl0^#-lq0 zc{)5uZg@+*ZkP(8v={a=4g#wOH#}=NOn_F)f)M+0~ zj-oCDC6)_R+1_=~p zRPmrW|9yut9D@oRgJ_)9nK!pE0VB!z@k$dS{)DlVOam?k5y+!oG0DQ0iExciQeI^j z;)#EjrN<6*&(f?g?gP2WIrIuPb3GfP`i0DcI!&KF&z!xFl!p4}WM<2K`5CXrEGw;ENXm9Yy*Z{$HcuJ<+DnegbwAqCx!g2qejo9^*`SYaNIc%1?gDTGYc@nlU<64l#au&Mn z_)AvDIAefm3-J&en0**eQ)RE0sqeX+t!-wIhe&1-eFXr3p{+GXqd=&lA_$NOy!ZuZ zQ%*}WOFQ_Fd9a2>dINjq=y*O-Vg90Yu9t1O5jU~?`p{t-aHHdLBj*7eT5G4PIw1|sld?bxKJSpghoHoiZvCjZB*V#auM z;-sor3TfCe%j&S?F_j{r^Y8F5i4$Jj(?aTl}%%gtc@l%1JcG4$l zhh%hmO8PeO82Rt9#uEsan5~Xvc^H-Wps?BvI#@3%X zqctw|^G+|Q{RkIr)?GeKfct)ftlWw+fWWyh+GQB)yCc`xD-LdvhC$p9nF#fGQ}4L& zezE$z`@=jf@AKTZ{-=8PZ6+kM8ka_UpMLm4wBDeaqe{IWw8-zpDNPcSBOf^@`?tF+ z#a>?i5u8#C5~pyM*;!PcUOmMFNY|{|C0VJVi^SKLDk}Wjm9kLxoKM3(kLW9ew_R3TkR6HGxE}BIMW&3v;#*DG+NKV0< zbyHt3Q+`)gH@I$97`sT$K0JA?zmYhtq(wxDpr(x;8=F6pjOrw>@GIbiI?GP_cm7^t zg3z@ZBDTm_g$Bf^s>935=$D2Q22(q%Y;5!Q$Sjwph_0mRKb;x3423f*1o%qSAQK>C zSTfu{s#X6TWw}1Q&f9IO^u@$*a{4=y_uvl6`TK!B48`3V-lw`bHnQ3BK(Vi^pZJ?c zhNommbt@GnxcX*lniM+}p?gQ)+*i=Mill}IJ^ZUG3LlJj?~dD0^^!@nbzXIAmTDpW z)B(AVK`Q}ggHDusJTH4TTT)g|_cIb|s?5`GZX&di!4ahJ1x2bpj0}yuB$LB2;;|n` zqFLsZWG>bDP0iv-pN`pE962gZZBy~SMboJNVhuU!WU_>G%_(1R5RZ=kXm)L!s-}$NWlyh2E_NSriIWiKUzmp_bTO8^ z(N6ZL&$j>i2=o{mpx}v$HQ%VEUW4x)u6!ql!NfY&opAShA8T54 z{#{L;bnpuRs&yz#f1Jzh!n17lQum#& z?vExOq5FzH3gVYZ&sR6{6ZK|uQhqP*YjSiKqhha@pEo*?sS8~ z6{1#|C8W^oIV^=k{9-AT(gpib8W<5ewbQ)_Sn8itTRYjDR<&|OOiCn2gTFxoQHWm;=G`SqCF}~Mql2eJVq(U$w;z}9{#n+wqt|H}6>rwDvhU`cL^v#p zot@3*Ua8%xr#C4f7qdKQh`Z3RVL~68^WliQl}|ScQ4+WPx6a$vuO6g>%B`Nvr6{>` zPy6L-F5kCaQh<1~weqCQ@1184)fePBO5lbz zjjCT+di2xZ;*ShvM=kp}H*zhziuzk;9xyO&yddD>f-|SANJlm>9 z2|1TTC(`|kDb@YG2tmlc2Ao=BeWF6c`W;?0Li3cZLN;%YafFkco=PMW+c#zVomBkB z@PAu7SJR5p`^M#2d z@Y)NEzXO zXc>DyGHtpk7wmMDlxB&^0!d7KiKOF9?_vo3yldL-G-K3gZou=sAT)ENnT~} z=mzEZ61C5avh0+~vZ7{StZiM3O$ zC(of&<@*jcm9>7j))H2KERTw-W_!245x=sXMJcKujC0(FYBbKpcU}F-EAbbN_Clam z)9+&EP>OhcEEd8z8dmrR1Qbcto5d&7UQjVTaviTK_mpJWT;@2mhj|Emcd@F;i=4qpn-d>MKw+hR&oYUi$~jc#XWjxaL08R3=`O~Q#4o0=0q zlcI8C-yy@&Tm1~r>XSs=&|Iu}J$9RwUt(aS183vHI8k4Hipr}DE=UF%qfk>yZ?7x#$9FiS-`r22UZ(j6f``Vsl;7c=; zC}YSambC6qZDq(Ob#x?NN;dN&^C)x|rC;c|kXwZ_f_I*! z8M5kFbIC&sCny#9YZhNw2`$DRt0yCyzpi0Wv$EXt-V%c>i3z4U_pNd&M#{Wy2j`c=b3oklc%Q@{Aowk{|(})?gmOTN%xRwIG_FM^0Vg96kNr3IR zP=+E0zuGRVU!-w^K)?{!b#0B70F?HJW@~EiQw^qUk1x&RsWPG%YY-q`lIDJ;BeM&z zG8C$~b3Sl8d16T6_ux`6qV$AosBbYtzP?jgt8T4tKAXqT){rK>E!bZ-a<5lX<-YYo z4WBw(EGj3bQ&1uB@e&3_F&YIR$yMKw$eme30@Nn`sI(xWN`0iE{IgUwwWe}&dn85ss`PXJgwEyJ{oPKT0PY&oyq7MmR0^MT7sN9^ zGdZkxH$wx9s+sIO#Mf#u7k+_M|3+_{gv@`}El;?Y?6F8c-3V`kMT_e{_~%q1GYcupKC{+RjAN&d z;Nj9n!k@xMO+HX-XWk?1S5!KWfv-$VkQk)z zJrgS}rH;`9td~ryPI2+5OxtvALo~gi;rvO9vN7G?Sr*N@vL>INgU}suIhij3NHyqX zIc?Hfuw?Dzb5Jq|w4ChSNe64)3(_ z`FGS-B|l%|pL4VOpRGKZp3O5^d#+Y8szhs7Mlc%1q~hq%TodD5dYZ4l(%RiTV6#c# zvVMAdWWs5vWH}{2`aT<*7NW=_=3MWz&G~F@ZY9BRdC_Ab*>mqIF=1~;T=lb!rCoW= z=$~;r_T%lWQYS_lJj34iWv6X#8lK4$QoRoCm0LA+tSh3xb6`#ojBT#dO#LSMFw*ov z-ai_~Dsk|IYHh#gUtg-ssvZn@-8M3@Z9P22nh-#Is_~(~E@9_}wV6V4Rax|8B}t7~ zTgT5Qd12X6#yJMlcWSHocA3@3?_losahmck!X)bM?P~}RqvLe8OZPa=%k0l4nsj(i zGZ*F*1R4D7kQIe4wVG-K%^$t5_Bnw_z?u}en$a%+m3`chTv29hbOk#mQzh?`JZbIecD#XZjI#QKLbmd681msO`g8mFD$%a6I8V)cU<5E-DWDr{}z-W{B^u>yW3~|X1Zs40!~wfM_tku zURM4o#7|ZIt$jHI(!Q8sL8!a~>&LR{SjPK^*rPeMu9c7JJH-urR0zyvt-8 zaR*MiVTIO}0DKG7*JZt7+&S5KEpBdlnn7tqL2&%t2nYT-$j06u$cv2}>=nNscwyp4 z1Z{pyjjooLe70n_7Bj%XpIEJ=`j_|bHKI{qpoQOo@bYr)jvbJedM}U|3oA$efW8|{ zc>dNS*O6{5U?6u-`$u_ts7T9YQh7v%MPK{N6*^etcKpOvfDO4;?g=|htG9G3w(l^i z`}dC68(MtVh)8_lehEuU#UW^p{738LArBPk*4a5QTUnWExtz7Ef9_cnLnPH{KR(Wv zcCG(5zTtc$-j0ktQ83O(Npo#^@yM9nIpCyzaPTKfCpZPA#;&EnZ9^|V0nyt?5j+D! z#QJAv6_$=_r0pkH(fLno)#7A+G#`lsg&J;TyPP(KF8wmxQ+UBid(KbKgux8atO-5#ZKqg5N`;vE={aVp_;N|j)vlE-03fD;TWC5m?)p0k8+zX0UXtH0D`~9BCn`X;ZlXd=##gh! z@+lvg-!!<#aweOc?jyA)#ax*{VzKB!pK~jm<^x zX~kU&>qk4b5~q*k;3wKp=Z``xlA4axlD~b1MI&*(L~p$+^)``;pBPI|aKaJR4Ps{I zQW;0oJ)!$xKPr6lC6QN(UKLY8<=Ijr28mjF*S4{0R5i08r9eF+B0DR#tP!$WXubFL7b)lomjuzg3F#mCIpx-PqQliAngBppdbD@)!}mS#O> zI=IFJj_G-Y0nkN%#u62^!;jjK)+enFP9}9_L*Y0EC~RP;ItdqHHLbj~|BAa$MlYvm z-~$Cp&Ml+pSa(eO3kHi6zdfW&-o0=l7mo&Y-c>TA)Ud&DDxTR#`xmizU72fmZhO(j z`RN*8Z0lTk?BGRfxx&}N54)^6IATHzXpS-TVxv!k{oIP-5%;mAwZEwt{bFe$&Cge< ziX6#OUF}VE!8NoyUEilDJL&qMDtEZ_sD|u5HVO-@W(8gmrp1shf(lo;Z~GEdOr%Y; zo#^zQ$2uTdL5X)|Saj=6I3oSM^0s{U`a8E0|5FI?a?YqSt&>ImM4Zd3p({R*{x|Ll zX{m6ns0FC4PdSkX#{!>l9e!kKs{H)8$I9E$?WYy_LL*ACX0%iWN@0IxWbj1>E;tgu zSGQrp#gaWDHT{F!1-v~%P@1%4d2EuTNsZa=Ee1SEtOd1jSAU)Z-qlk6Cj2dP$;_+* z;6UZA{3FxJ7!BvxxejNS;@Q8F_Fn=)w&rf+M^@-`^-U(tLqR^Iq+Uj|uA>zJJs77Q z;qKX$-HY4BmjN2o0UFAxsuRg4m=>_Qr`7D`tnc~Sz>+np`%ND01%{Hq{mn%`Ga(-0 zQS;tnW5xWRfG@2TZoW3DKg2x?f8LF3(SIQ7uv<#%nd!MEY;_7%#NjnavJulC76?KK zXO>xx!ez3(*~uT-7|mw0AOLTH{irCmCsp9;K8Zb@C|tGI9gvjKTwWM#wq}^X$AGbi zK0Z{`M21EB2WN2u&TiA(VFcm%BXR!)}sWqG=<{g}01E&p}0qwgwg;-J! zuX_6qXXkc{5~ApQQxT1YGmIMRld4vXg5On+qQ3eBRKF!6-d16$mKj5>6GZiOTS$1F zRWpV!Tbu)N!ump~tRKmZh!WPfqCfh4Z#-|EB>b&%100Hl$M%wgS$697>lNpBdX6$2 zSagl`o{+*)s9M$c6v)c&JO*bGkC2-NV_B%RM6T@Vbq>$w^DYhn-h6d}>7%s#l}%J^ zSGsGV`$`5=@R<)?giT^07mt_lbJ$_$t~wA)PHBV?zYXyqB?$Sk`%eih`VB*+eogQP z{CCg0+tz8VZq`8dfS|mz{%WYSbyl#0m47`^j0b=!52HUGs&(~Y?vzQpVyS`J>UEJnF^)w0aDxlnn4Mn z+JSDO0Slom8s}+wR;eZ;>MrlWE<_?O{M<`>Jbbg4T9q-G-EcXO5RT7C0l|Y|AAH7Y z%h6)#MZzaPr?|X?7#j=>6%*rcc;>{UyMjq%llp^5O;6s1K zSfChn`VWQRAFok#%-MKY#VILGt-s!LmzEgT1*qrNV~e&Y{=nG&xV7; zpZ%gicK#_bgM>WuFvOf4V>y5D>=D7p4=tmxG+Q>KKIxiswM?Vlh5E7Rv~GJ9o8L=fl}>L$X!Z1;rBzkm4G`>O@mc9*dL(yI zCGhjdacj>u6y?>Mza=3EEuVc_ZuNeq0{NyYoFz$1cim{#^)XaU&B|1HGJa^2<#AIX zhNzP-VTl#IZvoC4Wf8d=j#Y-97md=JCP0JBl$wMkp_nEe*}dohw*k=w1LK%g=;nM( zSHBn)UqlryCoKGv#|Hwg5^dCBA-qc|xaDOrqdvK06m-ITtE1+DKlV=1Boys|Patk9 zZZAa1_x5oZY3uDMlP36flXP?sU!}j$P?lXbj%&Wk8MR?PfB+}^Cq4ZISM@U-PT8_# zi3q>y)=5h7!xL+>BL!FJI2#Vb4TSYb{1`HeLp6qaPLhN|L7?ydOtu(?wt{*9>;ys zmBVwXK4XVcA2-CIlrzUmgS@(mgNg+J$WEIeEdkj*nTLqCt6iK1vavxron6ei#0{av z+h>T;|H0AUR7A@{;)~Fe@pJ2sm0uuEX*Cuhs(Fex0#6;=R{*&hr(()~mmG3+Pj7QA z+_tX;$laS+3gy~e)T=&%*eKw~;t{WfJt>fB&?CMVjJ(zCRyr8LCj^q4zYWh{VHq1& zE!s;H2VN`Q?TD(@KAeTwpCvDHkEsd<*_HabpwVy4DfI^{_1h`+FTI%arp)#fZf!op z)&?jaP66q`Dj5PZw8~eZ4Y}X-Q`TeEW3;>1zFNDy&)-eagTKmbHx=0D@Qfewg0^u_ z{U$Q21*JZgLr*vu^z69hv8nK(53S7*K}9X!`Bl7bj`82N?QR6lJ<3TQKEx@p z&JG0hkG)gTjdCRTzCZW(BAZpOrW<5i^U_5NJ zqn1mgEWVFxA1903Xe)e6=pi{5$h(htbnbP}uqXIuN-O-U zoe3!ltydn&g*yX*{f-=-R2th}Pr-{yPOcM7cFPS(wGO(fn@7banBvo&XrmizS?PP{ z%{$|;`47siPggcXOK06P*4>tQH`%up>7q-K;V*8nAzSXMIIU^a;>0n1nu<~Dj@<9w zxLghGM|PVr%7DtsD)$`b5KCEQ;Uyucub6U8YG((0@=GjINKxpM(yy|A2$t^bxy-DU zU9r|&O0RUOpDrtj4$x==_G+A{O!bW>5Y2n_yDRmh=F?V!Q~sgp>fxw9rk6!Dwa9TebV0$ml*ogX z?HuNR@1-3gli$HJ;In4Ob?o#U(W0vc6ZiRr zs0127&8#D#oAnWlDjU=mpI|0sMb8L4Jf9ay+%}~#cndzoDFo{uM!9ukjV42gLazDE zlhT}lH<&6NEP@M?#^wp=Npo`z0d^Dl$D%zz8SZi*uT_ry%HANRxD((Y4*xTTya?@trOr700}Nf|yHZ|o>up&e^xjhsXo zQ8w#NtL#)hUE34-zV~PayhIf{H}0=lU9?HzhLlaB$47btE}JMLcMkvTuSm|<7WikD zu?M9JPGH>kQa9;4?qoBR8TR3>8Za@X#J{m<2t{_lbKRb_Ol#mxKT1pQ21$!IuNEbe znY3C?MC8VK7H%S!K+-~UYKA;vCf)V*D=yM=^Q=`_=lbx3a;4a+7bNJ^QsPRuXbn6b z_$L>JQ!K>M6*GEsPG0d# zbg2KIB5ZzK78BTy61{(&9BoX`PAo{3bd(a`ogaSCc=^L;esCwM5$cBSkkfgi%D|q; zO%oB`AYf|clE=3xMd3vA__Oflj){cYp}t#yqh~Tpu5(mIFzYN#T2>YvPpzdbDbM6= zIZEva>(D<%rUbcAP!7JJaGxPwT%G1-T*B$;8Hd&41rGGkDgOS`g*kK+@=hc7x#TxG z$fm*P?K)riD zIZh5&`aq6Q?Xxe(z3!#|RcJFvPW8uvTy}|#y^?ot#{9Q5G}1J=#LUG#w~1%Bak8Us z7f|H8?Q+MQ@JP(rIsu$ONp74Hjf4{PXW#4Ue({iFnEUx@Sx33r+NekQ;`FW>a17L- zg!{mkdhjy48NM#6Y%p{~%_rl^S%3M3DVeSBec9K_RvGMRq8~=+FKQ6^%+wGx^VHIO zN`3%Bu;0p31s2xzJhx7ceC@!9-^QEDA=KddmeFZ{@re1xqNYk&_h^0Gtxb2Wg05%P zv27M)83X$H;rRH=_9Kfss|8kMgNzYpH}c_wSU${PHZvNc;F4jg=Y>Bq+MY01oKe5n zXIL`nav;p2pvSgg&d5>_es`_^QTGF6)ac=t8=<})k^o!z3joC-Rq-11zMQ`MI z4tF7r#o;j$jGzX~-DQ*OPm+m(-n~2Yg8mB2u#lv)HV|V>)wK$KRIrwxqNPc9;`$aR&5 zN5)pK&LSN^8bqi=i6nZjh63b=_nB+By8CTIf!6zw8r$>nW$OoUUSKqi)p*`~a~Kjj zVbh%RwQ5eKZ7IuBPyc(C%r9;(mzF2>)g$j5lJ{ja+-}TXaDHVcOzlOcqdDoO8}h6g zx_ao2Q0+n$Dn6Sp6kTd6m7*pVW$3vPAV92{TjaoP;o9puArNKY0ggg{iZVEponmiPBWgN--gfJ(FI ziRZpe$M9Di(f-I9&N~;q@-s(W^&6m>sci7iwA;N+*Cv&cO_p&nS(A*dWzS~}EzX?m zy4*K=Rj4oT;%ZD~J-;9pgIU$;aiB#LG6Mf+h9>@Z3v1K(0CwJMu3f{V14-Mu7-W#h zS;`E-zpUYO&G5f_MMpG*YD&W1-BeT$;h;Zo3M)gYa~2fLC=5#oHe6T(ulUYi4iiv= zILKat2j=ijAK{~MMII8)$XgO?9=Q`eG8fm=_BihaP+gz>++Bma%b9%tsXBM3ikq!vr)H*S-tK-Mt3O{= zUz6%7Q^L>Ij$+^Cc}FKKlhITlJM-Lcw6R#H!Z6TyzV2@|y~s%wAVP#o>ByGQ)W5W zuh&^T3WP-RiLB9)q=$25L_aACwe%Z{QUN6Mv%~jxs;|FX--<~QP;hxJKbP>DVKIy|?y~p3U-1cm4#Y_|pE62zu|`d1-s1 zrRQz@L`!MKuw5OolyTmJ;JqfqH9`i;r4vN27Hv}n{hI$iPj+CqE>3A&%@*`Tl_#L- zG>8CmbJV1=w4Xo4hD9AMWI{PxQ&6O7X_>6Al}-oXip{!r1%I5+Cg6iu$jxS838ep6{45mP?*p6gkAMPEz+cI3?5kh3(p`!!#0EkHf# z+$?D#sn-5Cig$ajJ5CEFzGv|WqQF=Y6Hm+89cd=PY#0X?c(Ks%*cYCN$J#EbYF~W+ zTuSmMl}C!Pq#Q$6Vp;c3drKSQs2NWTb)2fL)Gxg1eG49P_%M~)UtA$Kz3z2 zMSXq~5-VL{(kK9y1nW%ic-Pw{VoDim92i(j2QWea9Ix{+Ti}Mld7Gd#)5N= zCbV*l-n__u-#Kqmq_+k~t80rUPcn5SIf z#52>Wo?kpu-yPbv=z4mgZJ|Bk_&>n2LUK58_-gCaP)XQfFB_p;$`$kFz!dM>_7IPQ zJs<%3X#?m8`l@XW%+J!cyMu0s1mS2qK~3Ecb1rx|{Pb!#YM@Jzo5jnoVj*Nd$hCNpOpV`v>rJA&fF zvde)F2co0NMdgw{7-GnAxl=uW>u6U}3l5avMLI`hO1`TbVh#zEPP{aQ4c@U8Bj+Y6eHG7ft@KhtX-)_Uic?z)H9 zB<@T^9&3ycYqETMb^4ck3EG0S=NJ%@4sWS~Ymaa=52CYsS`l4+m^Q{L zu}7lP6NK8Bv#4@QT+Y*F`B&QQ3p&0POIE|PVbzSpYlAmw=mOQ`O-U5#m5Oy5LK@IE zz3NVw!G?vCw;wTfwXhF*FjsIwW7_x`i3zrN_%k#}`%@N=^&UyyHMLsUIDY>qq*xJl zKnLRCosx7(bZE+CCqATv2%VU)OU9`mf*zS5)VZpY;%&^uh;AZuQpK3 zQ^cQ2%7%^DtDKYsjcPTVW!c@1l{shI1^Qyc&DLPs9?deiCcwYI=UV~zDJvLN9eoq- zx&w8uACQ(hx|B}<4V0?m2_WoNj3(ldxi0m-S@^~#tJpVvYBCha?&&^jlxt*DTA`#D z&Uanqhxp;cw1FR)byL$;^*jZC+HS^<95ET$yB=o~h1;Lu0O^yM@#L;eOTd49S=aE(9%WZpvb08vtl z(SUc;d7QCn%X1GnLs67Gaw-J;YDsq@fUeteKXGhi#`AaIm2R^yW=P(1O5Vz--jTZD zjRI2{*~wk<-~cHJZARZ4=qD(Xz~UK)y?zsZ$_y1@l677W8&OhTEvZ(UI^tqNTf}Ew zr-;&w7FX+W;j(Vc9=K%{%UkwsYo(?ATp><=7wyMP!Rh?CZIO;vAze0UiKL7l=cUE= zB8KxFIxA8k*&M=XMId#1J9_RJnQ5s!UQHAb%Sv7Pbk3(U|JS14{r1ZNmX(2vDZgyH z$PM6CT3B~e$v|IT zWXK3sYq}oE9ziPq8ZSUo3Rr+O)8x~6tyx^kfkxzRI9RK-tSVZaX9!?UCRI9t9dM8t zRi#1IGQ8O_XI?XNmVazwys(glY$BfD%ds3l2LpJBgKfcy$oxac(PUU5t|o!l;&C@j zd>K~io_$12SOxSh8VXDck>A)@t{eP>gZnrRDKJn%LJl4=;t9Y1u%Ub}snk>lXV>|` zV!STfW>Xi#vA&a4y@wxrMO_2L6nfy45TaBglp2w4s`S5+(Sac9x znwG`kY)GvCPhFSqrDGGw%p)V>%N=Xv({N?BE|cl0I3Ls8ZtYT?`kJj{)MxmU@C;P* zXU$HQ2+nxSxPm+}8cz>Z2h}C9Hk3|FthkBc+a%RryqB zEn+?C>A3&x0n6OX4R!wJ_@vboV3A&O%`8Ta#q^cCK6zX>_Y*^)*`Iz3MFdTC^owcFN^mKq{vEu?l)}bVC4Y_O!?DXwUru6F}A?1NiI?yPQ_PUtIU{Cub=aF1y&1 zK-1J%d9{n8)@>%~EdibzVmy~Row|a4Cm$MK_RH(M9BR@34H#uh$C$T4kOSx*17)LL zZLpeaCrV=mKJI{NWpc4JxW@%K(}&CF3Mt!KRemY? z`eLs*L$jQ*6kJ*D+K`{0d)>6QHc$wQ=Z89d9riK|{gmuk@Vn=q7?L71HHv1RetYPo zxyC3QX?b>6IdItT!UnwhKH{Qd9mSG22$ z7}Y-xV|r8|@>o;EqAJ02(j*&HjsXg>oZH8_uIYIN{!#Bop#A+}4~pqOzjN|BQa9hu^O0NHvO*N0gbluVT)MacQjp{}#^MjMUJ$xl7#E9MF##`?iYEr_ z5ULe@`q}sN=g6icR`1We&Y{q{?sDfa*f+$u{eMNWGSEHjJ zpij2!mCE&JPR0b#{??JIL&mSB%ll^Uw0VYGT~e_$SanlnZ#g-8vN`u!YPN%v6?vi4 z$WR7N_epm`d&g_6a~a|dTStJ5L2+>4pun~X$Q@3iCnK3YYY^wR2nC|e6crjIh}f}-Cf&lkz0q zy@WZ<^qIweU|<%tqZ1qzdKVEJnxCJ|CLSg6 zjh%P~-T*YhV2yJkw(Cbr{~6T{1vahoh9~`(OuwZ$6YOR9XTFShgXgkTQ-j8fWCH7s zgqV@jv3jV202U(xt!Y-tVsIxR?{;)v(HE^_hw*REv*1*%;xP_-m&4f-j|d`ow6Mn$ z6}xfksy8egD*%urZyW`Bq#=^vC`~KUex>{;HBI0?--%P-+je~q z2WR7YZpO9Mqvqjx*(G6)4d?Ucgmg#_r54YLiT_zTET!t!)CC1T(3y))I!!QWk!ayy zHT2v43b#Y{2X0rs@cZIo7MnODt-nX%4`E@0!>kzRvuvlXpM)h>0IoixIZLmefNM_~ z_c1q(@?q?oOI^wb#aR;gE9p%OS1cUba8xX6 zQjTDylE2*uax#*v{y(eGVmJOQBM!-%PJ@vxMN4W_nkAw*{i+9!zrFh8G)#9u5Y-PUh3x?Fp+H#cMG% zB1e`jezb<43+cu=0df5i}JA+E0V8mbBLkn_V&G{M=uTBYQ7fN+c` zVZXZXTbH?BlX0ptVUODy;^^TV^Ovh|%BRZ)uN!wdNEQdtApxwewB*PZq#G9yAar%U0l=)exU%(x%#1u|{(2Pc~VWDp+fa7K31g}b^1ey7$& zv)*of&Bb1lx4h1OztWtv(**%lC)2m8rY!9o7!S&phGC;MzFo~o3||Q@SbQJ*MFcIa zZMh+1IepwLdn-V=1fP7R^%cvE3OSo;$Yv4O?`b#z5u~cz4Uc)na`QQPXR8#00@6pd zarX-h*JzGuVix}`LkwW`L|P%^0ZITkxZl5rj*mU-mOFG%6T?70zXmMG?=9_;cM{*~ z^TgK7=byBp zWz(~fbNK|I|A-&Hkd``o;ep?mJggPTRcT7BxHbjr(}BZ9t;(>!z#%bx--6h_YC+iB z*S0gid0k&x*{TIX3}E5Q+uRZKOoeKC_NTU~A$V`TX8crracTdMHKSObS%mvBx;JIV&BNwrJF{*8lAZ?;q4n~HyF!M7QT4cIS6eaVTN(G$ zHj4Wh%S#K|kvH%!$T-AM=P$>p^KvzLPTn;ZlI9@zjS~OxpBVyL?A-p60&^J&k-w>$ zpXyD3y3uB2Nb=r;cCA@A9y31f*KU0oNisV*ZT>D^f34qVWd4|tJ$<%b$F1oWjG~IM zAFM;Jcy>?2$;fY2W%(*Q8q4IB2=i&Q0RX|M;?|uqH9{t0ORRkOLMm)rp{dr;oZl_% zGbhXx>)~I?D-U@kIbN$`D^f=c$+^>f$8@i_Pp%4!{3ZiTSCkPEfgrX34TEd<{+B}o zCT_r?t%qOT#%6zSYHH@u)jZI2=^NMCL*Mr2R}mE#E>zw%@M?+xQGO#67CmIaa@Ma7 z3arYvhbxZLUCX(8Fg#<|H}?ULjkq%+PW=9 z3m30+VQhFf)#LRNzf;#0Mn=|>R71|XtyKLy%-rfR|)Rf$L*?21txQj5bsFv(@s$uG~d7Ml1 z{tBUq9msB9g;Ob-d^#ssz7^+@FwSdvK0tSe8wGI7f7FiM70IM_r7s%|6rQKB{AE+- z8YIU<7@s$nx-qEr4*Bx;+X*jOMI%QSC!XuviiD>4X7sO>BSUlp8a_YW zG!1E^@M@Ao+v$c6+Pc2$Eo_{65B%6c?$xP-B>R5(ug z(yFLU{n@FmfIHCD{4-DZP*Z)wk*}t5q7)XYV(a6#o`!iogjl)I;M*q5{f!O-0+Ta z_N*LAoR#eY_Pl_j33C(+>8+t6igAgEsQS0kD@>vA9^%o5DQ`pH?(Ble&%iinS)P77 zNjzLt);fmVYgDBB+E6h@s5?!Tud;~)yhFN)2@W%K%P=?O2BdO;m(5CzLa-eKaOKq$8xmb z&9LR5`W~V!He*6*0D8NlhLKs7CT{c@zDqdoOOnOKM!jcg+ePRraSXknZa9fsV?wG0 zrPl`I5Z8rEQz|M^==wtW*M)?c+w^41%c^}r`^?(K)+qT3y|PL~>-7kYd4a2MwfU7T zbVo8E9Gn@k{fA=Js^f+jg~^y+mJ7IYL}qr4Q$uY?Xuxktkh>VcdPal=oA@a*Y}8s~ zrh+y4V|*Ws3{)AL6}IcNx<*@#_-V=B{rB3-HTyM@+4#w`BRr>(coV*+Zh6RvYFQJa z+d2$jB;@HoBw?a3a=PM;sm$ct3kZ8tTB@wO0fAV!1oWPs7WQCQiD&@s(uESP5*25# zVgZK_Y1ai?qcN&h)gCrMS^3KrMe9CFk|1tqfELTwd=C)f!k29go6tmD1%X?Xe_WO~ zdxcI1x^d*9>Se8Wf`}teq9kx;H7Is`ym)7tuRKaA3n6CLh;NP3tSzIj@i`a^u;59) z0wW$stn-nyw0u^}TvQnwkWyHje)IWRY_`+l4QHMt`5Be%T83wS8I7g|qQ?FCqeK1rlOy+XItj_%Q9X3v=RJ%ps4x-6Ugj2MfmBe-$ z-LfcYu934Ka#-s#x#2uH(|NBYB&M%hJ}ceWQl zsw3%U)tti1X~C2!_HKyz;PGV>lteSNb}R`P8t|?%)zRgqg}p@z3o?(YI_+q%R6uG# z+>HTlEPKh-n^A?;m--NuU7Bx6rFLil1@M$m6VNFA5%MCXCI;+cP#kKGSEBZh@&y|H z9BKmk?nQo@mIcmIEeIxe%d#L;OmF>iw`8--%z;njO|MqMR`YH-?ifF1iTUZ8N~_ig z>57u9u?>@JjGnrSPFU}TiA5b)MOigBaIgG4FD{YCPIWz#kBPVpk?nb7Vu%TJu9peC zYFmd*Dw%?V&nupIXBmPz1ts3CSiH8%Mtfg^oD{y0imlH${g$^XjnoO%wrDqxbK<-K z3U4xqO=%FxeQo8MA62wdts$Ac&YJ*SY3tBh)`_f}yh9BsMBH^JkBm~&^;~MKBr$Wx z*Q~yM_Z_?QB^4e~qVzc~(N{L-*5t;9wt8HsOnEOaH_^JWi-=QT-7xw3t_|+FElbKeo?_WmIhBxiW9pkQhV#vk#1|egJSXrr z;*ZdU0T^MLj%eVVod&>#9_+Q_N_mK6gamS774$0W$bYhFO-3~9>roC>hC+pL&O- z$0t474_&nCcL1(%JMpdOeNf-|Uw>g|RqR@&wXh6THRG&Q1D{rG^>FAJY5x@f0oLk8 z?IdjbTJyLIHo`fFSV{=uf)KCTxLof_n_BsJzccnXr5QdZW`GLX`pd|D*G{FZ?3O4z z6D249!w=S={KiGKn?h{iQv!;Q!VomV@f$ zKUQ@h#~Ez;daJ!(RJPj92u@;$3U{{K!WXjH`lbF>V0^Q)r7>^rlRvA4%E1l~*Ov@k z9Un8Dkz+jGlxTf%hcl~0oqg|GLPU zx@s|ZKE2%P{nn2Yob94GUu=66Vsw1#pdQNo2y zS*>@|fxKCwnoM-vS(LrG!>W(H1`Q4!19KlPM_!CfQ*bS`Q@wIyR$8UR(m~mZ#^EX| zAkc}rRhJ3CIGUS;2Qg~|y5ZgBd!LWbiiEqR>1)_LnMxvlbn+CC{_m~<+UqjRlB?57 zo1C15-?X6i6$?AqvKw$UOh!1SDHE)01^QXfN++4JvJ`Q`As+lC4hPCw%>EKQ3*m|FS*wvHDt{BN|YP@R@H59W^WuhS9M!h+t z@qcj<(w&k4G?W1P(hj=I^r?M#>_(R1TUvxe)0DD#KXbCxFgI;9PsH6mRXIg2u+-E-Qj894H(SFrs35pJ5S3 z7&g<=EV;e#i6k`3$V>{gXXLG)Jn*bhslUUYRpk+&)eUZ}b_|^gPQ&33gjhssONO=; z(s)dZk`B>eD8bE6a{H+6r668OL;c4^rcDH zk0+Np_@6R$REkR1WcZ>|N=_L>p{OGX7FX>Ky;4#$PuQGJ}_=ykEZAZtYvXRN_*KYcO^6vQ)Ry5rlD}Vez|p2v01Q z_8L#p2)pMS;GJvLC31Kol?NG`QG^B&ApJP&248vESZfmoPjGZaRou+)pbR7vZH#tab`l@h*zMJ(?el-4`p!?Qkn+DQHS-Z9roClp)J34^)EZb&HHu+ zeQ_yCji8d&b^ALLTa8^ttH*;E^%1E1fRY#vKP_3}QynO5!Yv>0pwW${76>e)Gs|$^cUfV59^>sS@|5S%HAiNX^TzonWu>Wuj|N`-#qIQUn~zq__s zu%?S!hGf*|3#pUB@-Pyut3DVy^zVOI2*l?Gzf_!&Tmv$By`L61bdVYpr%Bt=>rZMhup~S(E9-5; z;^@_cEjlBu;h$I`-MZm3Gv88Wv|F9>es(|g;qU!v!;~LMGvgEd_ElbT#N$WUQN5KF(h~qY`s|2(|gV#Y#qGny%EiO-GQxCs;TDT=4@Y}Ybx!V z0P5HL1QpnVEH#zAjhk5I2hh4e;v#R+y5xY6v;lyYUW+2e(|q|mI_M2DENEA3WWa$P z!t>9bflWE|Bm9KPfhtd)@Xh?mEg7%E3W6^^wV?a}-{Vo`A(b*mNM{>~svcQ>QCphM z$?BT4ZO~K~N6p@^&CK#WuKRAx^?qRte}9yA{G-FF86JoUvRS@nLjpojwTN zWjnQnIIpter+k?5=c+XfdN`*;aM$v7-2KUfI*@0!Sxqmbk1E0d8u_bgc@PT1f+&V$ zEK1UI$DWm-BqSGYj=Qxz2ztA`#f%zeKV5iP&)SA0e!Dmt=3(NEoF{xmYQzdOPiN=l z-M5qWPD1|6-A}0f$Zcr4%%?U6Sn0MOFbL96v2su%ER+b9TzTIVvoiz`9#zz<{P-BV z@7wn2hre&rkwGq(WG@4`+(6==Ro6)gAQucUw=Yl=4V zkD4@goy`3UUN5&w#Q?k{9>&`VP8g` zd7WRnBO^apr(;v3iVShC(gNN%=p>;$tE*Zg0+Q_kL<4QhFzL`{mnk_efDBvlptLSA z5*koItCgsEyjW0a7l6iHF2WY)&_B0{_9a^kbzcp7I%PORzm56DhF zItX1A0AM5u01;O*DUvJLJIY=n_IODR--d?PuuVvM3`=$Gg{;ib(1 z8YwRq*2GUS=ZSTYZIVFy@L_0lMUQUt2Wve+ske!?lduUzY6?-#znvfRsHAOPltP|gB>TO;sM`P=oPz2c&xjP%lh44)>yGAyPAxxRK- zhxrx$;U?`Gktv_kD7X>o!>RJOj3m9(sBIX}p-=tG4q#94gsNrL;7^FJsqBn5Cg-2k zv=owJ@z-j!6QJG^;1$@J{9{MEyKg*+_@)c|eE+8N37f5x@T6S@@hX{JgFs(B_+b^k z1uS?#0n9P2boBBkGFMnBJBU_PT;6WU0k+GAkHcR~Wv3%c-^HN<+jLEKbDCIag(>xO$!sNS+_OCHCi{&(7 zJXd;Qv9o!o0G(vbZ=Cs)e#HLIg>KV0#Ht8j@WqlHq-^y+vU-h0r_W4Rde}o&X@GYDthW%eLKSQ!)9h4GTH}7w zve9A37ayNH`3LA|^~1@^Hcn0!%Gh>u6bH@!yPSpm!`W*#cuwWPA!S*_ucOuSu-q)w z|D%pFN{dcT%8L@A-%@(CfqCszGt+zIp{-?M&88sn&p3jXq>3vE07?EU%i&*EfIM@{ z+xGjG+Rq)GG8oSTfsZT&PM0qDOfRMdKLv2aM`D5XTRWvb47C~mJ>q@CkSrhYZhI1q zAK>Z(HfS;cH?4J<7V81twbWp#2lR z2g6^G*ajO62Es02PK@l8hfK*ItKce+Zoj*JvZ)W*MvzSg%!*u~Aw~M8U*x_oG**;t z$73=ASHr|YY7FuVVP?FjjyoIC?Vw#;A%iy<(KU>C&59J3g<~JPy0;kx zzxv6?_0so%@%qkT&0kV+hct}J++ab@xuDu1X&{f#0NH-VOfu0F?O@n78Q8`FYU%MdK1%d#NC1O94}D98l(S7fFFKe`&DCk0xdX{xGPxFbWmgPMaw zGAg8{;@}bi=UQF1CjdGSx{{K2e=7yo0_JYjt+SNY%f*`@3a2w8Eic|bOZy#0rMjmF zFi_@MbWs3_=q5BsLO)ZxfA)`q2bv}W$YvP-+MNPX4n?z?VtsM36MJ8ykvN4Sm;qNz z96lMY$~*2eDe4nEt$xlhDJI2&<@%td$ajPS6yw!Og)4u){dq)o^1ZP%t^eE5YSm}q7&fHtRWUI(PxJV69JVK z3wIR@pYx65CNwG18nov6{15ivq}E z6EG>i7ERi(EY6_6gF{W>Wi0R^@mMndmuula1QW;1Qae??&4bhgy-`Qnx8}BtN7Fr7 zA>mW<*#}c;{A>kG%IIR$)mByjwvk@A7}Wo@%nwTJFHq)4L&pRfDQXqdjt>!Buhhmq z2h#dhPMw>v$SVBcJMlGj(jF?VX^`IIrZ_eM5Ur~hAF$N9{J{#J2IYi`U+GZ;6T7}K z(M_;P*V~8{N=D&r6cz}0ImY9RVuD(AjkJ!4RwT%r)=M!~_$3Xm`V_VE8tkhZ?vSDd zSu=sFK)fvZ#c9LITHQ0@Yk@rg#|`@axf+=<_muc&iTVY}?L>;;B0G|r-vGco$mTEw zmPo_Hlspq+1z`a|O~Io2MX-eNK658OT>U}v-U<@bWn%w8hzhbfOsK)&aKds%GM9V@KPZ*Iitw6-C{(*lTvb+6_ z*@m?EZ)Cy@Zwnx95`WPy=mxd-YmK@W!DdKWEeDZw<7=9%@nDcr zEKZEA&O-qQR#T8}ST#TF@EDzNPaoi;tWd<&a+s?7a?UPqF*!LMOc@*yLxLrd_(pQ~ z${B<_x(*ykC?4U+f&P!tF00P8`_>j-1PzrAG^K% z^V@@19icq6h$9XfiN6UvSU;ZB%id@p4v4(t1h$Z^u5{#Vy}DbWR;_@TOZVkD>fMU* zkA#cAh`Z9zM&i@~Djv1bU0=O$b!!bdIc^uAy;l?<;ejS7-WouJKOdM6IDWa&>L!xa zZ=ogrhqX@NR+qChNhoO@Lr=Z|QoD8nPU~Mi<$YMaa8sm`fN?-tR0_36sQ)~eq^uJJ zjH60kyd4$a3*vV+NSnLy_+YXL1qB za~3q=e-JJdSxa#My7uW}u_3Sbes`N`GH+4U_&TLM*vy2`vH2(_5pBlF3Cm=p$Vv}u zp}-belZ6r+-pFWrb4vt2<%`3l6`o1aQgONCpFgD8z$gpI5#+XnYmN6|8UU`Pcy^?H zCbg9c{av9rnNdG6gK)jVU0|0wNS2yw#240;Y?BcinNPBi*xiiakJtrx}>%o9dr+|P8*#C?!p1+wwjunzp)_mJck^^WQ`c6kAHXD8gz zSk0i?3+P%FKmIY9q)F=CSO`A-Ls=F7>IpO~%EhZtJX^!>QCa+RwqcHE*TKROyBHL;;P5;OF)3O56t&dMCt@0TGiU&Dtqj zI|8ExQGPb*Gr&io@N~|F8d&YWO%DTcvVTII{UJ!E$@-13Xp2{{&xK@m&Kj0~833Ww z3fDSxMlTKz8xY^8bIg4Oc0eat+R#ui_fkpqxAJ$)0LdwxZNV2QH^QFDS1sw`&^Qp* zhVdA%KZ^W=iXzyrQtyI(bk=93=6esQ7d8=r-{S0QSMLlb-7@{c;iHKEH@ZAa!{hY) zqmjdi0C!)8Sry4h)QuPJM_QS3t32$bX0MGAnx=7-EWN`Oz9Q@t{`^MVPnoJR2XtXM zkj-D^YYiR9r-73dCcIT6N`5`m(Qa*eU>}OsIxK?ln%hb43YG^jTJMQlfs2h8!-!kqDWXGBSJa$hf|yaTpeVo;Ih_fZiNABFEHCB{r+hd;m_yFD1kf zSmEYjVo$85Dvcpw!e&g*sXPZ& z_|+RBM9BQ;%VFRWICX<6pQBJfQ(vU(Yu(sLZ3?U|>CCA5W66f#fKuGLK)M-0k)2@>t=4XQOqrj0p%g>*sL;k_Z6%)Y`A8Mv?er)tR_KP}l%B8)Zrv6Jn zr2=(3RTqw(R=q7`p~m4nx=W7u8L_V-In-;0m+o(*e3x5%zV~prq0vQr@WlcN#Qpsf zJ@8D#3gW-F;j>vJmM9E3Uf44Ah26eV19kv#tze#K=+nK!<>)6tv{4Aqu>|~j;pI%w z=1*4i&)EY=sTXoNmVwJopc4hAp%r{65OwlS5STrH7!qzMK$L0GRnk5%50ob(=poZE zcetu2XB*?@#sd7HC8ys0ixH5T5A}i(EUd-;`0Sxtr_E$K>Q> zXCR9hzS`xl@{r(TAtHs4(5m%oW8?cVp+_5-RgbMVRwIiZqClYG*`hEIX{e&`0WsiT z+S2Uo?6)d*28Of6uIy~+q4q&+b!7+m>S2-lY(;pps)NKXpl*D2F0=Ed)z??CO7K!U zP|Ag}v8k-^^rbskZQ6LU_X&QA=wkOY!+;;ly>EPOYTyHsBn z7b$?+)eqmijqP!7Ix{_q?U`B$WL&IKEL*2iu_HBlH)+eS8f+pC(SC5s$7lbSflEQ9 z2o+c?z9K2gs(i5o!Yt^F!UYm~@?wCv3Lt>DMf^uVL(?Y-iB}+K=->bIQOQDYP7oOz zfV>-vcc~~i)iAcIz|Vjn5Sm&9n_Iho1?S(#7<%C2IC*$X$ntUydBne?g{BP1(B~1ac!KhY|Y0;5cKo!7D z->Mt`08Qw7O@^TdQ6l`ih6!|RU@wB8w0qvXJdf}2mazcd0)`98DB!;@5sX9<_=3T_ zFua7AX+Y5D!$$Q_5m#Z-JqO9Gyt6-hCEiyb4#<6Df8}+JO}*wPf+(v4j*%t31lTwl z5f_VeQ4HagVY(^-^PB#F{s%*-T`<@h*xWo^ri76^kcNZ_DX!#=f4SXv#wCY$%mlVC z8t^#5U?7>rVqJONHOp5Yx$^ysKjm2tX?dT~e`f@gj0EM6z$pPCbn5GC%0u#E;Rdk4 zL95_j;WF{Xju5QAP!;`)NEyw>;y|MNe@pc>4BdCtpFlEQtu&Y_S0B(YeKT7>XXTwt z>@f&DJPV*6-vjS3u{=$#HP?SbPZhxcdsx{!!;3}P+Z6&13kH)2HWq*=R|&+9I9>|h z>=1?#Q}#;z7kFw$*-J;+iy3|VAmB+69F8aELm7Ox$fwx#u91M4cFT>Jb~`uBtNc?8 z_!m>KCUdaIlt=macVxi#jItMrviBopFR`&s*Xt^kB=i@H~xf3)X3%%6+CCTzNT?Ffy7!iQx=pbD*Kr}b)mX$ z5d{h<>Ptz~opTJ`(FZ0d(Qb)k(FcPOp+BLo+~gqOo`w0nc1Xkrgs5!FWsm0MGG4PP}=J z{y{99A<;vSy1?i6eEGNvI6v_k%C*iwF~dNF{NV16UM+?4B@)dz9pQ<{H@5wo@(J$? zxtM(LE(c6klN5HfgN&{vZe~GeCDIFjurZNSG~w^A09NH8VK0G#@dRK1S$ZzSP_U3$ zM1F&kb7cCud7lKN8Bf0{<4ZC2XP=0^m0_$B$)e^J3kU4f;_v;}RUy>yXWmi8AtPep z+)6Lzb-gq8p>E=z;=r39AVQ^+m!j)->pbfc5Zr2lW=)yDY5V6fq9~3mXPni6mCe6j zb2cFl0z@d3g*jgj)u*>_fXLUXV6gF*S;>#p_|1S%8>b63895iks==dztAk=e_1*ID$sYp|Nq}R@c#|VBArE{w8Ij! z!4ea|*?k!f0oS#zahDrsETzQ)`9ixVw+DhNWCkeTnS$?DMPfzs{$c_t4H)j7$SfSC zv{(F+X<3l;D63~2%lzE2G)eAqAQtQcaIhybLQp$(aPySK%YDd1_!+5j79VBrWoXh3 z6L|0ZK_vvt90E2#rPSw&giRZL@S_jDLu-~oaqIvaHB6vfm{N+jgEuW8SZ4tf)As4- zr~Z;5CvhR*&xHNV?^ex${LNK6p(&rbh}`qLDzWh2w6P+3-P@F(#wv0RG#f$b$MDovW zuyL?!$;x90esBU-q=)0swfsbtK$!>0AjkBs+@Z>lkAD0PndkQk2rrQ7n0B&x)9A@U@@bRPy0XO4PUKEBvim}Pl_G%v$AM$!z zkeH0<+4+0xRS{+n$cTKu?0E@)WG~7im`=X!v4KzPx_tgNn(-BDmyjn_RJ0Auvxd8h*lFK902U*)v@w9ywXu}guaOsyEN$z~$x%W}I;{hX<1I2`k9RBHz z6fhCP`XTzDQWTWK;lqy;+5JvJI8}IZR<-`22s=JjHxA!KBu%}gw0Ukz**gcNgUY4e z!8#;>jo?@6bN|n#2TfZT6_jK0Z{63Ms%ItM^Eb$s7XL|ah&=W!e`fOZplb_amP@w^ z@Ox<9XP(CzCiPqIVDtl%S&EL^f%jAL3G;|Q(9ul&8qn{$Sds^HnI5ol(E;p8fNs*_ zFch6?_`=uY zkN{FbFH!>uN#@1WSJD5U4Dg`o#1saDTyBQQr*& zV!U(uL!qR}`3nSkQLZHaMEkAD=Criud}s5y*960e)z|7Jppx%t-!hdXD_T5(b1SSw?n2fOsIl|j8&d_#F{t*nMXh~MvD%q2dX+R+0`-~ag>B5R! zyb;BnJfsmw?>Mt_PCfI61-o`mrEJ00`UwquiFBrt0r&pBWtgoyK`J$B+jU`e|Jc*L zlCPo2MSs0;#=8xyp1af_z8drDp>Om>cASNu>*M`O2( zbK_{7nSsETOlD_iIXX;dW{MG(*e0ZZ!^Lf1+%fL!n0^xk<)Z;U7ab6=t=;fMV~irs zhFk+d)6FgV?|+1lU|ok*n8}l?NMw_g|Mj3}Ao`nIlmSV^?b!_^T=vzUx{OxQpDl47 z2b;Kt4RRV;{czWbhudyffe|~NMeHJO?mZ(lEiL!%KG1zMtCI~%k#Yfxy*s$#ZS#kR zN9|je8jDSi_BH9)OE|5*v@KR@RHb8X=AdAl%5+ZDy&`u%TVm zIA&Wi-QZDWT(;2FE#hFgM_R(?`#J70;AcFMo|5R4%+$EQZaE|@oW%uRpIC3^B)O(AFt+!x1{oJG<%nn%mN#Y|q{6p*>MJe`aJzt+F=&ntSyTi|~IkDT|x|+p?i^q1%lONgVW?Q1bG0o`F5xwFXBm24)Wf8k)UNd>6VagW{ z=ykcPMyeg7V@9jv@IETO6XsYqltPJ;K4$0~D_?@vn6-CSc}jP7_yrWUc+2P*VbJtq zw;d1K=zh)d+FAFJEMzbF#vYr#%*-2^+Kk4m-7S>$;oSW2vV9$ltoy;G7JpE)p=SBE ztNuO9++2i;xp1+g<8DrcJ?2FO42DT478Hz_oADBqEzt12E=#JDy2UwS>YbOo?fos8 zbf8g}2lZN8L&I<0I%kY4)wo|M*6uAeuMjnv0-Cc~$`e96pBzhr7o=Ysn4DHJUtpl9<|lMXAoV>7qF2*$|FTXq7GbjKTh1_>=E z8#V{39%9KLD7CB>C5=)hw47x14bGp$wzR zq|q@~;gHQ!unss+v!Z;JZm+u!KEXUQ8jtbJ>D~<=Uzn$e8XXngG`4f#J(ShAtD)EF zE94lJ;hCNLz^}itaU#XfQFOh!qJGf(U_5oKEzclUS5hmrkLxkK75e&ho0ds;XiS%8 z9}2DPh=LPq^={BX53CUVlykO=1ynz>n!}5Nt|!v68cZ;Aj9U)W#2Guz40uYu;*%`# zDC<zab5)u;Tu9Z^3lHPOF)&Gm__-Br~S4=Ct z-9`2FOe>vMRLoHY|53t!ZFv5f`9D?n-z1(uFOqh5cTa7RXITr6N7L1z0+?JRVQ7p@%B+FL;^Eid^w?1v5@@Wm7s!v_ju`?21X% zL*J&UcX<6cGLwM%*RrRoK2DP%Ti5+5Q1;__>yDz}6M!o!mCJvw%Vq2MkE+)doR{@raDixwoxBdi%u$BP|ov;tLwOy+v8sDG0S9d*OERn@jmA zmCha>T5#qZ!PyEiS?EzF$*^^C+n&6IIANJN^YNAoI9=amRg#(@>(m@rbajD^2-SxFXt*IhH$Sr%-wHM^73e z-VF?S5bd?Fy4PZzusk8WL7}|Qoj|ym zR2bFI^ce~t`uey!p#$wE3|7(wM?Zr=B8zBTz_N%Sacp@6&h$rj{@##|u6~jxWN8z- zY3}&O`(ff#;2lfpZreh`s(Pbh6(Y(q|7>uTEtc#-9YbX z&TP2dbfOD~K@Pbo?xD`rgx(6p2I2E7T;Ru$26NAoqve`tR*)^qXxfN>nnMVG2R>Uq zdEkh!=&ILr7q=(ly2qI6M`Zc5jLE{!-HEgUjbn-R3ZJ^3@=5#MeRl}{cK?JV&z^s+sn=#H zQQ*MIv1QkZAgg?MiKDNow5EU?Jl(jJC&V-JqaU}Kli+xjG$ z;iU0w@HG%~FtEP(7KzocjQm%7ACq_cW)GNJQI;|=o)+h0jn|ReyH=!xrIrP+idxf} zw~GTuH4wuGUI}=w>pEBC>uU2aP=lP$11*n8h94Jw8$PiSLrck&vmKV7h?wfgjJk=K zYk(0t@fT4&$SbjwZ~O_Qowwu~a&yFVU?a>>iZ{n)!aqZWxQ^-p_Spn@4})k8-Xo8> zneO|KPy(pP(VvlNB!{jhb3S9I?&RLC_iIO_Ch{n$R1aJJmJPcbS(mu;_M~jY(=+*>_z|ICH`qG zu-xnm!UJaAnQb&yaEFpc{-66Y+k4b8C*N*d+AO9)jVfI_%|63&;)e18edGfm%S9jN0L@Z=95 zB^$6Pzn9W1P>2EB(mRSSJg5U038AY`4=JM5F;wX={Q;+*(<0fv)Y3ifpAuL@;eVGm zz^xCy!2?uDyxYFri3hn{!BI?%&Wp0iALHv+T=j;w7Y0v}jKP;DdOanM1 z)x&pIfFeF!29g(kILZ>r3reTEd=04l&CO}1=5JL-BUGUSS;f~^P|9~lno?4t$`E~Z zGco(49p#E zfLgBL-p_>cp~FQd1;xTb<|-&@6fjgix69qHI&^z=N41@ruv)tP=N9_6(?Yi!<$v3> z-w$j?d zR4cvsbnqEODBX7o!159GVh*zWKvBTPz4F9xjA@@jOuz;F8WO1w8KAmxZeHfQ$yB-j z+p{1AU=}vi$Ea?&0?(o>prCnR4mae+sD5$+b4dJnKoR;8Fn(3|ET}&CofuFw;WO|I zaP~@5+g}numjRUiS`e7G%MXDIYT&}z0I-A8z(RSv2wW5cOXuezP?V&r4fx&BzZzxt z@(0ediX0&IKmWyde9S#MDEWLBsIPtKZ87gUA}^ECJxva z#GU}x0k+)g9CE~$&7kSRolZ?njP zxPdkPSA9>2^HAuoVCoZ43Zo17#(4a}9eOL@KBEvcmd3L zN>!9T@;(uG*{Xq|>KoS?|D6T+tW)xe%Keq*|5`df@P&}iKwJ3dWt^kdD4JBp1U(@1 zpwoS$uTMrFsSTMq>kpa-Eab8!`LCoGfTYiZ)feeyL1n;a={ESK4R<|swzjyH?y+_& z^e5kb+JOr2@)JNH>|cugeSIIeZoQQB`+ATb_r7A-X-rTFXqys-N>hgp^VEVkAk908v zYEl4pSEaDUdPHmgj@niQmlsi(+Noi}G_Ky{;%e{4szgB_k+X60Mlt>=9Z&)LBzW~# z{=A;~TTcIAkz}c9$ns0kS_A?COOgj`qK;C>z104IV?u(e!&V8g;bQn0nxSv$xPW7uMzsJ@jHVF1MjAxKWpVe>7 z*0+GdF5|9GomBHZGTO905VUi7Xo7;2&g;Pq@%^6Sjp50y8sIl%!Xtu{A&ZIN`pxhi z(t>Lv`K?r{%xqM7A?s4V8u)o8TFPyi8te53-}Gf6U%Wvd)UQiwCNig=F4MR^wY^c6 zuw3HzWeAr|T-PLbLCZRym31E7N|x}Ag+MD(W3B;*G30Ls|fmdQ)d zbSRJ4)8^!eab|0yh&i#IR2qv~L{sAyB2l=#s`zzHf$68C@%vTCfKIySYG0V(FOZg| zGv3r4W&)~_k8=l%zWAtGCK#+fVoa5Paa<^RK!-9`*PmMBL4l8^9ykd)f!!XWlDkH@ z`0@_@)^&;m5`f0JCfdF?Eiw>SX)9DUfWX9_6mLB&G`svJUb_NOd6>+fthyDkuIVM{ zv|Tdi-#G8sSe?8Ba9(b+5M*lmz%)_BguOrN@!M%`CfPLmrxk09k1#`o&ks>!}u;IS-{Y2q4Q zcb3Fgej<0g_hUL?aHFLyr+@MRnbdy()x5LyFn7R{UJf?~Zcii8*&`|^Dt(i&f z>wWuFqVDv=UPbrT>>7OUha+LMF-eWYggbLPg1ovHi)FCuh5c*T2dFm{UBkoB)o3a6 z?c5&1T<~;N%VEO^(mT%p5iK3K3# z@9~LH_wg0n^?_TPke)r#nr!~(wywbzH}94abD4SW_29inmN2BhGbRB?~|KxA~59}h|txo4h_bl{7BltbGz{GEX(FJ0mr|NruhWUU5jR8As^bQENy9$ z1zN(JA28{Pm{KUdJgSrStXf;mRhzX4?NNzr zZ>ukPp1K97(Y2`;oW6+qw*r^Pfi3R(Gpjeks6N1g$?9MJcxl z^&Dp<9=HWy4Z9z(I<-t`ZwPp0ILugwuy*>4A4FAVnd_O@pi7Y*+Q>iBJS^r$^YX?q zXZ`DIkM-rpR$Df6nZG8WIySJf*eXaK@;I?$!p~^NFAg=|fiw`5;E0xuuNP1*;$wJ@ zhBd*GRCjj>$7cwa{E&dF@@Xi6^m@c24_51eGZilJ+d*MnH4Oj7Q95)J$XpObsYdPQ*bn*|3w#%ewAq-^=#MWgHuVB&t2 z2ixn5N!m?c3*W473@QImae6eCUuJS(Aj0QD>2jOpRJq&c^>_2%zd+la(uvI5EQ>`72 zJn11jX~w99mAV7V1gLS2@d|edeycFUe;~mnk%KgT5WPoEx5!;aQ3HV z^;QeSxP2ezu3*qY;?VRA2Tc3#nWlPucXF0_Mfd(#gU5ikhvWSh*=+J~-xOJCBnngY zw0jR-+8yNPeH|R(a91>$eBVbFJgQ}HW$7ey?nsxX)G@Peex`J-C;Rhyd_*te_TqY< zii!C7OC1&tMM?4fbQ(EC>A=!aOH-5ZW)1=Bz0ZuE9{;pK7o9Z)D#$^OG_>{rcST5P z>&V(UI+mq55b7OTN zupehvfAWD_#2L@~FUs~er7B3xb(;u)3D>R0z4V zQ(-HG;En7U2B@yh75u6@19?$~Jm%UgS(@dBhC~72HmZ=r(2`hxK3d6h9#60Q!6era zEEgt8I3e^80hAni;b`6r|0`sDeL6soM61%}x#%ij+*$Ba?{sJVktovk zv6DC0`qoxV@FtZhUPR=~t%d5Z$|tSn`zW!-?%p>sg!O0&(m0Nn-FT2-Ic-_3UH4sU zMWUhji=d9cXMU|2Q6b+Xjr^iGL!Kiagl6{L-9vF-5h!z9g{;KxSe0v$Z&^X(_Y)^M zj{;tFPcK)5u|ra-z8ACCEM!9Au!9n>ds3UiI@=Ve-*3^9Co|Bt_t*R_+lekSQhARZ*!ArIF_YFJ8KBZ*Hj8WIPBDvcaBw zMRiNXPj6^KbOlm*w?NF%;2HxLBR(C>qmRg5NJ50as==1QIO9pY2G3GL1!ReRGkdLa zqb)zX<&{Bd43k)2k4nn+%uxMaQ9O0vod;lQhg@7Af!Y@UgU=^ju1(b!bl*~w-tp8S zZ1#q$H&pQ&yEkTe=lFLUZ{yizDtF(~a@%upWxNTvkSv~FI)?}v%Dj?4TJr)=yVhhJ zn!cw5;42mQ-jtOrA7q`j8sah-S)!g_I65E+{Bg)d894Zsea%QuFJnZVjgJ}IF#1wn z`93nlrG^+J&D_)Su{_<+@TD=dVVE>mo@y z&xgk2l)!eqq+`3J(Z3^8i7@iGj84|N!6n}^IQEPaf{C4c60P^L00{Z%w43i=r>mFZc94F}>%A-WDTB-F)307&OM!XL#pq78_KGsv$x3y|k? zn=EJ!vfg2WHc(YT3J2k0X93 zw+75=s-hgr*G;7ciP)>51e(J8+X+ZR8Gk2b0>D`p!ug8z;xd%JVuz_wnoz-RhF@i4Z4k+m8Mbm|4Y`b*+anF! z83X3lnEF<@jnw%Tc;She@{G8@l)-D6M!$emT_Zsn%iq+i@RUOl7r^QU&sw&0rokEF`nWj^Y?!;xQgMftEH zlvVmG9qSdE2MtAfss`g>fw+QXl%P>aLS#dwchORl^$Wf)Oc3KrKA26iX`SsS?6>H9 z1=|(0(ne?XuWFa*Yzl2vBTK1tao1;h6AB3hm{LPpbji#{%{|p4~;zK4%yhB$(74;&%J7!G+B@x@*rwX&dK=w1)vtj^_Q#g3s#w_x|sCFZ5zB$ zJp$GpA%NnDMEFa!-ZXlTZ0hqx@77>w?L0j8D$ZkoFa*kv(lR=Ye9LHckazcHwEmsl zq*$VV79|ZVvfSX>9m{Lou;U&s=!z?I$TYNr<%`Kjv|T1qD_l}yRXhP zKg~#{>$!im>iBYYJLibKfU26xL=E7?iC^$Fx$p^Lgy-n%O5uh<&3&XYLYp*IWq|woa%|Oq!_cimM6lHLYQDMa5+l?8vPeGrpK2lZ zYlx&~j#g-J+0Y!Uk>3RB?!=3#+MjJ(Jl2dv2`3CYqqX$XJ+yu(dwYPP9CDCGbODcL&Tz=2gavBz=jU5kfDbn0C%n+z<{N2Dl!kbzkjOXtYF0$LP%``hV zCfgfTk>Gb#W({Db?JQHT?3dVV_}&+CwKP~@K0cAjdFBIbmhOAKnu%34(na9xNxhOg z%YF|r?U&e}Ku|OH8-> zmjCpEY7iaH&YKF+fdZ^QG)B%+s3#TD$pe(t>$_dt{`%lhI6z*%qmM$a5;ZA+QKEXS zchCxd{K64)HsSF7d4MQo`=NB2Q0ykadk$=nWKTIs-s^UJmj}lj*+WsSiHY*F0`CW7fXCPRXr?p}N5s z`2+tk42j?edIr#spey2Lr&V(T9d5;H^G-&nEa;6_K!G_K39UdlVfi%FVX={j0!EnzC_Lq}kHoBl zqK48K)#uZJ*{$a!;U56QLJRa>UDaF_0zaJ)`_rUpMf9))J2EKmbS#wE8DQpPUN&Gc z@M-C`=0FE2PV-FYWcr*I;COmtX!vJI$m#q(1rl+Wq%BxN-vRypt)ZH1|2ramM&6v* z<_nA~Afl=;6HUfIAXa$@cPNxBoG7wPnAAdGt_;EI=JMCe(@gy#QPlz8)u$db0rU0H z1)k408}+=Ve|Yt+pDr^*6%KY(fT=d0p&B@53iM`wxUgE8n`5p1!jJlcO73;a0&ZWR zId1bZU@lx{$ZA{LG!UEy^c~FrKqa723kfgvEV@Q(?6(N(H3h#&-(|=qWB0YjTmgUs zWuC_2Mi)CJ&u@H7+O~v(T;O%5nx?k4qn*v=Uf0H^`s3;ccb@C>Lm*q_O#L;|=lp+i zTHZrmouN#*?*f0l>&18|xaB{)r0LmGo1FEDJU)(7Sy+w0RtTjDY74k{Lm~Q2U*NFS z&XS>oy_)c?N3(mpCjb>FSTtggW#(4JukDp@Lj`B_fJ`H+2(%z)nt7S5NiwOIBuUD0 z8By0g9w18<{Uv7-$Og>C=-ht4Y&~Y#7vtac6gzn4JRPt`a2H zj!ZzdagD#qJMk>t#ik!kJtS?8w>wGepMa&MG$Oj0ylA9m{V&CBS8mfRR~9eSuhD#; zyNdLbyHC3`zjde4;U%Alzn`5SLFbQytXn!dc$=v=Xl8marca&_!OeoHfm!55Dq1^NOzStf{<)oNQnk7-NzKQv@{(Kok@#TiQ^D6W`d}k z`wZSH_+(J54M~6K;(;)X3m25opMgz%JWNJhw(Q`NHGpmc#q_sA8H*7_{c|MrOVY@Z z9Et6CaPzm@9J0Phm$^>ZAUs*R2a_jS?;pU69yo~?t{jV&JR$wUyV?f8V6h%Oiy966 z6{I5pvTk$piE0f`oHu#SbhwIm%=C5RS5Hk6doprQkW2csve{+^pBry|yTBnxp+5LK zvM6kSvHR%**0jXBMxQd*FEKU-h*g(3{p)RO1=RgU&5`Ll*!k3}b*d=r670|LbwBdOsM9PEcw5G!*?z_14M04UfyvO9ocx zvy}a7!t}WQKmEnk=VjR0oARugT_ns}1~}(AM+Q4zbNri-vwaI56!#=v9beW8yCoc_ z*y?$UW+ZC(3TrUp7QOl(#-RkOCtjDsjx++Z>BA<}1B0)Gy;Tq7x^jX0Pl5k==-YnI zRiKdy6q;O1XZ%VF(MktO`Qu;Yr7ISe`+Wq0FA%UqcW#61m>B;g$+O|Oar&7^|D>Rr zWwS%~)J@!J0OhAD-BnOZuvD9FU>?7y00=GNh=KQV;l{n|`Hdm7vTAR7%c@j=D4egt z2JbnPg#Da!jHPy97#3%r%)3+qN>6@p-J;o7?DOq*BA-I^zi*ywLie5AO#SdDnU+5$ zhohHm-u8+MeWrRR^3Ct7Yh|r$9a!4p%JCLp)~_P`Lx+Jy|5tGNY$rv{78mjhH{zHkAz#NFnambOCg*PLqIP_MC>e} z`;)OJ{vLY{mdcz6zyA4?{fxPs5Wr{q2n^a_3t1PG$mk z-Wq7Ttg5svBJ{rH&P@Yx7RUz|gCQFYig%Q}f-OOh)a2?Dw#ADFXz13k0<*i*DsRzW z^=wa$#*Pk)&abIFD^%rqqi_Ob2F)GG%XZUcJwh&7qOoVf3Z@M-YP$ zaPu_x2N0Q2Z%(VBMqT2w2S9(2rGnpsewprwH805X3OXi0ae41(mi(&3CmKvXVHVOh zNrRty@lM-ID)EK9k2d044p}=muZi^BI__WdASa)kdl!AC`N8Nx34(b23~azd)WBCI*?;cx=Wb^S71;Tv=*NbKcw28?M95js%l9d zMN(_!O6R8wW|_j@Z=zS|Yjp$FW*PgtZD6t1$JFhvm#lyfIC}g1fh9m`m7p}1R_!z< z6K2>Bxp&;c%RL*o)l z-q?$kZSoh!PaSQ~R9}2gIJE&{1lKhmY_^+&23inv(gXhzBRed+0nuqX;gvhsnmevZ zds(Mg*D98*Jvf|phSDJH4E=W&hq4Rvm0T-?Z*C8|4}CbD1dkVfF|{8v{F+^&i#75_ zShOr~CFlz3(7HWPLov}c>HB0`A~21iOIG@#McT1V4~oiu)pRIdLQPYo6Hly(G#g`noAObADn^%SZx%j5k^sMy1bi_B$B~xof z-yC@jxSHw{O-VwMzhfypd<}qknL5Afy6R#`$zFE4F_$OLL>t<3oAu)h3r&XVJuhpt zgcgO-W|U38`8qY}CU#|GTQ9)-hKYl*lFUOX@g%L`l1kw)VM~f<<>ES)w zimnz8kpti}>Z!GK;6G>&IT`IMmo8sRlx3P}p%4*mEUd%7hyl5P4OX%G8bZ^(?HW3t zj@{;@X$s5Aju{?5Xoa4)pg!beb5CF|5C|#gJbCiv52&5Ju}t4Lc}$2=Q^WXpMH;9c zd27#$n2!Af)Bi%@Yi+NHhtUI4N&6y>D~kcC*Aig~nYtDv9n+iE;}hzB@k)Si@RgMf za#|B{CcI~shp+Xzo&7oK>c^oWwR8pUxI$fx3sb7myPc5p+aWJ4NUuyetkQSf>896y z>a{}qQjt6MSu?NY1K(dwlszAFE?$8YE)=&YoQ4Syc_=330jM_xHp?bESgP!Rrw1?h(Bys${x4}U{yc}2{!-zWi^ zDiaz0h5Js*7pdJEYHEj)qWhO+;bwjDwf%|tID!O}rOHzB($crzT^>~9IHsPH9<-Cv zN2hs-y_}a>?gZP2@{}71G(x!Ue9P%P0hk}PF=6^!gPu|KaLZG-YjK~zR~Zy=wg}6% z_z5PUYFks*>&)k{oQ1LpvM{pUS$>^`l-!Fk*(TC?KAs0ebfcx4f?KrbB&^x(-d3yK zA!YkAv!c5O6u8nDTjQ#(wqc!_SIA*2Wx)qep;t$ZCn}Ve zG1~J^^&ypIH!pel&5K{zc@$PX9ILl}F!_x-z+3D#D_OAuxp5<*nP5&`buUl}jUxSt zzun#3epsqup2_5p^PMVjk8{gSfW;SP42#}hpCh3n%4y51|n_0@e=WcBE3^IX9{C0yYA zG&pV^jdfJ>2T#*Fxi2H~<1i5IgTgt=p%Rg4#M4WWQlWc%h5H{GQun9p$E&MoZ!}j= zqr#ZAaV!GyADjj0FB@uwAy?a-u*`2f#Xjt$gqn|j^JiAELUWyt#VAz5X2Z0?(bBwM zQ$XutOt9C_-qf7k8&=J}uT>nn;+{{=*-Dy8+|tv)SXH!hc!q3Moez2PWu)c_HPwW^ z`S?gYM89n5@#>!GZ5O!l#%$A4UgIs*=~s83<Jz5 z?*8-mQf?Qx4!fioi22WSx~9vxRb@E!cZ%41vH-3l0J_C@xAiXUTekKG;vQ?XLX)m{ zQGOqg`z{!$^mw)vDlvG45_C%3{HJ48Nv{;PnY$=K&R<)h23RlkYk*J-q#pS%YPtUh zs<^-OVV~WNO-zDygTyrO%6A$hGep_QVFTK?uAsxj!h&6Bfln15u6SfW6jTdi{JKW# zxdp4aW{QqNLdJkZzi0c6r=yU|mUS6)V5g>S$xWMN+6fVQ$i zp@ug9Vv0sB9n9OZ0ko}_uZ%GZRi0~PL`U|gNj`ep*W>SQx$nSP!E!nr=1U0-&PjT zdJ5kvG?cnzo}T0`wWNUA$!E4@J?#=O04q(BD?=U6(3Meo$43N9wJc>Vedb&l<0xmY z&y4g>9Tseuo5sJP<|4n{8DTHm-4VWE#sxTN{tW>K#HkBaxh>m@r$aoXsI}T8vrcDo zZ;H{iwYy_j2w&8Nf6C&Zk}G)<)jG@>n#jFZnkoX-A48?+C}`=3G0X@W&e~Oc%9cw$ zP~c-*4>eTGoti++KBIBP&NCY$o6Bc3^nviTgm*KuSL5CZ9e#mVl*!B>8!!qx>+>}p9Zu$ zJ3Z}VpPT0@J4K5B~fmHe~`MHE_`imNb>VkJ83sf4nhIQ zpjUt7;jG)hOsTs!t$7L+S3laaN=qCB=vUfpzSj#S>}PIFJ(wfjJ1;VJ8>}O?l63J} zj=#M~^vzfW5r!G~pD)ZK-**DZ_}o2QygYZtf3d|{-DH?ixJP2)%9I*O{6)GN$YOVs zl3GC@LeW5vn{4yF*=Y0f5DL7RkaM_bs!2_jSC1CHR(*{Yis=HfrLh4$S=MBbqT-bv zq~&fZ3XFsrP5(&3$atX1p&~4~6C9h6mTLM#arUR^97oF3;rHRwlcWP1zKn%WviPk8 zi80ZbS8vlFioGb4oIr9p+;Nxfqo$IiC5$+vyFGeQ#`zprzyvRszP+my)DT5 z1K*MH4vTa%-%8k}@q|)xV)up{=CI^DUV3dkg;sl`@32I}@PcH@rvUay=$h!jb++6I z_6qTynLK+kU6{9Fsit_-dmdP}fJZ(m%9_#MrUR?_R%8Lp{5DIj}tu2i$1wbwt z(U4Gs8y|-U0fZ*5CZR(xv$XSbfzu=gH}0f6M|=+@H5Hm?7h+H2|3dY$KXqdmG!Fw_ znYKR(RO<`MlfRepEnH~fMl;uPvvw4BN?2##v)mX)SM51A?ZE3)S472`vgI(hLjTX0 zf)C{e2n5DbFwN;KNnH-C&+ulO^Xf~6H*QJI`h^!Nyy(MoFI0qSh&{dbcWqoLX-Pu9 z!009kv8s3PERYtr`8FUJBBHknAN*cOS}#2I2m45i3HEG%p$IJdFO@3DPf=UoDDq7!(fZH9xBbMi>sP7nTfxc&oon6 z*K4~v;?CO>Z%mSiRKk2{^0$Drn*TN1~-417)N?u2UAv3nLxZJ*Zrrx2we>r3;uP=cnApPqRlz6NWW z9y@C}9D0PW-2c_XM^aBqlaQjVU<-y25HD!vX4{J?v3nh7u91}|Ix6HN zcAlN;txhT^oX9~;)CavD`qUs$_2rhL`ctJSC3*8M%*_8L?C50WaIQqa91jp9{s|>A z^@x51*di@0{KRCWu zZ|$vJt;(BKc`<90Pz3_LW;LDY+-YiBjBbH2FGqiaNX5Hi!(uPd52Dr3G>hH ztM(RtFgq9xT3!pVi!adMIjp@~zzo;YHeFb%FRY3vZw(UiU^VUiD=N6$Jkk_24+K>I zinF87+X2gB^&$aS3S0bFAN|vZua7KUGWwEKTnYyk#EhH^@cVusOHWYV|C*t>{4}~4 z`LP&w`53)Pc`vG5RUfrKzl0I2Zsz-Y zFO$3srFF_nqLoeoMS0Of8=jXUA4DY|eWBFmY6`ycDrVw%#HpY|IeQ#H696!bnh*pR zGHsbH+u{!Hhn+$m<)rTL+oB)p14QCDcoR)p#LQyE^e9(=6gj7P$_O;K*54`5LD}R| zZxxItQs>BPWjqsO0QRC^&4)bAG*LHGD4}QVH843E8R!Va0(I5M6R?IDMbbt7W|!!V z&JYP3I;yKtwTLr^uZ8;c^;X|`imkG$OdzAtb@yn?R*BJineW3l2T39rz2?iy1(`QR zr3b{>b0<8?Y$a!iA`B<3h9poQF;{l3@60f;!e*J%Bxc2N4?rs_8vf80f|p z5A)EE*;gL79a@uFs^d!gSDB||JXlx$VfXsm>3`!v8Re(;KL5$8Za7QOda|A7U3N@2 zQ~Cf5(&08Oi^|Hx&aio^W%M^_-9sMC&Ut}x!cl)a8B{(+o|IL`T0}uCZZ@nGW zsbXKj0MfazE4cn<()5!K^&+0BY>!sx?}5)3jtiP$eXcLpahpwvdNRM7GW7=l0e+A~ zR^Lh$+_Jggp@2zvoAk$hw*moa6h2AwEg1en7CK)RG@PE7lEs&4uc|n=qTqJ&)IoGK z2UT1a=ZrbHR?6!FpGh&Y(mh3sZg;u8CQugvuPyj$lt-(AP>b6q}r8O!neE^<8Nc9XjktIiF&4&yrTsC(!QkPx~r$ya3Nk|Jr)Nf zsmXp7sGE~lMeAdJ$)mezKs(n?(hQr?7l_NbF;bh=H}%{L z=-^1!p%<+VJ{_Ts#}_-?{46?FR-HJ>=mZ8UNw5FxN}|g%foMx<7@$fv1j(W#*nePs^xP$V1oC z%qymVTg3cy+#)0ZqFXeobM_4{fGAs1Qcu?)1F(XB)CZVItUQ%)CPbfZ98)RYTG#7k znkqw}wpFmRv~E$S{#k*Npo6*LlTl0&gX!iJ0|Mm#PKw?%T&yzG!4)&kC;kW7D)hh%Hf;?Yf@|1pzinQ z#anJ-0^bKt9qbJBa~;Hzms|?p3P6X^F8)d_uZDgVzBa{H=wMCH4Y<7QK$Xp90^hA? zxD~WABJCC85NGNELDmoU0vzC*cY2NwH1=upT?}w&_-{nOq&@S#% zlsW5~d)SWxf`TmtH*H2Q3~1}QeBmlfcSyMi;D|-wf1=>Nfik3uFe+#NNUw_%|8q{D z-MT+)P+MdX$E=b7MAhhjkyby1>oX3cSWX#N_0a%2j86LY!G^ zg+@6o&*-m9KT5jDYaz+B=8%1+Z;O)V*FF-HNJcI8ZM_F_3j zOa5xwW#)^GN8s}{iI>jew>Xia8l#XnAtuu`JhA|O+^qL9ZgSXb0SCz9nHi_a- zEwMBHHZ^Qk37QQyK3f&q0aU5usqlR4#snBh^iO;X3R$yGuS^7sOs42NjX+0-<^$BM zIBx%xxWw}hZKx~i+$XLn5qgYexVBFuB1=Vopt=w|QPdLi)&4gQ6ctLOes=_e0@eW#Ag$dW8F{>QQ1 z;-6X_8eC!2l{ZL8D#amnr^{b(>GB?|e{E%0@*P4@a%eOM#jfrQeMs;t+%GPzO~i&{ zuK71Pw+BxJ9SQs+P0pHsy>&te%a6O}AG}0ukJlv_deD9>{B8h0=kSoEq5Ein$%-ze zRc~tsbU^VY76+T@cf#1TirTW-23RUrP1m=50hXe>h#B39;D9p00m$Zh{Q_d>(?Ccfa@ACZAj#S zD?J)zx*lmPk~QUun5t|}@t;6X&C#pnmNDU;Nz~H~`yTKKJE6Xurxe}P?bka=C9yQc zV$>s1hMptTa&!7r`Sl$yQtyrxwzb|e-5!UF=TIRKw{Qk`SS=dMzK6}@!&TBsj{(%l zGy6_g&5&~G;*kUD=T8$ETXr><39&hS(scva`T6%v)x6n=lg56^OVi@leI+GjgAtJI zUsF!0aif{`qupFZ>$w_(i|er#4;gwvMyvqp27w%#SK8ss;?nv1k9%k$B$=GG*R-R9 zcoe$im9|cs);`#sO%pLZOVDMzGE!2KZx9L-!r8qKw37jJLGp+0R`?UF8RiOs?O z=T)l%jM%-GI|{7@sPDlA7TG@E)txJ%nVavSlm3 z;l2=bs+*kzbvDPF%af92K3xS@35Tl}EnT1PNu?M2Y~a_|J67dw!R0fq3eS>?8?)+Q z))F~Hu9=hff v1OA=OeR^-xo*dv^-|IMiwwoSSDlK-0Hy|w8JN7&UPY|*>Bs}