52 Entertainment version of GA4 including 2 changes
Description & motivation
Merge PR between PR188 and PR231
PR188 - Refactor daily & streaming support
Previously, this package supported daily + streaming (aka intraday) exports in a relatively naïve way by incrementally processing 'daily' data and then unioning the 'streaming' data as a view in the
stg_ga4__events
model. This can cause issues when tables are rotated from 'intraday' to 'streaming' as noted in Velir#80 and in other cases noted in Velir#182.This update ensures that daily & streaming data are handled using the same method: dats is loaded incrementally into the base model and deduped before being loaded into
stg_ga4__events
.As a byproduct, this PR removes support for 'dynamically' incremental processing based on the last date found in a table using
_max_partition
. Instead, thestatic_incremental_days
variable will need to be used which will ensure that data is reprocessed as the intraday table changes throughout the day.
PR231 - Implementing multi-project architecture ( issue Velir#231 )
Updating the
dbt_projects.yml
format to specify the GCPproject
and GA4property_id
.Use case: when sources are spread on different distinct GCP
Projects
. Related to Velir#231Note: This new config format can allow for a more specific configuration (example if property
11111111
is on daily frequency while property22222222
is on streaming import this could be implemented in this way too in the future).