Can I know the difference between data backfilling and refreshing, in the context of BigQuery Data Transfers.

Data Refresh window is a parameter that can be configured when setting up BigQuery Data Transfers for Google Ads and Search Ads 360 data (probably for some other BigQuery data transfers too).

Reference:
https://cloud.google.com/bigquery-transfer/docs/working-with-transfers#refreshing https://cloud.google.com/bigquery-transfer/docs/adwords-transfer

Tag:google-ads-api, google-bigquery

4 comments.

  1. ewertonvsilva

    -Data transfer:

    After you configure a data transfer, the BigQuery Data Transfer Service automatically loads data into BigQuery on a regular basis.

    I.e fetch data from last 24 hours to you BQ dataset.

    -Backfills:

    You can also initiate data backfills to recover from any outages or gaps.

    I.e fetch data gaps, insert new lines that wasn't imported in the data transfer.

    -Refresh: Overwrites a fresh copy of data in the destination table.

    source:

    Cloud Storage transfers  |  BigQuery Data Transfer Service  |  Google Cloud

    What is BigQuery Data Transfer Service?  |  Google Cloud

    Reply
    1. user1502505

      Thanks @ewertonvsilva. For Refresh if the Refresh Window is 7 days, does it mean that on a daily basis the last 7 day data gets continuously (as a sliding window of 7 days) I guess the idea is to capture any late recorded data?

      Reply
    2. ewertonvsilva

      Exactly, your understanding is right. The sliding window will visit the last 7 days and update them with the last data for those days.

      Reply
  2. user1502505

    Further references on RefreshWindow :

    https://cloud.google.com/bigquery-transfer/docs/reference/datatransfer/rest/v1/projects.locations.transferConfigs#TransferConfig

    Reply

Add a new comment.