Collecting high-volume customer feedback, lead qualification answers, and product onboarding surveys requires instant streaming to a data warehouse. While batch connectors like Fivetran or Stitch introduce 15-minute synchronization delays and steep per-row billing charges, streaming Typeform responses directly into Google BigQuery using Python Cloud Functions delivers sub-second analytics dashboards, zero row-connector markups, and complete schema flexibility.
In this technical implementation guide, we construct a production-ready Google Cloud Function (Gen 2) in Python that verifies Typeform webhook HMAC SHA-256 signatures, extracts complex nested answer arrays, handles schema evolution via BigQuery JSON columns, and streams records in real time.
1. Real-Time Streaming Architecture
| Phase | Technology | Latency & SLA |
|---|---|---|
| 1. Form Submission | Typeform Engine | Dispatches webhook within 200ms of user submission. |
| 2. Ingestion Handler | GCP Cloud Function (Python 3.11, Gen 2) | Validates HMAC signature and parses nested survey structures (<40ms). |
| 3. Data Warehouse Streaming | Google BigQuery Storage Write API | Direct streaming insertion with immediate SQL query availability. |
2. Production Python Cloud Function (main.py)
# main.py - Typeform to BigQuery Streaming Ingestion
import os
import json
import hmac
import hashlib
from google.cloud import bigquery
import functions_framework
bq_client = bigquery.Client()
DATASET_ID = os.environ.get("BIGQUERY_DATASET", "analytics_ingest")
TABLE_ID = os.environ.get("BIGQUERY_TABLE", "typeform_responses")
TYPEFORM_SECRET = os.environ.get("TYPEFORM_WEBHOOK_SECRET", "")
def verify_signature(payload_bytes: bytes, signature_header: str) -> bool:
# Validates Typeform sha256 HMAC signature.
if not TYPEFORM_SECRET or not signature_header:
return True # Bypass if secret is unconfigured
expected = "sha256=" + hmac.new(
TYPEFORM_SECRET.encode(), payload_bytes, hashlib.sha256
).hexdigest()
return hmac.compare_digest(expected, signature_header)
@functions_framework.http
def handle_typeform_webhook(request):
if request.method != "POST":
return ("Method Not Allowed", 405)
raw_body = request.get_data()
sig_header = request.headers.get("Typeform-Signature")
if not verify_signature(raw_body, sig_header):
return ("Unauthorized: Invalid Webhook Signature", 401)
try:
data = json.loads(raw_body)
form_response = data.get("form_response", {})
# Parse survey answers into a clean key-value lookup
extracted_answers = {}
for ans in form_response.get("answers", []):
field_id = ans.get("field", {}).get("id")
ans_type = ans.get("type")
# Extract primitive based on answer type
val = ans.get(ans_type)
if isinstance(val, dict):
val = val.get("label") or val.get("id")
extracted_answers[field_id] = val
row = [{
"event_id": data.get("event_id"),
"form_id": form_response.get("form_id"),
"token": form_response.get("token"),
"submitted_at": form_response.get("submitted_at"),
"answers_json": json.dumps(extracted_answers),
"raw_payload": json.dumps(data)
}]
table_ref = bq_client.dataset(DATASET_ID).table(TABLE_ID)
errors = bq_client.insert_rows_json(table_ref, row)
if errors:
print(f"BigQuery Insert Errors: {errors}")
return (f"Insertion Failed: {errors}", 500)
return (json.dumps({"status": "streamed"}), 200, {"Content-Type": "application/json"})
except Exception as e:
print(f"Pipeline Exception: {e}")
return (str(e), 500)
3. BigQuery Table Schema (SQL DDL)
CREATE TABLE `your_project.analytics_ingest.typeform_responses` (
event_id STRING NOT NULL,
form_id STRING NOT NULL,
token STRING NOT NULL,
submitted_at TIMESTAMP NOT NULL,
answers_json JSON NOT NULL,
raw_payload JSON NOT NULL,
inserted_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP()
)
PARTITION BY DATE(submitted_at)
CLUSTER BY form_id;
Frequently Asked Questions (FAQs)
What is the cost of streaming directly into BigQuery?
BigQuery streaming insert charges $0.01 per 200MB of data. For 100,000 survey responses per month, streaming data warehouse costs are literally under $0.05 per month, saving thousands compared to dedicated ETL SaaS pipelines.
Editorial Disclosure: Zelobiz publishes independent engineering benchmarks and architectural evaluations. We may receive affiliate compensation when you subscribe to partner hosting services through our links at no additional cost to you.