Dataflow Failing To Push Messages To Bigquery From Pubsub

I am trying to now work a data pipeline. I am using the Python client library to insert the record into PubSub. From there DataFlow is supposed to pick it up and then push into BQ.

Solution 1:

Why do you use data = base64.b64encode(message)? What is message exactly?

I tried this snippet with Pub/Sub to BigQuery provided Dataflow template, and it works:

defpublish_messages(project, topic_name):
    """Publishes multiple messages to a Pub/Sub topic."""
    publisher = pubsub_v1.PublisherClient()
    topic_path = publisher.topic_path(project, topic_name)

    for n inrange(1, 6):
        data = u'{"column1": "value1","column2": "value2"}'# Data must be a bytestring
        data = data.encode('utf-8')
        publisher.publish(topic_path, data=data)

    print('Published messages.')

Try this without encoding in base64.

[Pub/Sub python code] [Dataflow templates]