How To Continously Update Target File Using Luigi?

I have recently started playing around with Luigi, and I would like to find out how to use it to continuously append new data into an existing target file. Imagine I am pinging an

Solution 1:

You cannot. As the doc for LocalTarget says:

Parameters: mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options.

I.e. only r or w modes are allowed. Additional options such as a require an extension of the LocalTarget class; despite it breaks the desired idempotency on Luigi task executions.

Solution 2:

def output(self):
        return luigi.LocalTarget("data_test_%s.json" % self.date.strftime("%Y-%m-%d_%H:%M"))

It's not the 'luigi way', but it does the job. In the end those targets are just file objects.