Downloaders: Customizing the download

By default, pooch.Pooch.fetch and pooch.retrieve will detect the download protocol from the given URL (HTTP, FTP, SFTP, DOI) and use the appropriate download method. Sometimes this is not enough: some servers require logins, redirections, or other non-standard operations. To get around this, use the downloader argument of fetch and retrieve.

Downloaders are Python callable objects (like functions or classes with a __call__ method) and must have the following format:

def mydownloader(url, output_file, pooch):
    '''
    Download a file from the given URL to the given local file.

    The function **must** take as arguments (in order):

    url : str
        The URL to the file you want to download.
    output_file : str or file-like object
        Path (and file name) to which the file will be downloaded.
    pooch : pooch.Pooch
        The instance of the Pooch class that is calling this function.

    No return value is required.
    '''
    ...

Pooch provides downloaders for HTTP, FTP, and SFTP that support authentication and optionally printing progress bars. See List of functions and classes (API) for a list of available downloaders.

Common uses of downloaders include:

Creating your own downloaders

If your use case is not covered by our downloaders, you can implement your own. pooch.Pooch.fetch and pooch.retrieve will accept any callable obejct that has the signature specified above. As an example, consider the case in which the login credentials need to be provided to a site that is redirected from the original download URL:

import requests


def redirect_downloader(url, output_file, pooch):
    """
    Download after following a redirection.
    """
    # Get the credentials from the user's environment
    username = os.environ.get("SOMESITE_USERNAME")
    password = os.environ.get("SOMESITE_PASSWORD")
    # Make a request that will redirect to the login page
    login = requests.get(url)
    # Provide the credentials and download from the new URL
    download = HTTPDownloader(auth=(username, password))
    download(login.url, output_file, mypooch)


def fetch_protected_data():
    """
    Fetch a file from a server that requires authentication
    """
    fname = GOODBOY.fetch("some-data.csv", downloader=redirect_downloader)
    data = pandas.read_csv(fname)
    return data