You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
Johann Schmitz 7d09eef19c
Use SSL everywhere
3 weeks ago
.vscode Add .vscode settings 1 year ago
.dockerignore Add dockerignore and ignore dev docker-compose folders. 1 year ago
.gitignore Stop tracking changes to .vscode settings 1 year ago
.gitlab-ci.yml Add .gitlab-ci.yml 1 year ago
.pylintrc Add proper vscode settings 1 year ago
Dockerfile Add fitbit_export_loader.py script, add it to docker image, write README for it. 1 year ago
README.md Add Archive data export dashboard, unify name everywhere 1 year ago
api_poller.py Use SSL everywhere 3 weeks ago
docker-compose-dev.yml Move docker-compose to docker-compose-dev 1 year ago
docker-compose.yml Add sample of production docker-compose 1 year ago
fitbit_export_loader.py Add fitbit_export_loader.py script, add it to docker image, write README for it. 1 year ago
influxdb.conf Provide config file for influxdb and mount it in container, to enable tsi1 index. 1 year ago

README.md

Fitbit API exporter

The project is a simple script to export data from Fitbit's Web APIs to a custom InfluxDB database, and then graph it via Grafana.
Everything is made easier via docker and docker-compose. Grafana dashboards are available at:

Quick setup (API exporter)

Notes:

  • basic knowledge of docker is assumed
  • unless otherwise noted, leave fields at default
  • the script sources environment variables if missing, otherwise sources the saved config data (this is because the token gets refreshed automatically thanks to the refresh token)
  • if you don't run the scripts on a server, you may need to refresh the access token and refresh token at the next run, because the refresh token may have already expired (see below)

Step-by-step guide:

  1. Do a test-run of the docker-compose.yml provided, and customize it to your liking, ensure the containers come up, esp. the InfluxDB one
  2. Go at Fitbit dev login, login with your account
  3. Register an application: click on "Register an app" at the top
    • Application name and description to your liking
    • Application website, Organization website, terms of service, privacy policy, callback URL, you can all set "http://localhost:8080/"
    • OAuth 2.0 Application Type: Personal
    • Default Access Type: Read-Only
  4. Get first two parameters: click on "Manage my apps", click on your new application; you will need to note down the following in order to fill corresponding env vars:
    • OAuth 2.0 Client ID: CLIENT_ID
    • Client Secret: CLIENT_SECRET
    • Callback URL: CALLBACK_URL
  5. Generate the tokens: click on the small link at the bottom of the page "OAuth 2.0 tutorial page"
    1. Select "Flow type": "Authorization Code Flow"
    2. Select Scopes: activity, heartrate, profile, settings, sleep, weight
    3. Click on the link at the end of section 1 "We've generate the authorization URL for you, all you need to do is just click on the link below:"
    4. Select all scopes and click "Allow"
    5. Copy the code parameter from the URL of the window that opens: code=[.....]#=
    6. Paste it in the "1A Get Code" form
    7. Copy the curl call to a script, remove newlines, execute the script
    8. Copy the JSON output from the script and paste into the "2: Parse response" section
    9. Note down:
      • Access token: ACCESS_TOKEN
      • Refresh Token: REFRESH_TOKEN
  6. You can fill the provided docker-compose.yml with the parameters obtained
  7. Pull up the containers

Quick setup (Archive export loader)

  1. Navigate to www.fitbit.com, access with your credentials
  2. Go to "Account settings", then "Data Export"
  3. Select "Export account archive" (or similar - the whole of data available)
  4. Confirm the export via the export email
  5. Wait for the data to be available
  6. Download the data
  7. Extract the archive
  8. Pull up the influxdb container: docker-compose up -d influxdb
  9. Run the docker image: docker run -it --network fitbit-exporter -v<folder_containing_extracted_archive>:/dump -eDB_HOST=... -eDB_PORT=... -eDB_USER=... -eDB_PASSWORD=... -eDB_NAME=... registry.gitlab.com/fsvm88/fitbit-api-exporter:latest /fitbit_export_loader.py (recheck the network name, as it depends on docker-compose)
  10. Wait for the container to finish

The command can be rerun as required in case of errors, subsequent runs should skip already-written datapoints using the timestamps of the entries.

NOTE: the data folder must be provided at user-site-export parent level (e.g: /path/to/folder/username)

Limitations

Fitbit APIs

  1. Fitbit APIs are limited to 150calls/hour, the script will detect this and sleep for 1h 10s
  2. The script will sleep for 4h when there's no additional data to fetch, and only fetch the last day at each round
  3. Support for older responses of the APIs is limited/non-existing

Intra-day time series

Intra-day time series are not implemented.
I originally wanted to add them, but saw little benefit after finishing the normal time-series.
One additional issue with intra-day series is that they require additional setup on Fitbit's side, and you may need to request access to those via Fitbit support. I have not tried contacting them because I had no need for it.
One last note is that intra-day data is much more granular (1m/30s/1s intervals), so each query will return a lot more data to download and store, which I had no need for.

Development

api_poller.py is all there is to it, use docker-compose-dev.yml to pull up a local instance for testing

Deps

Python script, developed with: