Merge remote-tracking branch 'upstream/develop' into develop

This commit is contained in:
Romain Dorgueil
2017-10-03 07:35:30 +02:00
3 changed files with 23 additions and 19 deletions

View File

@ -1,44 +1,48 @@
Environmental Variables Environment Variables
======================= =======================
Best practice holds that variables should be passed to graphs via environmental variables. Best practice holds that variables should be passed to graphs via environment variables.
Doing this is important for keeping sensitive data out of the code - such as an Doing this is important for keeping sensitive data out of the code - such as an
API token or username and password used to access a database. Not only is this API token or username and password used to access a database. Not only is this
approach more secure, it also makes graphs more flexible by allowing adjustments approach more secure, it also makes graphs more flexible by allowing adjustments
for a variety of environments and contexts. Importantly, environmental variables for a variety of environments and contexts. Importantly, environment variables
are also the means by-which arguments can be passed to graphs. are also the means by-which arguments can be passed to graphs.
Passing / Setting Environmental Variables Passing / Setting Environment Variables
:::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::::::::::
The recommended way to set environmental variables for a given graph is simply to use Setting environment variables for your graphs to use can be done in a variety of ways and which one used can vary
the optional ``--env`` argument when running bonobo from the shell (bash, command prompt, etc). based-upon context. Perhaps the most immediate and simple way to set/override a variable for a given graph is
simply to use the optional ``--env`` argument when running bonobo from the shell (bash, command prompt, etc).
``--env`` (or ``-e`` for short) should then be followed by the variable name and value using the ``--env`` (or ``-e`` for short) should then be followed by the variable name and value using the
syntax `VAR_NAME=VAR_VALUE`. Multiple environmental variables can be passed by using syntax ``VAR_NAME=VAR_VALUE``. Multiple environment variables can be passed by using multiple ``--env`` / ``-e`` flags (i.e. ``bonobo run --env FIZZ=buzz ...`` and ``bonobo run --env FIZZ=buzz --env Foo=bar ...``). Additionally, in bash you can also set environment variables by listing those you wish to set before the `bonobo run` command with space separating the key-value pairs (i.e. ``FIZZ=buzz bonobo run ...`` or ``FIZZ=buzz FOO=bar bonobo run ...``).
multiple ``--env`` / ``-e`` flags.
Example: The Examples below demonstrate setting one or multiple variables using both of these methods:
.. code-block:: bash .. code-block:: bash
# Using one environmental variable: # Using one environment variable via --env flag:
bonobo run csvsanitizer --env SECRET_TOKEN=secret123 bonobo run csvsanitizer --env SECRET_TOKEN=secret123
# Using multiple environmental variables: # Using multiple environment variables via -e (env) flag:
bonobo run csvsanitizer -e SRC_FILE=inventory.txt -e DST_FILE=inventory_processed.csv bonobo run csvsanitizer -e SRC_FILE=inventory.txt -e DST_FILE=inventory_processed.csv
If you're naming something which is configurable, that is will need to be instantiated or called to obtain something that # Using one environment variable in bash (*bash only):
can be used as a graph node, then use camelcase names: SECRET_TOKEN=secret123 bonobo run csvsanitizer
# Using multiple environment variables in bash (*bash only):
SRC_FILE=inventory.txt DST_FILE=inventory_processed.csv bonobo run csvsanitizer
Accessing Environmental Variables from within the Graph Context *Though not-yet implemented, the bonobo roadmap includes implementing environment / .env files as well.*
Accessing Environment Variables from within the Graph Context
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Environmental variables, whether global or only for the scope of the graph, Environment variables, whether set globally or only for the scope of the graph,
can be can be accessed using any of the normal means. It is important to note can be can be accessed using any of the normal means. It is important to note
that whether set globally for the system or just for the graph context, that whether set globally for the system or just for the graph context,
environmental variables are accessed by bonobo in the same way. In the example environment variables are accessed by bonobo in the same way. In the example
below the database user and password are accessed via the ``os`` module's ``getenv`` below the database user and password are accessed via the ``os`` module's ``getenv``
function and used to get data from the database. function and used to get data from the database.

View File

@ -1,6 +1,6 @@
Bonobo with Jupyter Bonobo with Jupyter
=================== ===================
minimalistically
There is a builtin plugin that integrates (somewhat minimallistically, for now) bonobo within jupyter notebooks, so There is a builtin plugin that integrates (somewhat minimallistically, for now) bonobo within jupyter notebooks, so
you can read the execution status of a graph within a nice (ok, not so nice) html/javascript widget. you can read the execution status of a graph within a nice (ok, not so nice) html/javascript widget.

View File

@ -12,7 +12,7 @@ There are a few things that you should know while writing transformations graphs
purity purity
transformations transformations
services services
envrionmental_variables envrionment_variables
Third party integrations Third party integrations
:::::::::::::::::::::::: ::::::::::::::::::::::::