Merge pull request #1 from python-bonobo/master

Upstream
This commit is contained in:
Obiamaka Agbaneje
2018-07-28 14:23:34 +01:00
committed by GitHub
3 changed files with 58 additions and 6 deletions

View File

@ -36,6 +36,25 @@ The `sqlalchemy.engine` name is the default name used by the provided transforma
example if you need more than one connection) and specify the service name using `engine='myengine'` while building your
transformations.
Lets create some tables and add some data. (You may need to edit the SQL if your database server uses a different
version of SQL.)
.. code-block:: sql
CREATE TABLE test_in (
id INTEGER PRIMARY KEY NOT NULL,
text TEXT
);
CREATE TABLE test_out (
id INTEGER PRIMARY KEY NOT NULL,
text TEXT
);
INSERT INTO test_in (id, text) VALUES (1, 'Cat');
INSERT INTO test_in (id, text) VALUES (2, 'Dog');
There are two transformation classes provided by this extension.
One reader, one writer.
@ -50,12 +69,29 @@ Let's select some data:
def get_graph():
graph = bonobo.Graph()
graph.add_chain(
bonobo_sqlalchemy.Select('SELECT * FROM example', limit=100),
bonobo_sqlalchemy.Select('SELECT * FROM test_in', limit=100),
bonobo.PrettyPrinter(),
)
return graph
And let's insert some data:
You should see:
.. code-block:: shell-session
$ python tutorial.py
│ id[0] = 1
│ text[1] = 'Cat'
│ id[0] = 2
│ text[1] = 'Dog'
- Select in=1 out=2 [done]
- PrettyPrinter in=2 out=2 [done]
Now let's insert some data:
.. code-block:: python
@ -66,12 +102,14 @@ And let's insert some data:
def get_graph(**options):
graph = bonobo.Graph()
graph.add_chain(
...,
bonobo_sqlalchemy.InsertOrUpdate('example')
bonobo_sqlalchemy.Select('SELECT * FROM test_in', limit=100),
bonobo_sqlalchemy.InsertOrUpdate('test_out')
)
return graph
If you check the `test_out` table, it should now have the data.
Reference
:::::::::

View File

@ -41,7 +41,7 @@ Now, we need to write a `writer` transformation, and apply this context processo
@use_context_processor(with_opened_file)
def write_repr_to_file(f, *row):
f.write(repr(row))
f.write(repr(row) + "\n")
The `f` parameter will contain the value yielded by the context processors, in order of appearance (you can chain
multiple context processors).
@ -50,6 +50,20 @@ Please note that the :func:`bonobo.config.use_context_processor` decorator will
modify its behaviour. If you want to call it out of the |bonobo| job context, it's your responsibility to provide
the right parameters (and here, the opened file).
To run this, change the last stage in the pipeline in get_graph to write_repr_to_file
.. code-block:: python
def get_graph(**options):
graph = bonobo.Graph()
graph.add_chain(
extract_fablabs,
bonobo.Limit(10),
write_repr_to_file,
)
return graph
Now run tutorial.py and check the output.txt file.
Using the filesystem
::::::::::::::::::::

View File

@ -3,7 +3,7 @@ Part 4: Services
All external dependencies (like filesystems, network clients, database connections, etc.) should be provided to
transformations as a service. It allows great flexibility, including the ability to test your transformations isolated
from the external world, and being friendly to the infrastructure guys (and if you're one of them, it's also nice to
from the external world, and being friendly to the infrastructure people (and if you're one of them, it's also nice to
treat yourself well).
In the last section, we used the `fs` service to access filesystems, we'll go even further by switching our `requests`