@ -36,6 +36,25 @@ The `sqlalchemy.engine` name is the default name used by the provided transforma
|
|||||||
example if you need more than one connection) and specify the service name using `engine='myengine'` while building your
|
example if you need more than one connection) and specify the service name using `engine='myengine'` while building your
|
||||||
transformations.
|
transformations.
|
||||||
|
|
||||||
|
Lets create some tables and add some data. (You may need to edit the SQL if your database server uses a different
|
||||||
|
version of SQL.)
|
||||||
|
|
||||||
|
.. code-block:: sql
|
||||||
|
|
||||||
|
CREATE TABLE test_in (
|
||||||
|
id INTEGER PRIMARY KEY NOT NULL,
|
||||||
|
text TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE test_out (
|
||||||
|
id INTEGER PRIMARY KEY NOT NULL,
|
||||||
|
text TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
INSERT INTO test_in (id, text) VALUES (1, 'Cat');
|
||||||
|
INSERT INTO test_in (id, text) VALUES (2, 'Dog');
|
||||||
|
|
||||||
|
|
||||||
There are two transformation classes provided by this extension.
|
There are two transformation classes provided by this extension.
|
||||||
|
|
||||||
One reader, one writer.
|
One reader, one writer.
|
||||||
@ -50,12 +69,29 @@ Let's select some data:
|
|||||||
def get_graph():
|
def get_graph():
|
||||||
graph = bonobo.Graph()
|
graph = bonobo.Graph()
|
||||||
graph.add_chain(
|
graph.add_chain(
|
||||||
bonobo_sqlalchemy.Select('SELECT * FROM example', limit=100),
|
bonobo_sqlalchemy.Select('SELECT * FROM test_in', limit=100),
|
||||||
bonobo.PrettyPrinter(),
|
bonobo.PrettyPrinter(),
|
||||||
)
|
)
|
||||||
|
return graph
|
||||||
|
|
||||||
And let's insert some data:
|
You should see:
|
||||||
|
|
||||||
|
.. code-block:: shell-session
|
||||||
|
|
||||||
|
$ python tutorial.py
|
||||||
|
┌
|
||||||
|
│ id[0] = 1
|
||||||
|
│ text[1] = 'Cat'
|
||||||
|
└
|
||||||
|
┌
|
||||||
|
│ id[0] = 2
|
||||||
|
│ text[1] = 'Dog'
|
||||||
|
└
|
||||||
|
- Select in=1 out=2 [done]
|
||||||
|
- PrettyPrinter in=2 out=2 [done]
|
||||||
|
|
||||||
|
|
||||||
|
Now let's insert some data:
|
||||||
|
|
||||||
.. code-block:: python
|
.. code-block:: python
|
||||||
|
|
||||||
@ -66,12 +102,14 @@ And let's insert some data:
|
|||||||
def get_graph(**options):
|
def get_graph(**options):
|
||||||
graph = bonobo.Graph()
|
graph = bonobo.Graph()
|
||||||
graph.add_chain(
|
graph.add_chain(
|
||||||
...,
|
bonobo_sqlalchemy.Select('SELECT * FROM test_in', limit=100),
|
||||||
bonobo_sqlalchemy.InsertOrUpdate('example')
|
bonobo_sqlalchemy.InsertOrUpdate('test_out')
|
||||||
)
|
)
|
||||||
|
|
||||||
return graph
|
return graph
|
||||||
|
|
||||||
|
If you check the `test_out` table, it should now have the data.
|
||||||
|
|
||||||
Reference
|
Reference
|
||||||
:::::::::
|
:::::::::
|
||||||
|
|
||||||
|
|||||||
@ -41,7 +41,7 @@ Now, we need to write a `writer` transformation, and apply this context processo
|
|||||||
|
|
||||||
@use_context_processor(with_opened_file)
|
@use_context_processor(with_opened_file)
|
||||||
def write_repr_to_file(f, *row):
|
def write_repr_to_file(f, *row):
|
||||||
f.write(repr(row))
|
f.write(repr(row) + "\n")
|
||||||
|
|
||||||
The `f` parameter will contain the value yielded by the context processors, in order of appearance (you can chain
|
The `f` parameter will contain the value yielded by the context processors, in order of appearance (you can chain
|
||||||
multiple context processors).
|
multiple context processors).
|
||||||
@ -50,6 +50,20 @@ Please note that the :func:`bonobo.config.use_context_processor` decorator will
|
|||||||
modify its behaviour. If you want to call it out of the |bonobo| job context, it's your responsibility to provide
|
modify its behaviour. If you want to call it out of the |bonobo| job context, it's your responsibility to provide
|
||||||
the right parameters (and here, the opened file).
|
the right parameters (and here, the opened file).
|
||||||
|
|
||||||
|
To run this, change the last stage in the pipeline in get_graph to write_repr_to_file
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
def get_graph(**options):
|
||||||
|
graph = bonobo.Graph()
|
||||||
|
graph.add_chain(
|
||||||
|
extract_fablabs,
|
||||||
|
bonobo.Limit(10),
|
||||||
|
write_repr_to_file,
|
||||||
|
)
|
||||||
|
return graph
|
||||||
|
|
||||||
|
Now run tutorial.py and check the output.txt file.
|
||||||
|
|
||||||
Using the filesystem
|
Using the filesystem
|
||||||
::::::::::::::::::::
|
::::::::::::::::::::
|
||||||
|
|||||||
@ -3,7 +3,7 @@ Part 4: Services
|
|||||||
|
|
||||||
All external dependencies (like filesystems, network clients, database connections, etc.) should be provided to
|
All external dependencies (like filesystems, network clients, database connections, etc.) should be provided to
|
||||||
transformations as a service. It allows great flexibility, including the ability to test your transformations isolated
|
transformations as a service. It allows great flexibility, including the ability to test your transformations isolated
|
||||||
from the external world, and being friendly to the infrastructure guys (and if you're one of them, it's also nice to
|
from the external world, and being friendly to the infrastructure people (and if you're one of them, it's also nice to
|
||||||
treat yourself well).
|
treat yourself well).
|
||||||
|
|
||||||
In the last section, we used the `fs` service to access filesystems, we'll go even further by switching our `requests`
|
In the last section, we used the `fs` service to access filesystems, we'll go even further by switching our `requests`
|
||||||
|
|||||||
Reference in New Issue
Block a user