Update stable doc with warnings.
This commit is contained in:
9
docs/tutorial/0.5/_outdated_note.rst
Normal file
9
docs/tutorial/0.5/_outdated_note.rst
Normal file
@ -0,0 +1,9 @@
|
|||||||
|
.. warning::
|
||||||
|
|
||||||
|
This tutorial was written for |bonobo| 0.5, while the current stable version is |bonobo| 0.6.
|
||||||
|
|
||||||
|
Please be aware that some things changed.
|
||||||
|
|
||||||
|
A summary of changes is available in the `migration guide from 0.5 to 0.6 <https://news.bonobo-project.org/migration-guide-for-bonobo-0-6-alpha-c1d36b0a9d35>`_.
|
||||||
|
|
||||||
|
|
||||||
65
docs/tutorial/0.5/index.rst
Normal file
65
docs/tutorial/0.5/index.rst
Normal file
@ -0,0 +1,65 @@
|
|||||||
|
First steps
|
||||||
|
===========
|
||||||
|
|
||||||
|
.. include:: _outdated_note.rst
|
||||||
|
|
||||||
|
What is Bonobo?
|
||||||
|
:::::::::::::::
|
||||||
|
|
||||||
|
Bonobo is an ETL (Extract-Transform-Load) framework for python 3.5. The goal is to define data-transformations, with
|
||||||
|
python code in charge of handling similar shaped independent lines of data.
|
||||||
|
|
||||||
|
Bonobo *is not* a statistical or data-science tool. If you're looking for a data-analysis tool in python, use Pandas.
|
||||||
|
|
||||||
|
Bonobo is a lean manufacturing assembly line for data that let you focus on the actual work instead of the plumbery
|
||||||
|
(execution contexts, parallelism, error handling, console output, logging, ...).
|
||||||
|
|
||||||
|
Bonobo uses simple python and should be quick and easy to learn.
|
||||||
|
|
||||||
|
Tutorial
|
||||||
|
::::::::
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
Good documentation is not easy to write. We do our best to make it better and better.
|
||||||
|
|
||||||
|
Although all content here should be accurate, you may feel a lack of completeness, for which we plead guilty and
|
||||||
|
apologize.
|
||||||
|
|
||||||
|
If you're stuck, please come and ask on our `slack channel <https://bonobo-slack.herokuapp.com/>`_, we'll figure
|
||||||
|
something out.
|
||||||
|
|
||||||
|
If you're not stuck but had trouble understanding something, please consider contributing to the docs (via GitHub
|
||||||
|
pull requests).
|
||||||
|
|
||||||
|
.. toctree::
|
||||||
|
:maxdepth: 2
|
||||||
|
|
||||||
|
tut01
|
||||||
|
tut02
|
||||||
|
tut03
|
||||||
|
tut04
|
||||||
|
|
||||||
|
|
||||||
|
What's next?
|
||||||
|
::::::::::::
|
||||||
|
|
||||||
|
Read a few examples
|
||||||
|
-------------------
|
||||||
|
|
||||||
|
* :doc:`/reference/examples`
|
||||||
|
|
||||||
|
Read about best development practices
|
||||||
|
-------------------------------------
|
||||||
|
|
||||||
|
* :doc:`/guide/index`
|
||||||
|
* :doc:`/guide/purity`
|
||||||
|
|
||||||
|
Read about integrating external tools with bonobo
|
||||||
|
-------------------------------------------------
|
||||||
|
|
||||||
|
* :doc:`/extension/docker`: run transformation graphs in isolated containers.
|
||||||
|
* :doc:`/extension/jupyter`: run transformations within jupyter notebooks.
|
||||||
|
* :doc:`/extension/selenium`: crawl the web using a real browser and work with the gathered data.
|
||||||
|
* :doc:`/extension/sqlalchemy`: everything you need to interract with SQL databases.
|
||||||
|
|
||||||
13
docs/tutorial/0.5/python.rst
Normal file
13
docs/tutorial/0.5/python.rst
Normal file
@ -0,0 +1,13 @@
|
|||||||
|
Just enough Python for Bonobo
|
||||||
|
=============================
|
||||||
|
|
||||||
|
.. include:: _outdated_note.rst
|
||||||
|
|
||||||
|
.. todo::
|
||||||
|
|
||||||
|
This is a work in progress and it is not yet available. Please come back later or even better, help us write this
|
||||||
|
guide!
|
||||||
|
|
||||||
|
This guide is intended to help programmers or enthusiasts to grasp the python basics necessary to use Bonobo. It
|
||||||
|
should definately not be considered as a general python introduction, neither a deep dive into details.
|
||||||
|
|
||||||
@ -1,7 +1,10 @@
|
|||||||
Let's get started!
|
Let's get started!
|
||||||
==================
|
==================
|
||||||
|
|
||||||
To get started with Bonobo, you need to install it in a working python 3.5+ environment:
|
.. include:: _outdated_note.rst
|
||||||
|
|
||||||
|
To begin with Bonobo, you need to install it in a working python 3.5+ environment, and you'll also need cookiecutter
|
||||||
|
to bootstrap your project.
|
||||||
|
|
||||||
.. code-block:: shell-session
|
.. code-block:: shell-session
|
||||||
|
|
||||||
@ -13,24 +16,21 @@ See :doc:`/install` for more options.
|
|||||||
Create an empty project
|
Create an empty project
|
||||||
:::::::::::::::::::::::
|
:::::::::::::::::::::::
|
||||||
|
|
||||||
Your ETL code will live in standard python files and packages.
|
Your ETL code will live in ETL projects, which are basically a bunch of files, including python code, that bonobo
|
||||||
|
can run.
|
||||||
|
|
||||||
.. code-block:: shell-session
|
.. code-block:: shell-session
|
||||||
|
|
||||||
$ bonobo create tutorial.py
|
$ bonobo init tutorial
|
||||||
|
|
||||||
This will create a simple example job in a `tutorial.py` file.
|
This will create a `tutorial` directory (`content description here <https://www.bonobo-project.org/with/cookiecutter>`_).
|
||||||
|
|
||||||
Now, try to execute it:
|
To run this project, use:
|
||||||
|
|
||||||
.. code-block:: shell-session
|
.. code-block:: shell-session
|
||||||
|
|
||||||
$ python tutorial.py
|
$ bonobo run tutorial
|
||||||
|
|
||||||
Congratulations, you just ran your first ETL job!
|
|
||||||
|
|
||||||
|
|
||||||
.. todo:: XXX **CHANGES NEEDED BELOW THIS POINTS BEFORE 0.6** XXX
|
|
||||||
|
|
||||||
Write a first transformation
|
Write a first transformation
|
||||||
::::::::::::::::::::::::::::
|
::::::::::::::::::::::::::::
|
||||||
@ -107,9 +107,6 @@ To do this, it needs to know what data-flow you want to achieve, and you'll use
|
|||||||
The `if __name__ == '__main__':` section is not required, unless you want to run it directly using the python
|
The `if __name__ == '__main__':` section is not required, unless you want to run it directly using the python
|
||||||
interpreter.
|
interpreter.
|
||||||
|
|
||||||
The name of the `graph` variable is arbitrary, but this variable must be global and available unconditionally.
|
|
||||||
Do not put it in its own function or in the `if __name__ == '__main__':` section.
|
|
||||||
|
|
||||||
|
|
||||||
Execute the job
|
Execute the job
|
||||||
:::::::::::::::
|
:::::::::::::::
|
||||||
@ -133,9 +130,9 @@ Rewrite it using builtins
|
|||||||
There is a much simpler way to describe an equivalent graph:
|
There is a much simpler way to describe an equivalent graph:
|
||||||
|
|
||||||
.. literalinclude:: ../../bonobo/examples/tutorials/tut01e02.py
|
.. literalinclude:: ../../bonobo/examples/tutorials/tut01e02.py
|
||||||
:language: python
|
:language: python
|
||||||
|
|
||||||
The `extract()` generator has been replaced by a list, as Bonobo will interpret non-callable iterables as a no-input
|
The `extract()` generator has been replaced by a list, as Bonobo will interpret non-callable iterables as a no-input
|
||||||
generator.
|
generator.
|
||||||
|
|
||||||
This example is also available in :mod:`bonobo.examples.tutorials.tut01e02`, and you can also run it as a module:
|
This example is also available in :mod:`bonobo.examples.tutorials.tut01e02`, and you can also run it as a module:
|
||||||
@ -1,6 +1,8 @@
|
|||||||
Working with files
|
Working with files
|
||||||
==================
|
==================
|
||||||
|
|
||||||
|
.. include:: _outdated_note.rst
|
||||||
|
|
||||||
Bonobo would be pointless if the aim was just to uppercase small lists of strings.
|
Bonobo would be pointless if the aim was just to uppercase small lists of strings.
|
||||||
|
|
||||||
In fact, Bonobo should not be used if you don't expect any gain from parallelization/distribution of tasks.
|
In fact, Bonobo should not be used if you don't expect any gain from parallelization/distribution of tasks.
|
||||||
@ -59,7 +61,13 @@ available in **Bonobo**'s repository:
|
|||||||
|
|
||||||
.. code-block:: shell-session
|
.. code-block:: shell-session
|
||||||
|
|
||||||
$ bonobo download examples/datasets/coffeeshops.txt
|
$ curl https://raw.githubusercontent.com/python-bonobo/bonobo/master/bonobo/examples/datasets/coffeeshops.txt > `python3 -c 'import bonobo; print(bonobo.get_examples_path("datasets/coffeeshops.txt"))'`
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
The "example dataset download" step will be easier in the future.
|
||||||
|
|
||||||
|
https://github.com/python-bonobo/bonobo/issues/134
|
||||||
|
|
||||||
.. literalinclude:: ../../bonobo/examples/tutorials/tut02e01_read.py
|
.. literalinclude:: ../../bonobo/examples/tutorials/tut02e01_read.py
|
||||||
:language: python
|
:language: python
|
||||||
@ -1,6 +1,8 @@
|
|||||||
Configurables and Services
|
Configurables and Services
|
||||||
==========================
|
==========================
|
||||||
|
|
||||||
|
.. include:: _outdated_note.rst
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
This section lacks completeness, sorry for that (but you can still read it!).
|
This section lacks completeness, sorry for that (but you can still read it!).
|
||||||
@ -30,7 +32,7 @@ Configurables allows to use the following features:
|
|||||||
class PrefixIt(Configurable):
|
class PrefixIt(Configurable):
|
||||||
prefix = Option(str, positional=True, default='>>>')
|
prefix = Option(str, positional=True, default='>>>')
|
||||||
|
|
||||||
def __call__(self, row):
|
def call(self, row):
|
||||||
return self.prefix + ' ' + row
|
return self.prefix + ' ' + row
|
||||||
|
|
||||||
prefixer = PrefixIt('$')
|
prefixer = PrefixIt('$')
|
||||||
@ -48,7 +50,7 @@ Configurables allows to use the following features:
|
|||||||
url = Option(default='https://jsonplaceholder.typicode.com/users')
|
url = Option(default='https://jsonplaceholder.typicode.com/users')
|
||||||
http = Service('http.client')
|
http = Service('http.client')
|
||||||
|
|
||||||
def __call__(self, http):
|
def call(self, http):
|
||||||
resp = http.get(self.url)
|
resp = http.get(self.url)
|
||||||
|
|
||||||
for row in resp.json():
|
for row in resp.json():
|
||||||
@ -68,7 +70,7 @@ Configurables allows to use the following features:
|
|||||||
class Applier(Configurable):
|
class Applier(Configurable):
|
||||||
apply = Method()
|
apply = Method()
|
||||||
|
|
||||||
def __call__(self, row):
|
def call(self, row):
|
||||||
return self.apply(row)
|
return self.apply(row)
|
||||||
|
|
||||||
@Applier
|
@Applier
|
||||||
@ -114,7 +116,7 @@ Let's see how to use it, starting from the previous service example:
|
|||||||
url = Option(default='https://jsonplaceholder.typicode.com/users')
|
url = Option(default='https://jsonplaceholder.typicode.com/users')
|
||||||
http = Service('http.client')
|
http = Service('http.client')
|
||||||
|
|
||||||
def __call__(self, http):
|
def call(self, http):
|
||||||
resp = http.get(self.url)
|
resp = http.get(self.url)
|
||||||
|
|
||||||
for row in resp.json():
|
for row in resp.json():
|
||||||
@ -1,6 +1,8 @@
|
|||||||
Working with databases
|
Working with databases
|
||||||
======================
|
======================
|
||||||
|
|
||||||
|
.. include:: _outdated_note.rst
|
||||||
|
|
||||||
Databases (and especially SQL databases here) are not the focus of Bonobo, thus support for it is not (and will never
|
Databases (and especially SQL databases here) are not the focus of Bonobo, thus support for it is not (and will never
|
||||||
be) included in the main package. Instead, working with databases is done using third party, well maintained and
|
be) included in the main package. Instead, working with databases is done using third party, well maintained and
|
||||||
specialized packages, like SQLAlchemy, or other database access libraries from the python cheese shop.
|
specialized packages, like SQLAlchemy, or other database access libraries from the python cheese shop.
|
||||||
@ -1,6 +1,8 @@
|
|||||||
Part 2: Writing ETL Jobs
|
Part 2: Writing ETL Jobs
|
||||||
========================
|
========================
|
||||||
|
|
||||||
|
.. include:: _wip_note.rst
|
||||||
|
|
||||||
What's an ETL job ?
|
What's an ETL job ?
|
||||||
:::::::::::::::::::
|
:::::::::::::::::::
|
||||||
|
|
||||||
|
|||||||
@ -1,6 +1,8 @@
|
|||||||
Part 3: Working with Files
|
Part 3: Working with Files
|
||||||
==========================
|
==========================
|
||||||
|
|
||||||
|
.. include:: _wip_note.rst
|
||||||
|
|
||||||
* Filesystems
|
* Filesystems
|
||||||
|
|
||||||
* Reading files
|
* Reading files
|
||||||
|
|||||||
@ -1,6 +1,7 @@
|
|||||||
Part 4: Services and Configurables
|
Part 4: Services and Configurables
|
||||||
==================================
|
==================================
|
||||||
|
|
||||||
|
.. include:: _wip_note.rst
|
||||||
|
|
||||||
In the last section, we used a few new tools.
|
In the last section, we used a few new tools.
|
||||||
|
|
||||||
|
|||||||
@ -1,6 +1,8 @@
|
|||||||
Part 5: Projects and Packaging
|
Part 5: Projects and Packaging
|
||||||
==============================
|
==============================
|
||||||
|
|
||||||
|
.. include:: _wip_note.rst
|
||||||
|
|
||||||
Until then, we worked with one file managing a job.
|
Until then, we worked with one file managing a job.
|
||||||
|
|
||||||
Real life often involves more complicated setups, with relations and imports between different files.
|
Real life often involves more complicated setups, with relations and imports between different files.
|
||||||
|
|||||||
12
docs/tutorial/_wip_note.rst
Normal file
12
docs/tutorial/_wip_note.rst
Normal file
@ -0,0 +1,12 @@
|
|||||||
|
.. warning::
|
||||||
|
|
||||||
|
This section is being rewritten for |bonobo| 0.6, and it's now in a "work in progress" state.
|
||||||
|
|
||||||
|
You can read :doc:`the tutorial for the previous version (0.5) <0.5/index>`. Please note that things changed a bit
|
||||||
|
since then and you'll have quirks here and there.
|
||||||
|
|
||||||
|
You can also read the `migration guide from 0.5 to 0.6 <https://news.bonobo-project.org/migration-guide-for-bonobo-0-6-alpha-c1d36b0a9d35>`_
|
||||||
|
that will give you a good overview of the changes.
|
||||||
|
|
||||||
|
Hopefully, this document will be updated soon, and please accept our apologies about this doc status until then.
|
||||||
|
|
||||||
@ -1,3 +1,8 @@
|
|||||||
Working with Django
|
Working with Django
|
||||||
===================
|
===================
|
||||||
|
|
||||||
|
.. warning::
|
||||||
|
|
||||||
|
This section does not exist yet, but it's in the plans to write it quite soon. Meanwhile, you can check the source
|
||||||
|
code and other links provided below.
|
||||||
|
|
||||||
|
|||||||
@ -53,3 +53,4 @@ out.
|
|||||||
If you're not stuck but had trouble understanding something, please consider contributing to the docs (using GitHub
|
If you're not stuck but had trouble understanding something, please consider contributing to the docs (using GitHub
|
||||||
pull requests).
|
pull requests).
|
||||||
|
|
||||||
|
.. include:: _wip_note.rst
|
||||||
|
|||||||
@ -1,4 +1,8 @@
|
|||||||
Working with Jupyter Notebooks
|
Working with Jupyter Notebooks
|
||||||
==============================
|
==============================
|
||||||
|
|
||||||
|
.. warning::
|
||||||
|
|
||||||
|
This section does not exist yet, but it's in the plans to write it quite soon. Meanwhile, you can check the source
|
||||||
|
code and other links provided below.
|
||||||
|
|
||||||
|
|||||||
@ -1,4 +1,7 @@
|
|||||||
Working with SQL Databases
|
Working with SQL Databases
|
||||||
==========================
|
==========================
|
||||||
|
|
||||||
|
.. warning::
|
||||||
|
|
||||||
|
This section does not exist yet, but it's in the plans to write it quite soon. Meanwhile, you can check the source
|
||||||
|
code and other links provided below.
|
||||||
|
|||||||
Reference in New Issue
Block a user