Update stable doc with warnings.
This commit is contained in:
9
docs/tutorial/0.5/_outdated_note.rst
Normal file
9
docs/tutorial/0.5/_outdated_note.rst
Normal file
@ -0,0 +1,9 @@
|
||||
.. warning::
|
||||
|
||||
This tutorial was written for |bonobo| 0.5, while the current stable version is |bonobo| 0.6.
|
||||
|
||||
Please be aware that some things changed.
|
||||
|
||||
A summary of changes is available in the `migration guide from 0.5 to 0.6 <https://news.bonobo-project.org/migration-guide-for-bonobo-0-6-alpha-c1d36b0a9d35>`_.
|
||||
|
||||
|
||||
65
docs/tutorial/0.5/index.rst
Normal file
65
docs/tutorial/0.5/index.rst
Normal file
@ -0,0 +1,65 @@
|
||||
First steps
|
||||
===========
|
||||
|
||||
.. include:: _outdated_note.rst
|
||||
|
||||
What is Bonobo?
|
||||
:::::::::::::::
|
||||
|
||||
Bonobo is an ETL (Extract-Transform-Load) framework for python 3.5. The goal is to define data-transformations, with
|
||||
python code in charge of handling similar shaped independent lines of data.
|
||||
|
||||
Bonobo *is not* a statistical or data-science tool. If you're looking for a data-analysis tool in python, use Pandas.
|
||||
|
||||
Bonobo is a lean manufacturing assembly line for data that let you focus on the actual work instead of the plumbery
|
||||
(execution contexts, parallelism, error handling, console output, logging, ...).
|
||||
|
||||
Bonobo uses simple python and should be quick and easy to learn.
|
||||
|
||||
Tutorial
|
||||
::::::::
|
||||
|
||||
.. note::
|
||||
|
||||
Good documentation is not easy to write. We do our best to make it better and better.
|
||||
|
||||
Although all content here should be accurate, you may feel a lack of completeness, for which we plead guilty and
|
||||
apologize.
|
||||
|
||||
If you're stuck, please come and ask on our `slack channel <https://bonobo-slack.herokuapp.com/>`_, we'll figure
|
||||
something out.
|
||||
|
||||
If you're not stuck but had trouble understanding something, please consider contributing to the docs (via GitHub
|
||||
pull requests).
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
tut01
|
||||
tut02
|
||||
tut03
|
||||
tut04
|
||||
|
||||
|
||||
What's next?
|
||||
::::::::::::
|
||||
|
||||
Read a few examples
|
||||
-------------------
|
||||
|
||||
* :doc:`/reference/examples`
|
||||
|
||||
Read about best development practices
|
||||
-------------------------------------
|
||||
|
||||
* :doc:`/guide/index`
|
||||
* :doc:`/guide/purity`
|
||||
|
||||
Read about integrating external tools with bonobo
|
||||
-------------------------------------------------
|
||||
|
||||
* :doc:`/extension/docker`: run transformation graphs in isolated containers.
|
||||
* :doc:`/extension/jupyter`: run transformations within jupyter notebooks.
|
||||
* :doc:`/extension/selenium`: crawl the web using a real browser and work with the gathered data.
|
||||
* :doc:`/extension/sqlalchemy`: everything you need to interract with SQL databases.
|
||||
|
||||
13
docs/tutorial/0.5/python.rst
Normal file
13
docs/tutorial/0.5/python.rst
Normal file
@ -0,0 +1,13 @@
|
||||
Just enough Python for Bonobo
|
||||
=============================
|
||||
|
||||
.. include:: _outdated_note.rst
|
||||
|
||||
.. todo::
|
||||
|
||||
This is a work in progress and it is not yet available. Please come back later or even better, help us write this
|
||||
guide!
|
||||
|
||||
This guide is intended to help programmers or enthusiasts to grasp the python basics necessary to use Bonobo. It
|
||||
should definately not be considered as a general python introduction, neither a deep dive into details.
|
||||
|
||||
@ -1,7 +1,10 @@
|
||||
Let's get started!
|
||||
==================
|
||||
|
||||
To get started with Bonobo, you need to install it in a working python 3.5+ environment:
|
||||
.. include:: _outdated_note.rst
|
||||
|
||||
To begin with Bonobo, you need to install it in a working python 3.5+ environment, and you'll also need cookiecutter
|
||||
to bootstrap your project.
|
||||
|
||||
.. code-block:: shell-session
|
||||
|
||||
@ -13,24 +16,21 @@ See :doc:`/install` for more options.
|
||||
Create an empty project
|
||||
:::::::::::::::::::::::
|
||||
|
||||
Your ETL code will live in standard python files and packages.
|
||||
Your ETL code will live in ETL projects, which are basically a bunch of files, including python code, that bonobo
|
||||
can run.
|
||||
|
||||
.. code-block:: shell-session
|
||||
|
||||
$ bonobo create tutorial.py
|
||||
$ bonobo init tutorial
|
||||
|
||||
This will create a simple example job in a `tutorial.py` file.
|
||||
This will create a `tutorial` directory (`content description here <https://www.bonobo-project.org/with/cookiecutter>`_).
|
||||
|
||||
Now, try to execute it:
|
||||
To run this project, use:
|
||||
|
||||
.. code-block:: shell-session
|
||||
|
||||
$ python tutorial.py
|
||||
$ bonobo run tutorial
|
||||
|
||||
Congratulations, you just ran your first ETL job!
|
||||
|
||||
|
||||
.. todo:: XXX **CHANGES NEEDED BELOW THIS POINTS BEFORE 0.6** XXX
|
||||
|
||||
Write a first transformation
|
||||
::::::::::::::::::::::::::::
|
||||
@ -107,9 +107,6 @@ To do this, it needs to know what data-flow you want to achieve, and you'll use
|
||||
The `if __name__ == '__main__':` section is not required, unless you want to run it directly using the python
|
||||
interpreter.
|
||||
|
||||
The name of the `graph` variable is arbitrary, but this variable must be global and available unconditionally.
|
||||
Do not put it in its own function or in the `if __name__ == '__main__':` section.
|
||||
|
||||
|
||||
Execute the job
|
||||
:::::::::::::::
|
||||
@ -133,9 +130,9 @@ Rewrite it using builtins
|
||||
There is a much simpler way to describe an equivalent graph:
|
||||
|
||||
.. literalinclude:: ../../bonobo/examples/tutorials/tut01e02.py
|
||||
:language: python
|
||||
:language: python
|
||||
|
||||
The `extract()` generator has been replaced by a list, as Bonobo will interpret non-callable iterables as a no-input
|
||||
The `extract()` generator has been replaced by a list, as Bonobo will interpret non-callable iterables as a no-input
|
||||
generator.
|
||||
|
||||
This example is also available in :mod:`bonobo.examples.tutorials.tut01e02`, and you can also run it as a module:
|
||||
@ -1,6 +1,8 @@
|
||||
Working with files
|
||||
==================
|
||||
|
||||
.. include:: _outdated_note.rst
|
||||
|
||||
Bonobo would be pointless if the aim was just to uppercase small lists of strings.
|
||||
|
||||
In fact, Bonobo should not be used if you don't expect any gain from parallelization/distribution of tasks.
|
||||
@ -59,7 +61,13 @@ available in **Bonobo**'s repository:
|
||||
|
||||
.. code-block:: shell-session
|
||||
|
||||
$ bonobo download examples/datasets/coffeeshops.txt
|
||||
$ curl https://raw.githubusercontent.com/python-bonobo/bonobo/master/bonobo/examples/datasets/coffeeshops.txt > `python3 -c 'import bonobo; print(bonobo.get_examples_path("datasets/coffeeshops.txt"))'`
|
||||
|
||||
.. note::
|
||||
|
||||
The "example dataset download" step will be easier in the future.
|
||||
|
||||
https://github.com/python-bonobo/bonobo/issues/134
|
||||
|
||||
.. literalinclude:: ../../bonobo/examples/tutorials/tut02e01_read.py
|
||||
:language: python
|
||||
@ -1,6 +1,8 @@
|
||||
Configurables and Services
|
||||
==========================
|
||||
|
||||
.. include:: _outdated_note.rst
|
||||
|
||||
.. note::
|
||||
|
||||
This section lacks completeness, sorry for that (but you can still read it!).
|
||||
@ -30,7 +32,7 @@ Configurables allows to use the following features:
|
||||
class PrefixIt(Configurable):
|
||||
prefix = Option(str, positional=True, default='>>>')
|
||||
|
||||
def __call__(self, row):
|
||||
def call(self, row):
|
||||
return self.prefix + ' ' + row
|
||||
|
||||
prefixer = PrefixIt('$')
|
||||
@ -48,7 +50,7 @@ Configurables allows to use the following features:
|
||||
url = Option(default='https://jsonplaceholder.typicode.com/users')
|
||||
http = Service('http.client')
|
||||
|
||||
def __call__(self, http):
|
||||
def call(self, http):
|
||||
resp = http.get(self.url)
|
||||
|
||||
for row in resp.json():
|
||||
@ -68,7 +70,7 @@ Configurables allows to use the following features:
|
||||
class Applier(Configurable):
|
||||
apply = Method()
|
||||
|
||||
def __call__(self, row):
|
||||
def call(self, row):
|
||||
return self.apply(row)
|
||||
|
||||
@Applier
|
||||
@ -114,7 +116,7 @@ Let's see how to use it, starting from the previous service example:
|
||||
url = Option(default='https://jsonplaceholder.typicode.com/users')
|
||||
http = Service('http.client')
|
||||
|
||||
def __call__(self, http):
|
||||
def call(self, http):
|
||||
resp = http.get(self.url)
|
||||
|
||||
for row in resp.json():
|
||||
@ -1,6 +1,8 @@
|
||||
Working with databases
|
||||
======================
|
||||
|
||||
.. include:: _outdated_note.rst
|
||||
|
||||
Databases (and especially SQL databases here) are not the focus of Bonobo, thus support for it is not (and will never
|
||||
be) included in the main package. Instead, working with databases is done using third party, well maintained and
|
||||
specialized packages, like SQLAlchemy, or other database access libraries from the python cheese shop.
|
||||
@ -1,6 +1,8 @@
|
||||
Part 2: Writing ETL Jobs
|
||||
========================
|
||||
|
||||
.. include:: _wip_note.rst
|
||||
|
||||
What's an ETL job ?
|
||||
:::::::::::::::::::
|
||||
|
||||
|
||||
@ -1,6 +1,8 @@
|
||||
Part 3: Working with Files
|
||||
==========================
|
||||
|
||||
.. include:: _wip_note.rst
|
||||
|
||||
* Filesystems
|
||||
|
||||
* Reading files
|
||||
|
||||
@ -1,6 +1,7 @@
|
||||
Part 4: Services and Configurables
|
||||
==================================
|
||||
|
||||
.. include:: _wip_note.rst
|
||||
|
||||
In the last section, we used a few new tools.
|
||||
|
||||
|
||||
@ -1,6 +1,8 @@
|
||||
Part 5: Projects and Packaging
|
||||
==============================
|
||||
|
||||
.. include:: _wip_note.rst
|
||||
|
||||
Until then, we worked with one file managing a job.
|
||||
|
||||
Real life often involves more complicated setups, with relations and imports between different files.
|
||||
|
||||
12
docs/tutorial/_wip_note.rst
Normal file
12
docs/tutorial/_wip_note.rst
Normal file
@ -0,0 +1,12 @@
|
||||
.. warning::
|
||||
|
||||
This section is being rewritten for |bonobo| 0.6, and it's now in a "work in progress" state.
|
||||
|
||||
You can read :doc:`the tutorial for the previous version (0.5) <0.5/index>`. Please note that things changed a bit
|
||||
since then and you'll have quirks here and there.
|
||||
|
||||
You can also read the `migration guide from 0.5 to 0.6 <https://news.bonobo-project.org/migration-guide-for-bonobo-0-6-alpha-c1d36b0a9d35>`_
|
||||
that will give you a good overview of the changes.
|
||||
|
||||
Hopefully, this document will be updated soon, and please accept our apologies about this doc status until then.
|
||||
|
||||
@ -1,3 +1,8 @@
|
||||
Working with Django
|
||||
===================
|
||||
|
||||
.. warning::
|
||||
|
||||
This section does not exist yet, but it's in the plans to write it quite soon. Meanwhile, you can check the source
|
||||
code and other links provided below.
|
||||
|
||||
|
||||
@ -53,3 +53,4 @@ out.
|
||||
If you're not stuck but had trouble understanding something, please consider contributing to the docs (using GitHub
|
||||
pull requests).
|
||||
|
||||
.. include:: _wip_note.rst
|
||||
|
||||
@ -1,4 +1,8 @@
|
||||
Working with Jupyter Notebooks
|
||||
==============================
|
||||
|
||||
.. warning::
|
||||
|
||||
This section does not exist yet, but it's in the plans to write it quite soon. Meanwhile, you can check the source
|
||||
code and other links provided below.
|
||||
|
||||
|
||||
@ -1,4 +1,7 @@
|
||||
Working with SQL Databases
|
||||
==========================
|
||||
|
||||
.. warning::
|
||||
|
||||
This section does not exist yet, but it's in the plans to write it quite soon. Meanwhile, you can check the source
|
||||
code and other links provided below.
|
||||
|
||||
Reference in New Issue
Block a user