paulin/bonobo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update services.rst

Browse Source
This commit is contained in:
Romain Dorgueil
2017-04-27 09:47:24 +02:00
committed by GitHub
parent 56ffaf1a8f
commit 02f0c1d972
1 changed files with 21 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
docs/guide/services.rst
View File

@ -8,7 +8,9 @@ Services and dependencies (draft implementation)
Most probably, you'll want to use external systems within your transformations. Those systems may include databases, Most probably, you'll want to use external systems within your transformations. Those systems may include databases,
apis (using http, for example), filesystems, etc. apis (using http, for example), filesystems, etc.
Hardocing those services in your code can do the job at first, but you'll pretty soon feel limited, for a few reasons: You can start by hardcoding those services. That does the job, at first.
If you're going a little further than that, you'll feel limited, for a few reasons:
* Hardcoded and tightly linked dependencies make your transformations hard to test, and hard to reuse. * Hardcoded and tightly linked dependencies make your transformations hard to test, and hard to reuse.
* Processing data on your laptop is great, but being able to do it on different systems (or stages), in different * Processing data on your laptop is great, but being able to do it on different systems (or stages), in different
@ -61,10 +63,15 @@ Let's see how to execute it:
import bonobo import bonobo
bonobo.run( graph = bonobo.graph(
[...extract...], *before,
JoinDatabaseCategories(), JoinDatabaseCategories(),
[...load...], *after,
)
if __name__ == '__main__':
bonobo.run(
graph,
services={ services={
'primary_sql_database': my_database_service, 'primary_sql_database': my_database_service,
} }
@ -75,13 +82,13 @@ A dictionary, or dictionary-like, "services" named argument can be passed to the
provided is pretty basic, and feature-less. But you can use much more evolved libraries instead of the provided provided is pretty basic, and feature-less. But you can use much more evolved libraries instead of the provided
stub, and as long as it works the same (a.k.a implements a dictionary-like interface), the system will use it. stub, and as long as it works the same (a.k.a implements a dictionary-like interface), the system will use it.
Future Future and proposals
:::::: ::::::::::::::::::::
This is the first proposed implementation and it will evolve, but looks a lot like how we used bonobo ancestor in This is the first proposed implementation and it will evolve, but looks a lot like how we used bonobo ancestor in
production. production.
You can expect to see the following features pretty soon: May or may not happen, depending on discussions.
* Singleton or prototype based injection (to use spring terminology, see * Singleton or prototype based injection (to use spring terminology, see
https://www.tutorialspoint.com/spring/spring_bean_scopes.htm), allowing smart factory usage and efficient sharing of https://www.tutorialspoint.com/spring/spring_bean_scopes.htm), allowing smart factory usage and efficient sharing of
@ -89,8 +96,12 @@ You can expect to see the following features pretty soon:
* Lazily resolved parameters, eventually overriden by command line or environment, so you can for example override the * Lazily resolved parameters, eventually overriden by command line or environment, so you can for example override the
database DSN or target filesystem on command line (or with shell environment). database DSN or target filesystem on command line (or with shell environment).
* Pool based locks that ensure that only one (or n) transformations are using a given service at the same time. * Pool based locks that ensure that only one (or n) transformations are using a given service at the same time.
* Simple config implementation, using a python file for config (ex: bonobo run ... --services=services_prod.py).
* Default configuration for services, using an optional callable (`def get_services(args): ...`). Maybe tie default
configuration to graph, but not really a fan because this is unrelated to graph logic.
This is under development, let us know what you think (slack may be a good place for this). This is under development, let us know what you think (slack may be a good place for this).
The basics already work, and you can try it.
Read more Read more