python - context switching with 'yield' -

python - context switching with 'yield' -

i reading gevent tutorial , saw interesting snippet:

import gevent def foo(): print('running in foo') gevent.sleep(0) print('explicit context switch foo again') def bar(): print('explicit context bar') gevent.sleep(0) print('implicit context switch bar') gevent.joinall([ gevent.spawn(foo), gevent.spawn(bar), ])

in flow of execution goes foo -> bar -> foo -> bar . not possible same without gevent module yield statements? i've been trying 'yield' reason can't work... :(

generators used purpose called tasks (among many other terms), , i'll utilize term here clarity. yes, possible. there are, in fact, several approaches work , create sense in contexts. however, none (that i'm aware of) work without equivalent @ to the lowest degree 1 of gevent.spawn , gevent.joinall. more powerful , well-designed ones require equivalent both.

the fundamental problem this: generators can suspended (when nail yield), that's it. kick them off again, need other code calling next() on them. in fact, need phone call next() on freshly-created generator anything begin with. similarly, generator isn't best place decide should run next. need loop initiates each tasks's time piece (runs them next yield) , switches between them, indefinitely. called scheduler. tend become hairy quickly, won't effort write total scheduler in 1 answer. there core concepts can seek explain:

one treats yield giving command sheduler (in effect similar gevent.sleep(0) in code). means, generator whatever wants do, , when it's in place context switch convenient , perchance useful, yields. in python 3.3+, yield from useful tool delegate generator. if can't utilize it, have create scheduler emulate phone call stack , route homecoming values right place, , things result = yield subtasks() in tasks. slower, more complex implement, , unlikely yield useful stack traces (yield from free). until recently, best had. depending on utilize case, may need wide range of tools manage tasks. mutual examples spawning more tasks, waiting task complete, waiting 1 of several tasks complete, detecting failure (uncaught exception) of other tasks, etc. these handled scheduler , tasks given api communicate scheduler. neat (but not perfect) way communication yielding special values. one rather of import difference between generator-based tasks , gevent (and similar libraries) context switches in latter implicit, while tasks create trivial identify context switches: things yield [from] can perchance run scheduler code. example, can create sure whether piece of code atomic (w.r.t. other tasks; if add together threads mix, have worry them independently) looking @ code, without inspecting anything calls.

finally, may interested in greg ewing's tutorial on creating such scheduler. (this came on python-ideas while brainstorming on pep 3156. these mail service threads may of involvement your, though web-based archive not suited reading hundreds of mails in dozens of threads written half year ago.)

python gevent