zzzeek - Latest Comments

Re: zzzeek : Django-style Database Routers in SQLAlchemy

Ahmad Javed — Sat, 14 Feb 2015 13:48:33 -0000

Thanks Mike for the example.

I am facing the same problem pointed by Chris and Han Cold Kim. I am working on a web site using Python Django and SQLAlchemy ORM. I want to do all read operations in separate database and write operations in separate database.
So I used your thoughts but query.update() and query.delete() didn't work properly. I debugged some code and modified SessionRouter class as following:

class SQLAlchemySessionRouter(Session):
def get_bind(self, mapper=None, clause=None):
if self._flushing:
return ENGINES['write-only']
elif isinstance(clause, sqlalchemy.sql.expression.Update):
return ENGINES['write-only']
elif isinstance(clause, sqlalchemy.sql.expression.Delete):
return ENGINES['write-only']
else:
return ENGINES['read-only']

Now it is working fine in my case (I tested this manually).

Mike I need your guidance. Do these modifications looks good? Should I go with this solution? Or is there some thing better that can help me?

Thanks in advance,

Re: zzzeek : Using Beaker for Caching? Why You'll Want to Switch to dogpile.cache

Guest — Wed, 14 Jan 2015 13:49:07 -0000

I am very new to python and have been assigned a task to
upgrade an application's caching from beaker to dogpile. The
configuration for beaker was done in an .ini file as follows:

cache.data_dir = cache/data
cache.lock_dir = cache/lock
cache.regions = short_term, long_term
cache.short_term.type = memory
cache.short_term.expire = 1000
cache.long_term.type = file
cache.long_term.expire = 3600

From what I read online here(https://github.com/moriyosh... the dogpile configuration I have done is as follows:

dogpile_cache.backend = memcache
dogpile_cache.regions = short_term,long_term
dogpile_cache.expire = 1000
dogpile_cache.short_term.ar... = 127.0.0.1
dogpile_cache.short_term.arguments.port = 6379

I just copied from the above mentioned link, but dont understand what
is the purpose of host and port here? Is it needed, please note that
my application will have memcache backend.
My apologies if the question is vague, as I mentioned this is my first
time with python. Would appreciate some help.

Thanks

Re: zzzeek : Pycon 2012 : Hand Coded Applications with SQLAlchemy

zzzeek — Sat, 03 Jan 2015 21:39:40 -0000

good idea.

Re: zzzeek : Pycon 2012 : Hand Coded Applications with SQLAlchemy

Bastien Gandouet — Sat, 03 Jan 2015 14:19:24 -0000

Great talk! You should probably add the "Talk" tag to this post.

Re: zzzeek : The Enum Recipe

canni — Sat, 13 Dec 2014 15:31:12 -0000

Hey, I stumbled upon this article and was using this recipe for some time, but eventually this can be effectively merged with Enum implementation from python 3.4 to reduce code duplication (there is a backport for p2.7)
Take a look on full implementation example: https://gist.github.com/can...

Re: zzzeek : Django-style Database Routers in SQLAlchemy

김한기 — Wed, 03 Sep 2014 02:43:13 -0000

Thank you. Good article.

All is well.

But, my problem is `UPDATE` query is sent to `slave` database.

How can I solve this?

Re: zzzeek : Django-style Database Routers in SQLAlchemy

김한기 — Wed, 03 Sep 2014 02:39:55 -0000

Thanks for the comment.

> query.delete() or query.update() I always have to use session.using_bind("master") explicitly.

How...?

Re: zzzeek

jasmine nile — Mon, 14 Jul 2014 00:18:17 -0000

good one,, thanks for sharing it

Re: zzzeek : The Enum Recipe

jasmine nile — Mon, 14 Jul 2014 00:16:19 -0000

http://techspot.zzzeek.org/

Re: zzzeek : The Enum Recipe

Nguyễn Hồng Quân — Mon, 07 Apr 2014 14:11:17 -0000

I fixed it. It turns out to be Python 3 compatibility. I redefine the DeclEnum class with Python3 style:
class DeclEnum(metaclass=EnumMeta):

Re: zzzeek : The Enum Recipe

Nguyễn Hồng Quân — Mon, 07 Apr 2014 13:54:27 -0000

When I define like this (with Flask-SQlAlchemy):

class OAServiceType(DeclEnum):
facebook = ('facebook', 'Facebook')
twitter = ('twitter', 'Twitter')
google = ('google', 'Google')

class WhiteDomain(db.Model):
__tablename__ = 'whitedomains'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(100), nullable=False, unique=True)
service = db.Column(OAServiceType.db_type())

I get warning from SQLAlchemy:

venv/lib/python3.3/site-packages/sqlalchemy/sql/expression.py:1987: SAWarning: The IN-predicate on "whitedomains.service" was invoked with an empty sequence. This results in a contradiction, which nonetheless can be expensive to evaluate. Consider alternative strategies for improved performance.
return o[0](self, self.expr, op, *(other + o[1:]), **kwargs)

What's wrong with it?

Re: zzzeek : The Enum Recipe

zzzeek — Mon, 31 Mar 2014 10:20:22 -0000

everything I put up here is more or less public domain but I'd need to set a copyright term for the blog overall.

Re: zzzeek : The Enum Recipe

Joshua Randall — Mon, 31 Mar 2014 06:59:05 -0000

Mike - thanks for this great example of how to do application-level enums using SQLAlchemy! Could you just confirm whether your example code in decl_enum.py is intended to be freely available for reuse? I'm assuming you probably intend it to be under MIT license, but I couldn't find anywhere on the blog that states license terms for your posts. Perhaps you could just pop a quick copyright / license header on top of decl_enum.py ?

Re: zzzeek : The Enum Recipe

zzzeek — Thu, 20 Feb 2014 11:37:01 -0000

It's sort of unavoidable if you have a data structure which consists of a list of 2-tuples, and then you'd like to change it to be a list of 3-tuples including that the iteration accessor now returns those 3-tuples (or just returns an N-tuple), that code which refers to it as a 2-tuple would have to change. Certainly, if you're opposed to using nameless tuples as data structures, then that's a whole different issue :).

As it turns out, the `__iter__()` on this particular recipe only returns "value, description", e.g. a 2-tuple no matter what, and the above recipe is also fixed to only accepting 2-tuples for values, though you can certainly change that. The one I use I think supports some extra fields and various accessors and iterators, but that's all just Python programming stuff. Because Python isn't statically typed its not very easy to have the Java "refactor" button that can just change things across the board when signatures change.

So if you want to say, "don't use Python, it's not statically typed and therefore you lose the benefit of code refactoring tools!" feel free. I've done lots of Java and I do certainly miss that aspect of it.

Re: zzzeek : The Enum Recipe

Matan Aharoni — Thu, 20 Feb 2014 09:52:44 -0000

If I wanted to add a key or a message and change
class EmployeeType(DeclEnum):
part_time = "P", "Part Time"
full_time = "F", "Full Time"
contractor = "C", "Contractor"
to part_time = "P", "Part Time", 3, "resistance is futile"
previously functional loops like
for symbol, value in EmployeeType:
would result in an error, there could be dozens of instances across multiple files. It also appears that the constructor and iterator function will both have to be updated to support the new functionality.

Re: zzzeek : The Enum Recipe

Guest — Thu, 20 Feb 2014 09:08:02 -0000

Fair enough, I'll read through your code more thoroughly and delete my comment or edit it accordingly.

Re: zzzeek : The Enum Recipe

zzzeek — Thu, 20 Feb 2014 09:01:57 -0000

If you read the article, you will see that I am not talking about arbitrary data like lists of country codes or timezones, I am talking specifically about values that some part of the application needs to be aware of. So as far as "change the type", the options are adding new values (no change required), removing existing values (any part of the code which refers to that value explicitly needs to change, but this change would have occurred in the code first anyway) and changing the names of values (no change required). It seems very much like you had some experience with an approach that may have seemed similar to what is proposed here, however is not actually the same thing. We use this approach extensively in a very large application with many dozens of such enums, we make extensive database changes all the time, and have had no issues.

Re: zzzeek : The Enum Recipe

Matan Aharoni — Thu, 20 Feb 2014 06:38:17 -0000

I recommend against this approach.
If all you need is to iterate over the enum options hard code a list (you have to manually type in the values for this approach anyway). If you need lots of extra functionality you're better off with a separate table.
I may be wrong; this has been my experience.

Re: zzzeek : Introduction to SQLAlchemy - Pycon 2013 - Wrapup

Yi LIU — Thu, 26 Dec 2013 04:50:53 -0000

Great job!

Re: zzzeek : Introduction to SQLAlchemy - Pycon 2013 - Wrapup

disqus_xzfOzjzVbV — Sun, 15 Dec 2013 08:13:48 -0000

hey. i am developing a personal project with heavy use of sqlalchemy. i want to pay you to review my code and help me improve it. please send me an email at dmaxashman@gmail.com to let me know what you think.

Re: zzzeek : Akiban-SQLAlchemy - a Webinar

zzzeek — Mon, 05 Aug 2013 14:37:41 -0000

so it is, guess cuz they got bought out. ill take this post down...

Re: zzzeek : Using Beaker for Caching? Why You'll Want to Switch to dogpile.cache

Bright Wang — Thu, 09 May 2013 01:59:28 -0000

we use dogpile and memcache as cachebackend, it seems everything is ok,but sometimes when the memcache broken,dogpile report connection error, in production the app will stop runing, how could we deal with it? , catch the excepion?

Re: zzzeek : Django-style Database Routers in SQLAlchemy

Chris — Fri, 19 Apr 2013 06:22:46 -0000

Thanks for the example.

When using the RoutingSession I encountered a stumbling block, i.e. my program sent write/delete operations to a database slave - ouch.

After debugging/single-stepping it became clear to me that for the idiom query.delete() or query.update() I always have to use session.using_bind("master") explicitly.

Reason: Query.delete() and .update() might call session._autoflush(), but after that _flushing is reset to False, and then session.execute calls session.get_bind while _flushing is False.

Maybe me stumbling over this might deserve a little cautionary remark in http://docs.sqlalchemy.org/....

Thanks again.

Re: zzzeek : Using Beaker for Caching? Why You'll Want to Switch to dogpile.cache

zzzeek — Mon, 25 Mar 2013 13:44:14 -0000

dogpile works as a key/value store, so when the value for a particular key is invalidated, it's gone. invalidation is also "soft" based on expiration time, so there's no way an active "event" can be generated unless the backend supports such an event, or some kind of worker thread is scanning the whole cache all the time detecting values that have timed out.

It seems like you're referring to the idea of storing multiple copies of an item in a cache, that is, when you have a series of query results individually stored in the cache, as well as potentially a single copy of an object that is also in those query results. if you want cross-referencing cache invalidations like that, there's two general strategies - one is to tailor the query caching such that the results are cached only as a collection of other cache keys - this will add performance overhead as retrieving a cached query also means you need to retrieve each of the referenced objects. The other is to somehow relate a series of cached queries together, and potentially associated with each individual key in the query as well. There's many ways to approach that, and it depends on specifics. Such as, if I wanted to cache the result of "SELECT * FROM table" as well as have cache records for each individual row, I might keep track of each row loaded from that query in an additional lookup table, so that when the "individual row" record is expired, a search is performed in the cache for all queries that also referred to that row so they can also be expired.

These are very complex scenarios and I'm pretty skeptical that a Django plugin can provide a foolproof and performant out-of-the-box solution - IMHO, this is the kind of thing you really need to understand and implement fully, relying on a 3rd party system to "magically" do it will almost immediately lead to failure cases that can't be explained without fully understanding the whole system's inner workings.

Re: zzzeek : Using Beaker for Caching? Why You'll Want to Switch to dogpile.cache

Shahaf Abileah — Mon, 25 Mar 2013 13:33:05 -0000

It sounds like dogpile doesn't do automatic invalidation (e.g. when an object is updated, invalidate all query results that include that object). Is that correct? Are there plans to add that support? Or any tips on how to approach it?

Related discussion:
https://groups.google.com/f...

Looks like Django has a Cache Machine layer that supports automatic invalidation:
http://jbalogh.me/2010/02/0...