zzzeek - Latest Comments

Re: zzzeek : Using Beaker for Caching? Why You'll Want to Switch to dogpile.cache

Bright Wang — Thu, 09 May 2013 01:59:28 -0000

we use dogpile and memcache as cachebackend, it seems everything is ok,but sometimes when the memcache broken,dogpile report connection error, in production the app will stop runing, how could we deal with it? , catch the excepion?

Re: zzzeek : Django-style Database Routers in SQLAlchemy

Chris — Fri, 19 Apr 2013 06:22:46 -0000

Thanks for the example.

When using the RoutingSession I encountered a stumbling block, i.e. my program sent write/delete operations to a database slave - ouch.

After debugging/single-stepping it became clear to me that for the idiom query.delete() or query.update() I always have to use session.using_bind("master") explicitly.

Reason: Query.delete() and .update() might call session._autoflush(), but after that _flushing is reset to False, and then session.execute calls session.get_bind while _flushing is False.

Maybe me stumbling over this might deserve a little cautionary remark in http://docs.sqlalchemy.org/en/....

Thanks again.

Re: zzzeek : Using Beaker for Caching? Why You'll Want to Switch to dogpile.cache

zzzeek — Mon, 25 Mar 2013 13:44:14 -0000

dogpile works as a key/value store, so when the value for a particular key is invalidated, it's gone. invalidation is also "soft" based on expiration time, so there's no way an active "event" can be generated unless the backend supports such an event, or some kind of worker thread is scanning the whole cache all the time detecting values that have timed out.

It seems like you're referring to the idea of storing multiple copies of an item in a cache, that is, when you have a series of query results individually stored in the cache, as well as potentially a single copy of an object that is also in those query results. if you want cross-referencing cache invalidations like that, there's two general strategies - one is to tailor the query caching such that the results are cached only as a collection of other cache keys - this will add performance overhead as retrieving a cached query also means you need to retrieve each of the referenced objects. The other is to somehow relate a series of cached queries together, and potentially associated with each individual key in the query as well. There's many ways to approach that, and it depends on specifics. Such as, if I wanted to cache the result of "SELECT * FROM table" as well as have cache records for each individual row, I might keep track of each row loaded from that query in an additional lookup table, so that when the "individual row" record is expired, a search is performed in the cache for all queries that also referred to that row so they can also be expired.

These are very complex scenarios and I'm pretty skeptical that a Django plugin can provide a foolproof and performant out-of-the-box solution - IMHO, this is the kind of thing you really need to understand and implement fully, relying on a 3rd party system to "magically" do it will almost immediately lead to failure cases that can't be explained without fully understanding the whole system's inner workings.

Re: zzzeek : Using Beaker for Caching? Why You'll Want to Switch to dogpile.cache

Shahaf Abileah — Mon, 25 Mar 2013 13:33:05 -0000

It sounds like dogpile doesn't do automatic invalidation (e.g. when an object is updated, invalidate all query results that include that object). Is that correct? Are there plans to add that support? Or any tips on how to approach it?

Related discussion:
https://groups.google.com/foru...

Looks like Django has a Cache Machine layer that supports automatic invalidation:
http://jbalogh.me/2010/02/09/c...

Re: zzzeek : Supporting a (Very Interesting) New Database

avoidscorn — Wed, 06 Mar 2013 13:08:20 -0000

Could the nested cursor and/or nestedload() concepts be applied to plain PostgreSQL using arrays? Example:


SELECT
  customer.id AS customer_id,
  customer.name AS customer_name,
  ARRAY(
    SELECT (
      "order".id,
      "order".customer_id,
      "order".order_info,
      ARRAY(SELECT item FROM item WHERE "order".id = item.order_id)
    )
    FROM "order"
    WHERE customer.id = "order".customer_id
  ) AS anon_1
FROM customer
WHERE customer.id = %(id_1)s

This could save row processing time and avoid the row explosion associated with multiple joinedloads.

Re: zzzeek : Polymorphic Associations with SQLAlchemy

Kevin — Mon, 07 Jan 2013 20:33:32 -0000

Good point about the ORM and migration making it easier to change later. I will need to reverse the lookup -- it's a temporal triplestore / eav I'll be implementing. I'll see how far the normalized approach gets me. Thanks, Mike!

Re: zzzeek : Polymorphic Associations with SQLAlchemy

zzzeek — Mon, 07 Jan 2013 15:50:07 -0000

90% of the time I use the table-per-association approach, which has just one join, and it joins on simple integer indexed columns without an awkward string lookup thrown in. Because we're using ORMs and migration tools, the fact that you end up creating more tables doesn't impact development complexity at all.

I really never need to look up a polymorphic association in reverse (that is, all the objects that refer to Address), and I rarely want all the "associations" across many different kinds of objects to all be aggregated in one huge table. So really, the things that the AR pattern offers even functionally I either don't need or I can get in a much simpler and more efficient way.

I think in relational databases, diverging dramatically from a normalized approach as the AR approach does is a consistency trap. At the end of the day a normalized and consistency-enforceable design saves my bacon in such a vast majority of cases that I'm not just very interested in the "fake foreign key" pattern, even if it did solve some problem I actually couldn't solve more easily in other ways.

Re: zzzeek : Polymorphic Associations with SQLAlchemy

Kevin — Mon, 07 Jan 2013 13:52:03 -0000

Thank you very much for this and particularly the updated example!

I was wondering if you could further elaborate on the design choice. Yours is normalized, but AR eliminates extra joins. Is it worth worrying about efficiency and minimizing space and joins at this point? Do databases automatically denormalize / cache these cases? If not, what would you do to optimize this later?

Re: zzzeek : Thoughts on Beaker

zzzeek — Mon, 17 Dec 2012 11:37:55 -0000

there's no plans for an encrypted cookie lib within the dogpile.* project. I'd like Ben or someone to re-package Beaker's encrypted cookie as its own thing.

Re: zzzeek : Thoughts on Beaker

dbv — Mon, 17 Dec 2012 11:32:13 -0000

We use the beaker cookie session facility (in development mode so far) and wondered if there was a status for the new dogpile.* library for encrypted cookie sessions?

Re: zzzeek : Supporting a (Very Interesting) New Database

Padraig O'Sullivan — Fri, 26 Oct 2012 11:51:35 -0000

Sounds good. Let me know if you have any further questions or if something is not answered by the page I linked to.

Re: zzzeek : Supporting a (Very Interesting) New Database

jmafc — Fri, 26 Oct 2012 11:43:52 -0000

Yes, that's what I was referring to. Your reply wasn't here when I came to respond to Mike.

Re: zzzeek : Supporting a (Very Interesting) New Database

jmafc — Fri, 26 Oct 2012 11:40:29 -0000

No, it's not Oracle GROUPING. It's an Oracle CLUSTER. From the docs: "Clusters are groups of one or more tables physically stored together because they share common columns and are often used together. Because related rows are physically stored
together, disk access time improves." So it allows you to store Customers together with their Orders, but as stated, it's strictly physical. You still get a relational model view of the data (well, SQL "relational").

Re: zzzeek : Supporting a (Very Interesting) New Database

Padraig O'Sullivan — Fri, 26 Oct 2012 11:33:58 -0000

I believe you are referring to table clusters in Oracle (http://docs.oracle.com/cd/E118.... We have a page on our support portal that compares this feature with Akiban's table grouping - http://support.akiban.com/entr...

Re: zzzeek : Supporting a (Very Interesting) New Database

agilevic — Fri, 26 Oct 2012 11:12:32 -0000

Thank you for your interest in SlashDB (/db), Simon. /db is often a part of client's SaaS product (cloud installation), used as a gateway to enterprise databases (DMZ) or to realize benefits of Resource Oriented Architecture in data integration (behind firewall).

The new version 0.6b is coming out as I write this. /db is available directly from VT Enterprise as an AMI virtual machine for Amazon EC2 or as a VMWare image or can be installed client's hardware. Currently we help our clients with installation and configuration. We are also planning a self-service version of the product to be generally available on AWS Marketplace by year end.

Since you have already registered for a demo on our website we will be in touch with you shortly.

Re: zzzeek : Supporting a (Very Interesting) New Database

zzzeek — Fri, 26 Oct 2012 10:38:01 -0000

there's a good article on Oracle's GROUPING and related over here: http://www.oracle-base.com/art... . I've never taken the time to understand how these operations work. Not sure if you're referring to another feature that actually changes the on-disk storage ...

Re: zzzeek : Supporting a (Very Interesting) New Database

jmafc — Fri, 26 Oct 2012 09:30:14 -0000

I seem to recall that Oracle had that kind of "GROUPING" hierarchical storage capability, somewhat like Postgres' CLUSTER, but possibly combining tables like you describe.

Re: zzzeek : Supporting a (Very Interesting) New Database

Simon Wittber — Thu, 25 Oct 2012 22:21:25 -0000

SlashDB looks interesting, is it released yet? I can't see any information on how to get a hold of it...

Re: zzzeek : Polymorphic Associations with SQLAlchemy

zzzeek — Tue, 25 Sep 2012 10:58:13 -0000

OK, well I can't run that to see the specific issue but in general association_proxy has a "creator" argument that allows you to set up what it does when the target collection or attribute doesn't exist: http://docs.sqlalchemy.org/en/... hopefully that's where the solution to your issue would be

Re: zzzeek : Polymorphic Associations with SQLAlchemy

Jökull Sólberg Auðunsson — Tue, 25 Sep 2012 05:03:42 -0000

Sorry about that. Won’t be using hastebin again :s

http://dpaste.de/nVvuC/

Re: zzzeek : Polymorphic Associations with SQLAlchemy

zzzeek — Mon, 24 Sep 2012 20:01:41 -0000

it was down earlier, and now there's nothing there. can you use a more common pastebin...

Re: zzzeek : Polymorphic Associations with SQLAlchemy

Jökull Sólberg Auðunsson — Mon, 24 Sep 2012 11:49:08 -0000

Here is a snippet: http://hastebin.com/vinexumevo...

I am using the new declarative form as a base. Notice in line 67 where I am having to check if the association is set or not.

Re: zzzeek : Polymorphic Associations with SQLAlchemy

zzzeek — Mon, 24 Sep 2012 11:38:56 -0000

which example are you using (not the really old ones here I hope) and can you post a full example please.

Re: zzzeek : Polymorphic Associations with SQLAlchemy

Jökull Sólberg Auðunsson — Mon, 24 Sep 2012 08:46:17 -0000

Love this pattern. Using it for an event stream, where different models can be subscribed to news items. I have one question: Since `assoc_id` is nullable the accessor raises an error when trying to append the first item through the association proxy and the association one-to-one hasn’t been set.

Here is the relevant traceback
File "/Users/gydaloa/Envs/wodboard/lib/python2.7/site-packages/sqlalchemy/ext/associationproxy.py", line 229, in __get__ return self._scalar_get(getattr(obj, self.target_collection)) AttributeError: 'NoneType' object has no attribute 'events'

Is there an elegant way to set this automatically?

Re: zzzeek : Server Side Templates and API Centric Development

blaflamme — Sun, 23 Sep 2012 11:10:39 -0000

Partial templates rendering are now part of Pyramid 1.4 as package:path/to/template#defname.mako

Thanks for the write up and suggestion.