python - Django Rest Framework / Django performance issue -

python - Django Rest Framework / Django performance issue -

i deploying django w/ django-rest-framework on ec2 little instance server provide set of apis couple of android apps.

the problem facing serious performance issue had profile. found out of time single request spent within drf's core.

sorry making long post think have show can proper reply whats going on. allow me go ahead:

my setup nginx / uwsgi. here's how running uwsgi using upstart:

description "pycms" start on [2345] stop on [06] respawn # start virtualenv path chdir /www/python/apps/pycms/ exec uwsgi -b 25000 --chdir=/www/python/apps/pycms --module=wsgi:application --env django_settings_module=settings --socket=127.0.0.1:8081 --processes=5 --harakiri=20 --max-requests=5000 --vacuum --master --pidfile=/tmp/pycms-master.pid

assuming request next api:

http://ip_address/api/nodes/mostviewed/9/

which matches next rule:

url(r'^nodes/mostviewed/(?p<category>\d+)/$', mostviewednodeslist.as_view(), name='mostviewed-nodes-list'),

here's class-based view:

class mostviewednodeslist(generics.listapiview): """ api endpoint lists featured nodes """ model = objectstats serializer_class = nodeserializer permission_classes = (permissions.allowany,) def get_queryset(self): if(self.kwargs.has_key('category')): category_id = self.kwargs.get('category') homecoming objectstats.get_most_viewed(category_id) else: homecoming []

the serializer class:

class nodeserializer(serializers.modelserializer): images = imageserializer() favorite = objectfield(source='is_favorite') rating = objectfield(source='get_rating') meta = objectfield(source='get_meta') url = objectfield(source='get_absolute_url') channel_title = objectfield(source='channel_title') class meta: model = node fields = ('id', 'title', 'body', 'images', 'parent', 'type', 'rating', 'meta', 'favorite', 'url', 'channel_title')

and classmethod 'get_most_viewed' (i know it's wrong utilize classmethods rather manager method)

@classmethod def get_most_viewed(cls, category_id): homecoming list(node.objects.extra( where=['objects.id=content_api_objectviewsstats.node_id', 'content_api_objectviewsstats.term_id=%s'], params=[category_id], tables=['content_api_objectviewsstats'], order_by=['-content_api_objectviewsstats.num_views'] ).prefetch_related('images', 'objectmeta_set').select_related('parent__parent'))

as can see this, it's normal request redirected designated view, fetching info mysql , returning serialized result. nil out of ordinary or complex processing.

executing:

ab -c 500 -n 5000 http://ip_here/api/nodes/mostviewed/9/

notice without caching. next logged:

harakiri: --- uwsgi worker 5 (pid: 31015) managing request /api/nodes/mostviewed/9/ since sat feb 16 13:07:21 2013 --- damn ! worker 2 (pid: 31006) died, killed signal 9 :( trying respawn ... respawned uwsgi worker 2 (new pid: 31040)

load average during testing goes ~ 5. , here's ab result:

concurrency level: 500 time taken tests: 251.810 seconds finish requests: 1969 failed requests: 1771 (connect: 0, receive: 0, length: 1771, exceptions: 0) write errors: 0 non-2xx responses: 1967 total transferred: 702612 bytes html transferred: 396412 bytes requests per second: 7.82 [#/sec] (mean) time per request: 63943.511 [ms] (mean) time per request: 127.887 [ms] (mean, across concurrent requests) transfer rate: 2.72 [kbytes/sec] received

first of all, 7 requests per sec disappointing. ~ 1700 failed requests due timeout errors because of performance lag facing here.

to honest. expecting ~ 60 - 70 requests per sec without caching. know caching speeds process hides performance issues have why pursuing solve before cache stuff.

i decided profile on vmware centos machine using django-profiling adding ?prof request shows phone call stack:

instance wide ram usage partition of set of 373497 objects. total size = 65340232 bytes. index count % size % cumulative % kind (class / dict of class) 0 2270 1 7609040 12 7609040 12 dict of django.db.models.sql.query.query 1 19503 5 6263400 10 13872440 21 dict (no owner) 2 63952 17 5739672 9 19612112 30 str 3 51413 14 5090344 8 24702456 38 list 4 58435 16 4991160 8 29693616 45 tuple 5 24518 7 4434112 7 34127728 52 unicode 6 8517 2 2384760 4 36512488 56 dict of django.db.models.base.modelstate 7 2270 1 2378960 4 38891448 60 dict of django.db.models.query.queryset 8 2268 1 2376864 4 41268312 63 dict of 0x14d6920 9 6998 2 2088304 3 43356616 66 django.utils.datastructures.sorteddict <619 more rows. type e.g. '_.more' view.> cpu time request 663425 function calls (618735 primitive calls) in 2.037 cpu seconds ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 2.037 2.037 /usr/lib/python2.6/site-packages/django/views/generic/base.py:44(view) 1 0.000 0.000 2.037 2.037 /usr/lib/python2.6/site-packages/django/views/decorators/csrf.py:76(wrapped_view) 1 0.000 0.000 2.037 2.037 /usr/lib/python2.6/site-packages/rest_framework/views.py:359(dispatch) 1 0.000 0.000 2.036 2.036 /usr/lib/python2.6/site-packages/rest_framework/generics.py:144(get) 1 0.000 0.000 2.036 2.036 /usr/lib/python2.6/site-packages/rest_framework/mixins.py:46(list) 1 0.000 0.000 2.029 2.029 apps/content_api/views.py:504(get_queryset) 1 0.000 0.000 2.029 2.029 apps/objects_stats/models.py:11(get_most_viewed) 23/21 0.000 0.000 2.028 0.097 /usr/lib/python2.6/site-packages/django/db/models/query.py:92(__iter__) 4/2 0.003 0.001 2.028 1.014 /usr/lib/python2.6/site-packages/django/db/models/query.py:77(__len__) 1 0.000 0.000 1.645 1.645 /usr/lib/python2.6/site-packages/django/db/models/query.py:568(_prefetch_related_objects) 1 0.002 0.002 1.645 1.645 /usr/lib/python2.6/site-packages/django/db/models/query.py:1596(prefetch_related_objects) 2 0.024 0.012 1.643 0.822 /usr/lib/python2.6/site-packages/django/db/models/query.py:1748(prefetch_one_level) 2288 0.007 0.000 1.156 0.001 /usr/lib/python2.6/site-packages/django/db/models/manager.py:115(all) 6252 0.019 0.000 0.762 0.000 /usr/lib/python2.6/site-packages/django/db/models/query.py:231(iterator) 4544 0.025 0.000 0.727 0.000 /usr/lib/python2.6/site-packages/django/db/models/query.py:870(_clone) 4544 0.109 0.000 0.694 0.000 /usr/lib/python2.6/site-packages/django/db/models/sql/query.py:235(clone) 2270 0.004 0.000 0.619 0.000 /usr/lib/python2.6/site-packages/django/db/models/query.py:619(filter) 2270 0.013 0.000 0.615 0.000 /usr/lib/python2.6/site-packages/django/db/models/query.py:633(_filter_or_exclude) 1144 0.019 0.000 0.581 0.001 /usr/lib/python2.6/site-packages/django/db/models/fields/related.py:456(get_query_set) 1144 0.019 0.000 0.568 0.000 /usr/lib/python2.6/site-packages/django/db/models/fields/related.py:560(get_query_set) 55917/18180 0.192 0.000 0.500 0.000 /usr/lib64/python2.6/copy.py:144(deepcopy) 2270 0.003 0.000 0.401 0.000 /usr/lib/python2.6/site-packages/django/db/models/query.py:820(using)

and can see, of time spent on django views & drf views.

can point out if doing wrong here? why requests slow? python / django scale? read how many requests / sec should expect on simple db read & rendering operation such 1 doing?

does python / django scale

django powers massive services, such disqus , instagram, yeah, scales fine.

i decided profile this

as can see entire time spent within .get_most_viewed() method, looks issue you've got. (could wrong there, profile suggests 2.028 out of 2.037 of time spent there, 99.6% of time)

my orm skills aren't figuring out how ought dealing complex query, , in case need see model definitions, need debugging , simplifying query, rather looking @ other parts of view.

you might want drop django shell using manage.py shell, can profile particular query in exclusion rest of view.

also maybe seek getting help on how can simplify query here, or on django irc channel or django mailing list (you'll have more luck if inquire specifically query, rather more general django/rest framework phrased question here, of doesn't appear relevant issue seeing.

hope helps point in right direction getting resolved.

python django profiling mysql-python django-rest-framework