Speeding up / optimising / tuning Tendenci

Register or login to post to the forum.
09 Oct, 2016 01:21

Hi, I'm trying to find and fix speed issues with our tendenci install now we've gone live and had a chance to observe real traffic.

Key things:

  • set up according to install guide, servers ubuntu 14.04, 2gb ram
  • two servers, one tendenci/gninx/memcache and one postgresql So far I've taken a number of steps:

  • fixing 404s

  • memcache enabled and 512mb of ram allocated on web node (though it appears virtually none is in use)
  • linking directly to https wherever feasible (we redirect all http -> https so non https adds a small amount of latency)
  • ran pgtune on our db server (which helped noticeably from the defaults but is likely to be quite crude still)
  • adjusted caching in nginx so "location ~ /themes/([a-zA-Z0-9-_]+)/media/" is cached for 30d
  • turned on gzip compression for a slew of filetypes
  • turned up lifetime on postgresql connections to 5 minutes (not sure about this one)

Depite all that we're seeing ~800ms TTFB on requests, most things (images/css/js/...) are downloaded pretty quickly once the html is down.

Other untried sources of improvement I've come across are:

  • linking to files directly rather than via /files/number (could save ~200ms per file)
  • minifying rendered html using something like django-htmlmin (i attempted this and it resulted in a question on GitHub - https://github.com/tendenci/tendenci/issues/519)
  • changes to caching https://docs.djangoproject.com/en/1.8/ref/templates/api/#django.template.loaders.cached.Loader
  • on the fly minification of various filetypes http://django-pipeline.readthedocs.io/en/latest/installation.html (possibly not useful to us with static assets)

Linking directly won't won't change our TTFB, the only option there which might help (from my understanding) is the caching section. After all that My questions are:

  • Are there any built in tools/add on tools we can use to profile/benchmark tendenci to find where it is falling over?
  • Is there existing documentation on Speeding up / optimising / tuning Tendenci?
  • If yes, where is it?
  • If not, what other steps can you suggest for finding and fixing our performance problems? thanks, kk PS. For anyone else looking at this, the following are helpful for tracking down speed issues

  • https://developers.google.com/speed/pagespeed/

  • http://yslow.org
  • https://developer.chrome.com/devtools#improving-network-performance
Edited 09 Oct, 2016 01:24
14 Oct, 2016 22:53

@kgoetz

You are correct. There is a LOT of room for improvement. I know because I see it over and over using django debug toolbar and see the ridiculous number of queries for a view page. Something that would be untenable without django template caching and memcache - but it still means the code isn't clean.

Some low hanging fruit for expectations - depending on the modules enabled you probably need to consider allocating more room for RAM given most cloud providers don't "really" have swap. Check if KVM or GP

Check your settings for shmax* settings and /etc/sysctl.conf and /etc/default/grub - but especially look at your /etc/nginx/nginx.conf and /etc/nginx/sites-enabled

The docs still say "gunicorn". You really want to run uwsgi instead.

In production we've moved entirely to dockers and have been running there for several years for everyone on 6.0 and up. What I have not done is pushed out the dockerfile because currently I'm shifting us from docker-engine to docker-engine on 16.04 with docker-swarm enabled and using multiple sockets for communication as opposed to docker-gen.

Which then leads to two questions: 1) When will we publish the dockerfile? When we are confident it is secure and works without causing a flurry of inbound communication when we are just starting to grasp the magnitude of the aufs to overlayfs switch. 2) Until then - caching and pull requests are your best bet. For example: - eventlogs is SUPPOSED to be similar to regular log files recorded when multiple objects are in memory. They are most definitely NOT supposed to be relational at all as that defeats the purpose of an audit trail. This could be done with log files to elasticsearch or keeping it in the database but REMOVING the "JOIN" which slows things down. - has_perm - the bane of my existence - but my fault for not blending security models.

I have other speed suggestions if I know more about your environment. Like if you put it behind a reverse proxy just blocking the port scans will radically speed up the code. But there is no question the code needs optimization.

And finally - you hit a lot of topics, most of which I'm aware of and it stings to read but I needed to hear it. I could use some help categorizing them and adding them as distinct items on github.com/tendenci/tendenci/issues .

Milestone suggestions are also appreciated so we can keep it organized. Several clients have been applying for grants to improve the core - which I believe should be transparent and pushed out to developers in the community as options to increase security and speed of improvement. We aren't there yet - only because we need to hear more wisdom from you and others in the community on how to improve!

What can I do to help organize this with you? I'm open to a skype call with folks, perhaps some clients in the mix as well, people who care. I tried IRC but couldn't keep up. Skype project calls have worked in the past with a core group but open to all. The project clearly needs an advisory board that is engaged (again, tried, but that was before we had critical mass so it might be time to try again?)

I'm grateful for your comments. I'm also overwhelmed seeing them in a single post. Help!

Ed smile

14 Oct, 2016 23:31

Hi Ed! (Could you check the contact us form? I sent a message a few days ago smile)

As an update to my original post:

linking to files directly rather than via /files/number - we do that now and there is a noticeable improvement in page speed minifying rendered html using something like django-htmlmin - jenny helped me with my issue and htmlmin is now working. It certainly saves some space! For the records: wc -l index.html* 624 index.html # pre minification 36 index.html.1 # post minificatino changes to caching - I added a number of {% cache %} blocks throughout my templates but didn't see any performance improvement. on the fly minification of various filetypes - I've been trying django-compressor but have hit what I think are some bugs. I haven't tried django-pipeline yet. - I just opened https://github.com/django-compressor/django-compressor/issues/799 about the problem I'm having - if you have any suggestions I'd love to hear them!

Following the success with htmlmin I'll try doing some late night testing with the debug bar turned on too.

I've also gone redirect hunting and found a number of the pages in our nav bar had redirects. Our nav was to /contact-us which redirected to /contact-us/ which then rendered. Fixing these helped chop a little bit of load time from the respective pages.

Now on to the reply!

While I suspect these aren't affecting my our site, these passages sound like the might be helpful to have somewhere in the documentation; after a little bit of expanding to add some rule of thumb stuff. If you can throw the basics here I'll try and bash them in to shape for a PR.

Check your settings for shmax* settings and /etc/sysctl.conf and /etc/default/grub - but especially look at your /etc/nginx/nginx.conf and /etc/nginx/sites-enabled The docs still say "gunicorn". You really want to run uwsgi instead.

(Obviously the bit about uwsgi vs unicorn isn't a pure documentation fix as it is a system change but if its somehow a superior way of operating I'm happy to move our instance over).

Our environment is fairly vanilla, I'm happy to go in to more detail in a private channel because its starting to evolve.

I'm happy to open GH issues (if not PRs) for speed problems but given the size of the problem (based purely on your comments above, no snakiness intended) having me open them adhoc might not make sense.

I'm going to cover off the bit about community engagement very briefly here, but yes I think its important and should be considered. Even if it can't be a priority. If you'd like to have a call about that or the technical stuff I'm happy to be involved in one ; we should organise that privately (you will still have my details on file and they are in the contact i made a few days ago).

thanks, kk

Edited 15 Oct, 2016 03:05