Install gevent + greenlets on OS X
The last few years a lot more people have been focusing on async IO in Python. This is also referred to as “non-blocking IO”. Gevent is one project that makes it easy to do this sorta stuff. How it works is simple, you spawn up “greenlets” which are sorta like threads but very lightweight. You are spawning these in a single Python process, and single thread (which because of the GIL really can only have one synchronous flow of execution anyhow). So this normal Python process will start processing your greenlet(s) and when one hits a point where IO is performed, say it goes out and talks to Redis, execution yeilds back while IO is being performed and another greenlet is picked and then put into the running state. If you had a bunch of greenlets all waiting for IO then they could come back and start processing as soon as one got it’s output. The important thing to grasp here is only one process, and single thread of execution is happening, but if one of the greenlets hits IO rather than block we can start executing some other code, and process the piece of code waiting for IO to return when it is ready.
If you’re looking to try Greenlets and gevent and you are on a mac you should be able to follow these directions.
First install libevent via Mac Ports. If you do not already have Mac Ports then first install it. I hear you can also use homebrew but I’ve been a ports user and it’s more like the apt-get I’m used to.
sudo port install libevent
Now you should be ready to install gevent (which’ll install greenlets).
First set this flag so that gevent can build:
export CFLAGS=-I/opt/local/include
That’ll point the C compiler to the ports folder where the event.h (and other) files are.
I created a virtualenv for this project, therefore to install gevent I simply did a:
pip install gevent
After which I had gevent and greenlets ready to go! I advise you go try this example to get a very simple start using greenlets and gevent.
Grails broken list() on domain object, fetchMode eager
I am using Grails 1.3.7 and found a very confusing issue. You could possibly run into this problem, maybe you have and that’s why you’re here! A colleague of mine setup a domain class that had
fetchMode = [aClass: 'eager']
This is almost always wrong, as GORM when using ‘eager’ or the synonym ‘join’ for this value will do a very dumb thing. It will use an outer join from parent to children, then limit on that cartesian product! So what you say? Well if you do this:
SomeDomainClass.list(max: 10)
You will see some odd results. Basically say SomeDomainClass has many AnotherDomainClass and that is set up with eager/join fetch. Well since the query works this way you could get very odd behavior. If the parent class has ten children expect to only get the one parent back from list(), not ten instances of the parent! Indeed, that sucks.
So you are saying “Ouch, this one hurts!” There probably should be a giant red stop sign in the documentation that makes you stop, and then be drilled on the pitfall(s) this encompasses, followed by a multiple choice quiz, or something.
For completeness here is the SQL GORM/Hibernate generates for the ‘eager’ case:
select ... from test_case_execution_context this_ left outer join test_case_execution_context_test_case_execution_step_status stepstatus2_ on this_.id=stepstatus2_.test_case_execution_context_step_status_id limit ?, ?
You see how that is not what you wanted, right?
Luckily the fix is pretty easy, though you might not like it for any odd reasons. Use the property lazy: false!
static mapping = {
aMember lazy: false
}
BooYahhh! Now you will get the results you are expecting.
With the lazy set to false you get results like this:
select ... from test_case_execution_context this_ limit ?, ?
Oh but where are the child domain results? Well, they get mapped in with the JDBC connection open, right away, in separate selects. Yes you are going to be making a bunch of db calls, but it’s all up front using one JDBC connection. This is the way it works folks. On the bright side now offset and max work!
Charity: Nginx, Bind, Grails, SSL etc
I was given an opportunity to help my wife setup a website for a charity she is involved in. It wasn’t anything particularly earth shaking, just a single page with a form for donating money via the web. Her charity wanted to run credit cards manually, since they have a credit card terminal. We would be collecting card info and thus we needed to have the traffic encrypted.
First things first I went and registered the domain packsoflove.com which is the domain my wife had chosen. Next I needed to setup DNS information for the new site, basically point the domain to my name servers so that our domain would resolve to the proper IP address. I used the registrars web based software to input the two name servers I manage. To finish the DNS setup I went and configured BIND on my servers so that they properly resolved packsoflove.com and www.packsoflove.com to my server’s IP. Now we had a place in the world wide web, but there wasn’t anything there!
At this point we needed an app. One that would be located at the domain we just setup. Since we where going to be processing form data it made sense to use a web application framework as opposed to a static site. The question was which framework to use? These past 10 or so years I’ve dived into a ton of them, writing apps of all size in numerous languages. For this task I choose Grails, partly because I’m going to be using it at my new job, but also because it’s very modular. You can install plugins for doing things like email or security and be done with it in minutes. Other choices that may have worked are Django, or Rails.
So I built the webapp, and it took a couple hours. I was having to prime myself a bit, I hadn’t done any Grails in over two years. I built a form for entering the card info that posts to a simple action which then emails off the data (securely of course) to my wife. I also built one more very very simple page that basically says thanks for donating and links back to the first page. So now I built the application and uploaded it to my server.
Grails runs on the Java language, and also uses most of the standards that Java has made famous (or is it infamous?) Anyhow you build a WAR file (short for webapplication resource file) which you install in an application container which serves up the app. In this case I used Jetty. It’s pretty easy to install via apt-get if you are using Ubuntu. Once installed you can put applications in the /usr/share/jetty/webapps folder. So I put the WAR file into the webapps/ folder, then rebooted Jetty. All good, except Jetty runs on localhost, which means it’s not accessible from the internet! What I had to do now is setup our proxy server, Nginx, so that it handles web requests for packsoflove.com and proxied them back to the Grails application running in Jetty, Pheeew!
My Vim
Vim, let me count the ways I love thee. If not for ye I would have fell so short in my text editing.
I use pathogen, which makes installing, updating and especially uninstalling\ plugins a breeze. You can even checkout plugins from Git or Mercurial repos real easy and then when you need to update you just use the client to pull down the latest updates. To use it follow the directions on the site, putting the bundle in .vim/autoload.
I also use the pyflakes-vim plugin, which makes keeping python files clean. Unused imports and variables will be highlighted and explanations given, so you don’t end up with a ton of cruft when you refactor. To install it just get the latest zip file and unzip it into .vim/bundle (if you are using pathogen).
Anyways here is my config.
filetype off call pathogen#helptags() call pathogen#runtime_append_all_bundles() filetype plugin indent on set nocompatible set modelines=0 syntax on set tabstop=4 set shiftwidth=4 set softtabstop=4 set expandtab set encoding=utf-8 set scrolloff=3 set autoindent set showmode set showcmd set hidden set wildmenu set wildmode=list:longest set visualbell set ttyfast set ruler set backspace=indent,eol,start set laststatus=2 set relativenumber set undofile nnoremap / /\v vnoremap / /\v set ignorecase set smartcase set gdefault set incsearch set showmatch set hlsearch nnoremap <leader><space> :noh<cr> nnoremap <tab> % nnoremap <tab> % set nowrap nnoremap <up> <nop> nnoremap <down> <nop> nnoremap <left> <nop> nnoremap <right> <nop> inoremap <up> <nop> inoremap <down> <nop> inoremap <left> <nop> inoremap <right> <nop> nnoremap j gj nnoremap k gk nnoremap <leader>b :buffers<cr>
Installing Jenkins on Ubuntu
Really for most experienced admins this and this should suffice . After install I changed the port Jenkins runs on. To change the port I altered the /etc/defaults/jenkins file changing the ‘HTTP_PORT’ value to 11939. Yeah that’s just an arbitrary port you could pick whichever you like. We run Nginx, so I’ll elaborate on how I made Jenkins available from anywhere. So in my case I run Jenkins on it’s own domain, hudson.blueleftistconstructor.com, and so the Nginx conf was really simple, here it is:
server {
listen 46.4.29.134:80;
server_name hudson.blueleftistconstructor.com;
location / {
proxy_pass http://127.0.0.1:11939;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
So that was all it took, reloaded Nginx and I was off. If you use Apache or some other proxy server you’re on your own in figuring out how to hook in Jenkins.
Now one thing to watch out for, if you make your Jenkins open to the ‘nets. It doesn’t have any security setup by default. You’ll want to use it’s built in security database to setup some users. This’ll save you from opening Jenkins one day and finding that someone came along and messed with your jobs.
Luckily this is pretty easy. Go to “manage jenkins > configure system”. Look for the “Access Control” section, then select “Jenkins’s own user database”. In the “Authorization” section below select the “Matrix-based Security” option. Enter a new user, given the name you prefer. Add the user then select ALL the check boxes in the row besides it. For the ‘Anonymous’ user I suggest selecting just the ‘read’ permission in the “Overall” category. Now you’ll want to save. Then you’ll see on the front page a link with title “signup now”, click it and sign up using the username you just created.
Now you should be all good to go. You should sign in and start creating some jobs!
BitBucket and Jenkins
BitBucket is a great place to host your open source projects; Jenkins is a great tool for keeping track of how buildable your projects are. Jenkins has become a tool used across many different languages, and it’s easy to install on any newer Ubuntu install. BitBucket hosts Mercurial project’s; Mercurial being a light weight distributed source code management tool. You probably now want to know how to build projects hosted in BitBucket via Jenkins, here it comes!
It’s really easy in fact. First off install the mercurial scm plugin for Jenkins. I’m not going to go over step by step on how you install a plugin in Jenkins. You should read up on Jenkins. Once that is done you’ll want to mosy on over to the sys config screen (manage jenkins > configure system) so you can setup a mercurial binary for Jenkins. On Ubuntu I was able to fill in the name field (I used mercurial1.4.3) and the ‘Executable’ field with ‘/usr/bin/hg’. Scroll down and save. You now should be able to setup a BB project.
Create a new job. In my case I was creating Maven 2/3 projects. Once you’ve done that you’ll configure as you like, when you get to the source code management section select Mercurial and then select the binary you created just a bit ago. In the ‘repository url’ field I put in the value you find on the overview in BB, say ‘https://bitbucket.org/joe’/someproject’. In the ‘repostiory viewer’ drop down select BitBucket, then in the ‘URL’ field that will appear put in the page in BB where the overview of the project lives. For example something like ‘https://bitbucket.org/joe/someproject/overview’.
Now you’ll probably want to poll for changes, so that not long after pushing up new code it’ll be checked out in Jenkins and run through. Easy. In the section ‘Build Triggers’ you’ll want to check the ‘Poll SCM’ box and in the ‘Schedule’ put in a logical value. These are cron expressions, for my case I used ‘*/5 * * * *’ so that Jenkins polls every five minutes looking for changes.
That’s about it. Setup the rest of the job’s settings as you wish and then save. You should be able to build the project right away to verify that it is working. Now sit back and enjoy the info Jenkins provides on builds coming into BitBucket.
Migrating the Site
If things are a little buggy around here it’s because we are doing a large migration of the site from one machine to another. There are a lot of apps we are using, and it’ll take a bit of time (2 days is my guess) before everything is back to normal. Bare with us.
Mac Ports Load / Unload a Process (like postgres)
This’ll be quick. If you use MacPorts on your OS X machine to install and manage *nix’y apps like MySQL, Postgres, Redis, Memcached etc you’ll want an easy way to start and stop them. On Ubuntu (or other *nix envs) you might have used /etc/init.d/somescript to start and stop an app. In OS X with MacPorts you’ll have to try something else.
MacPorts has the load and unload commands, which work with the Mac’s launchctl app to start and stop things. So once I have installed postgresql90-server I can start it via “sudo port load postgresql90-server”. Then you can try “ps aux | grep postgres” to find the process is running. This works for many of the other apps you will install. Look in the /Library/LaunchDaemons folder to see what plist files you have installed. For any one port load/unload will work if you give it the last part of the plist file name prior to the plist extension itself. So for org.macports.memcached.plist you can do a “sudo port load memcached”.
Creating a Quick and Dirty Parser with LEPL
I’ve been working with a behavior driven development tool named Lettuce. It’s built using Python and meant to test Python code. More or less it’s a Python based version of Ruby’s Cucumber. To make this blog post even more hard to follow, I’m using lettuce-webdriver, which builds on lettuce to allow writing tests that utilize Selenium and exercise a web application. Pheeew!
The end result is it’s really quick to write tests that actually open a web browser, click around, and verify the results. These tests are in English too! An example of a lettuce-webdriver test would be:
Scenario: Filling out the signup form
Given I go to "http://foo.com/signup"
When I fill in "Name" with "Foo Bar"
And I fill in "Email" with "nospam@gmail.com"
And I fill in "City" with "San Jose"
And I fill in "State" with "CA"
And I uncheck "Send me spam!"
And I select "Male" from "Gender"
And I press "Sign up"
Then I should see "Thank you for signing up!"
This isn’t hard to understand, right? Most non-programmers that are web literate could follow this. That is what is so nice.
So on to what this post is really about. As I’ve been hacking on webdriver I naturally crossed over here and there into lettuce itself. I noticed there wasn’t a simple parser that can take the .feature files and parse them into an abstract syntax tree, so I figured hey this will make a nice little project. I asked some colleagues what Python parsing libs they’d recommend and one that came up was LEPL.
After looking at the neat little demo on the front site I installed LEPL and got to playing around with it. I then went into the actually tutorial and followed through it. After doing so I was able to start on my little feature parser.
It didn’t even take a day to put this parser together. It’s not really done yet, I have yet to include the various multi-lingual literals that lettuce supports but that is just busy work. Also I think I need to build better lexing into it so that it makes sure the number of keys is equal to the number of values in outlines. You can see the end result here.
For quick and dirty jobs LEPL seems to be a great idea. It can parse simple stuff really easily. I’m sure it can do more, I’ve just skimmed the surface; to it’s credit it’s amazing how easy it is to pick up and get something done. Having used Java’s ANTLR (a much more full featured parsing/lexing library) I found LEPL to be really simple to get up and running, no special IDEs or plugins needed, just a Python console.
EDIT: I figure I should show the parser in action, here is the parsing of a simple feature and what the AST looks like.
>>> from lettuce import parser
>>> from lettuce.parser import print_ast
>>> from tests.unit.test_parser import featurestr
>>> ast = parser.feature.parse(featurestr)
>>> print_ast(ast)
Feature
+- FeatureDef
| `- 'I go to the bank'
+- Explanation
| `- 'some eplanation why\nmoar explanation why\n'
`- Scenarios(...)
>>> print_ast(ast[0])
FeatureDef
`- 'I go to the bank'
Explanation
`- 'some eplanation why\nmoar explanation why\n'
Scenarios
+- Scenario
| +- ScenarioDef
| | `- 'I drive car'
| `- Steps
| +- Step
| | `- 'When I go to garge'
| +- Step
| | `- 'Then I get into and start car'
| `- Step
| `- 'Then I put up garage and drive'
`- Scenario
+- ScenarioDef
| `- 'I use atm'
`- Steps
+- Step
| `- 'When I go to the bank'
+- Step
| `- 'Then I put my card in atm'
`- Step
`- 'And I get some money out'
OS X, Vim and PyFlakes
I find the Vim PyFlakes plugin to be a must when using vim to edit Python code. Essentially it tells you about all the nasty stuff like unused variables and imports that otherwise prove annoying to clean up; clean up that in my experience myself and others just end up not bothering to do. Normally this is a snap to install (meaning on Ubuntu and other *nix flavors) but I’ve found in OS X for whatever reason it’s got a few nasty bits. After you have downloaded the plugin and put the files in the appropriate locations you’ll need to make sure you have a properly built vim with python support. You can do this with Macports, but you’ll need the +python26 flag to make things work, +python25 will fail. I used the following to get a newer Vim installed recently with support that works with the PyFlakes plugin
sudo port install vim @7.3.146_0+python26
It’ll take a bit to build, once it has, open vim and check the version it reports. If all is well you should be able to open a python file now and see highlighted areas that are being reported. Move over them to get info on why the area is flagged. Enjoy!