Python-Twitter v1.0rc1 is ready

A bunch of folks have been working on getting the current python-twitter calls to work with the Twitter v1.1 API prior to the scheduled shut-off.

I can’t thank all of the contributors enough – their help and work made this release possible.

THANKS!

I won’t be updating PyPI until after someone other than myself gives a once-over, and then it will be released as v1.0rc1. After a good couple of days of testing, if everything looks good, i’ll pull the switch and release it as v1.0 – if you do find any errors or mistakes, please do submit a bug report.

After this we are going to plan a major refactoring to remove a lot of cruft that has been hanging around and look into some of the streaming api changes.

again, thanks!

Automating Node JS Testing when HAProxy is involved

Automating your NodeJS project’s testing is a solved problem if you’re working with a normal app that has a server.js process and listens on a socket for HTTP traffic. It gets even easier when you use the Ploy module from Substack – you can now push code to be tested to Ploy’s GIT service and your tests are run and QA is possible.

But what happens when you are dealing with an app that needs to sit behind HAProxy or some other alternative proxy-forward tool? Then it gets more complicated because Ploy tries to assign a random port to your app by setting the PORT environment variable. Often, your choice is to pre-configure HAProxy with dozens of ports, but then you also need to map sub-domains to those ports… it quickly gets complicated and your QA environment starts to impose a structure that isn’t really wanted or required.

The solution I came up with was to create a bash script to use for npm’s scripts.start in package.json that creates my HAProxy config dynamically and then tells HAProxy to restart itself. The ability of HAProxy to restart is not obvious at all and is currently not found in the docs but it works. Using scripts.start instead of scripts.prestart, or one of the pre- / post- variants is required because currently Ploy only sends the PORT environment variable to it’s spawn of scripts.start.

Ok, enough chatter, let’s see some code :)

This is the replacement for what most people have in their package.json for scripts.start:

#!/bin/bash
if [ -f /etc/haproxy/build_config ]; then
   if [ ! “${PORT}” = “” ]; then
   echo -e “backend ${BRANCH}_myapp\n balance roundrobin\n server myapp_${BRANCH} 127.0.0.1:${PORT} weight 1 maxconn 2500 check\n” > /tmp/backend-${BRANCH}.cfg
   echo -e “acl is_${BRANCH} hdr_beg(Host) -i ${BRANCH}\n” > /tmp/acl-${BRANCH}.cfg
   echo -e “use_backend ${BRANCH}_myapp if is_${BRANCH}\n” > /tmp/use-${BRANCH}.cfg
   sudo /etc/haproxy/build_config ${BRANCH}
   fi
fi
node server.js

The script first detects if it’s being run in the QA environment by checking if the HAProxy build_config script is present – this allows us to have the script always present and not hinder any non-QA environments.

If HAProxy is detected we write out the three pieces of information most HAProxy configs require when working with sub-domains: ACL, use case and backend definition. These will be combined by the build_config script into an HAProxy config.

The HAProxy build_script below and has the assumption that it is added to your sudoers list for the deploy user so they can only run it (that’s also why all of the files are placed into /tmp/* – it’s one place a non-privilidged user can write files).

#!/bin/bash
# usage: build_script BRANCH
if [ -f /tmp/backend-${1}.cfg ]; then
  cp /tmp/backend-${1}.cfg /etc/haproxy/conf.d/
  cp /tmp/acl-${1}.cfg /etc/haproxy/conf.d
  cp /tmp/use-${1}.cfg /etc/haproxy/conf.d
  cat /etc/haproxy/haproxy.base /etc/haproxy/haproxy.acls /etc/haproxy/conf.d/acl-*.cfg /etc/haproxy/haproxy.uses /etc/haproxy/conf.d/use-*.cfg /etc/haproxy/conf.d/backend*.cfg > /etc/haproxy/haproxy.cfg
  if [ -f /var/run/haproxy.pid ]; then
    service haproxy restart
  fi
fi

The core part of build_script is to combine different parts of an HAProxy config into a single config – we accomplish this by breaking it into a combination of a base template (haproxy.base, haproxy.acls and haproxy.uses) and then add all of the dynamic pieces at compile time. The *cat* command will do all of that for us, we just have to get the templates correct ;)

Our example uses the Ubuntu upstart method of service management and the upstart config for haproxy is:

# HAProxy
description “HAProxy”
start on runlevel [2345]
stop on runlevel [016]
env CONF=/etc/haproxy/haproxy.cfg
pre-start script
  [ -r $CONF ]
end script
exec /usr/local/sbin/haproxy -f $CONF -sf $(cat /var/run/haproxy.pid)

The magic part is the haproxy exec line – the option that does the required work to get HAProxy to stop the current process, drain its sockets and then switch to the new process is handled by the “-sf” command line option, we just have to pass it the older PID – which happens to be the current PID at the time the command line is evaluated.

Google Hangout chat and the missing XMPP

My friend Jason Salas (who is an amazing and smart man – you should follow him in all of his endevours) asked me to comment on a post on Google+ from the Android Authority about XMPP being dropped by Hangout(s). I wanted to repost what I said here for anyone who reads my posts but is not on G+.

From Google+

it’s a good summary about the issue with XMPP – I may quibble with some of his thoughts about how Google “helped” XMPP but not enough to reject the post completely.

We in the XMPP community have seen this before and we will continue to support everyone who wants to use a stable, robust and open protocol. In some ways it may be an odd blessing as Google did not support some of the more secure features of federation and now that the largest (public) client base is out of the picture we will be able to revisit some items that were previously put on hold.

I’m also sensing that Google itself may not be completely aware of what just happened, this may be a case of the marketing/product folks driving the release of a product(s) that were not yet set in order to meet the I/O deadline – which is in-of-itself another worrying point.

But yea, there have been a number of great non-XSF folks posting concerns and questions. The XSF is ready to help when Google realizes the full impact of what has happened ;)

XMPP features now missing from GTalk (aka Hangout Chat)

I posted a snarky Twitter message about not wanting to spam the people following me with the features I’m now missing from GTalk, but I now really want to track them so…

Features that are now missing

  1. Outside IM clients can no longer send messages to Google+ (aka GTalk, aka GMail) users
  2. Outside IM clients can no longer receive messages from Google+
  3. Presence updates to/from outside networks not allowed
  4. Invisible status is missing
  5. Unable to block other Google+ users from seeing my status
  6. Contacts are now in whatever order some silly person at Google thinks is accurate/friendly/useful instead of the at-least-it’s consistent alphabetical order
  7. The only way to remove someone from the active Contacts list is to … wait for it… block them!

ok, so it’s not the 15 that I said on Twitter but it’s still damn annoying

EDIT:
ok, so I may have been a bit harsh and in a really cranky mood when writing the above – here is a post from the G+ conversation where some of the community members correct me ;)

Buried in a conversation over on this post https://plus.google.com/u/0/107968525907303243288/posts/HFe3W7A9Dor is some comments from +Ben Eidelson about how XMPP support is still in the plans for Hangouts, just either not implemented or buggy.

So it could be that I’m just reacting poorly to some really bad coding and/or product management – don’t get me wrong, i’m still irked.

I just may be restructuring and changing my reaction over the next couple of updates.

Thanks to +Fernando Miguel for pointing that out.

and here is the conversation:

Ben Eidelson May 15, 2013

+Daniel Rose currently those settings affect who causes invites to you, not who shows up for you to send messages to. Once you start using Hangouts for a few days the list should re-order based on who you are contacting most often.

+Thomas Heinen Thanks for your report of the issue. Hangouts supports basic interop with XMPP, so you can-for the time being-continue to use 3rd party clients. It does not work the same way as Talk, and so I believe the issue you’re having with the XMPP bridge will not resolve in Hangouts.

I welcome Google as the new Borg

yea, that’s a subtle title alright :/

I posted this earlier on Google+ and decided to cross-post it here as I realized that it was more hipster ironic than I wanted to be (to post about Google changing things on the same G+ stream they are changing things to focus on.)

So here is a cut-n-paste of what I said over there

I just found out the hard way that GTalk on Android has been silently replaced by Hangouts and the do-no-evil geniuses have removed everything dealing with XMPP from it … everything

The “hard way” is from the fact that I did not get a server alert on my phone because Hangouts no longer supports outside messages from XMPP sources (also known as Federation.)

This has me all worked up in a lot of ways:

Personally – a lot of my friends are XMPP users so now I no longer have a single client for my messaging needs and this means that GTalk will not be my primary tool, heck I may not load it at all now.

Professionally – I’m a member of the XMPP Software Foundation and we have been very proud about XMPP being the core of a lot of commercial communication products (Google, Facebook, Microsoft, Cisco and many others) and even tho each commercial entity does odd things (heck, breaks them also) the one thing that stood out was Federation.

Now Google is removing a core part of their messaging and replacing it with something that is not an Open standard and hell, to be blunt, is a horrible implementation of a messaging system. The Hangouts support forum is full of people wondering, like me, what happened to many features they depended on and also asking WTF

So even tho I know Google engineers may not ever see this, I just have to ask: when will you be changing your motto from “do no evil” to “meh, we just do what we want because we are now the new borg”

yea, I’m pissed, pissed enough that I won’t be buying Android as my next phone and I’m going to be putting my full effort behind the infrastructure behind Firefox OS