Andy Young's Blog

Problems with Twitter's new Retweet - and a tip/fix

“If someone retweets you (new-style), you won’t see it (it’s not in your @-replies or main feed) - even if you follow them!”

My twitter feed is a bit of a mess right now after I discovered a bunch of things in quick succession and typed something wrong initially - so here’s what’s annoying me about Twitter’s new (built-in) retweet function:

1. When someone retweets you (new-style), it doesn’t show up in your @-replies list. This also means it doesn’t show up in your list of mentions on clients that use Twitter’s mentions feed e.g. Tweetdeck.

2. When someone that you follow retweets you (new-style), it doesn’t even show up in your own feed i.e on the Twitter.com homepage when you’re logged in, or in your list of “all friends” in clients like Tweetdeck. For example, I follow Jake and he retweeted me earlier (another annoyance - I can’t link directly to his retweet of mine, instead the links now point back to my original tweet). However Jake’s RT of mine didn’t appear in my feed, so I didn’t notice until later, by chance.

The combination of 1 & 2 mean that it’s pretty difficult to find out when people retweet you, unless you’re really keen to keep checking manually (see below). This means you can’t judge when people are in agreement with you or interested in things you post, even if you’re friends and you follow them. You also can’t discover new people who retweet you - so much for discovering new connections!

The flip side of this is that when you retweet people, they’ll likely not get to know about it. So much for subtly attracting someone’s attention..! ;)

There is a “workaround” however - do a search for your Twitter username and all retweets will show up (well, almost - see below). I use Tweetdeck so it was pretty simple just to add a new column with a search for andyy. The problem with this is that it works most of the time - but a lot of people’s tweets are blocked from showing up in twitter search (as I’ve come to discover from the number of people that having problems trying to use my Selective Twitter Status app - Twitter seems to have a big problem being overzealous in it’s detection of spam-like activity or something. The end result is that this still doesn’t guarantee to show 100% of retweets.

I’m actually a big fan of the potential for the new retweet functionality - there’s already some cool stuff if you click on “Retweets” in the right-hand menu on twitter.com - “Your tweets, retweeted” shows when people retweet you, although it doesn’t show cases where you’ve been the “influencer” - where you retweeted someone else and that was subsequently noticed by people that follow you and retweeted by them. The “Retweets by others” that shows tweets that people you follow have retweeted seems a particularly cool way to discover stuff that your friends find interesting. Here’s hoping Twitter manage to get this all worked out soon!

PHP Trick: Catching fatal errors (E_ERROR) with a custom error handler

Implementing a custom error handler using set_error_handler() in PHP can be a useful technique (Google search for more info/examples)

Unfortunately, set_error_handler() doesn’t catch fatal errors - as the PHP docs say:

The following error types cannot be handled with a user defined function: E_ERROR, E_PARSE, E_CORE_ERROR, E_CORE_WARNING, E_COMPILE_ERROR, E_COMPILE_WARNING, and most of E_STRICT raised in the file where set_error_handler() is called.

Handily, I just discovered it’s possible to catch fatal E_ERROR errors and direct them to your custom error handler using a combination of register_shutdown_function() and error_get_last():

set_error_handler('myErrorHandler');
register_shutdown_handler('fatalErrorShutdownHandler');

function myErrorHandler($code, $message, $file, $line) {
  ...
}

function fatalErrorShutdownHandler()
{
  $last_error = error_get_last();
  if ($last_error['type'] === E_ERROR) {
    // fatal error
    myErrorHandler(E_ERROR, $last_error['message'], $last_error['file'], $last_error['line']);
  }
}

The key is that functions registered with register_shutdown_function() are called even on a fatal error - including out of memory errors. error_get_last() can then be used to detect whether we’re ending the script because of a fatal error, and pass the error info to your custom error handler if so.

Stickyness and success: Techcrunch has it backwards

Just read the guest post How to measure the true stickiness (and success) of a Facebook app by Nabeel Hyatt from Conduit Labs over at Techcrunch.

Interesting data, but the conclusion is backwards.

“You would expect stickiness to go down as you get huge”. yes, the table he gives shows this to be TRUE:

An increase in MAUs between 9/22 and 10/22 (col 4 > col 5) corresponds with a decrease in DAU/MAU ratio (col 2 < col 3) - and with one exception (RockYou Pets) the opposite is also true - where MAUs went down, stickiness went up.

In other words, all the apps that got MORE users (grew in size) in the last month became LESS sticky. the last row (Cafe World) is the extreme example - it grew MAUs 15x in the month but went from 92% to 33% stickness.

The other point - that stickiness and success are correlated - is surely obvious. if you have a stickier app, your users will visit more often, and assuming it is even the smallest bit viral, this means that your users are spending more time being exposed to that virality and so will invite more friends, so over time you’ll get more total MAUs too.

Put these together and it suggests a different model:

1. stickiness depends on the individual characteristics of each specific app
2. stickiness decreases slowly as apps grow
3. apps with more stickiness are more likely to be successful…

1st month as a Spotify premium user

Spotify founder Daniel Ek has just written a very interesting blog post to mark the first anniversary of Spotify, giving his views on Spotify’s progress, the music industry, where they’re heading and what the journey will be like.

It’s now a month since I took the plunge and gave Spotify my credit card details. The tipping point for me was the release of their iPhone app allowing me to stream to my phone, in particular to download entire albums for offline play.

In the last month I’ve “borrowed” the following albums:

• Tiga - Sexor collector’s edition
• Tiga - Ciao!
• Simian Mobile Disco - Temporary Pleasure
• Justice - A Cross The Universe
• 2raumwohnung - Kommt Zusammen
• Soulwax - Most of the Remixes…
• DJ Kicks - The Exclusives
• Pulp - Different Class (2 CD edition)
• Muse - The Resistance
• Snow Patrol - Final Straw
• Snow Patrol - Eyes Open
• Daft Punk - Alive 2007
• Jet - Get Born

..as well as a couple of singles, and listed to a bunch of other stuff without saving it for offline play.

The verdict - in the first month certainly I’ve had fantastic value for money. My £10 has given me repeat listening to a whole bunch of music, mostly albums I hadn’t heard before and including 2 brand new releases (Muse & Simian Mobile Disco). Sure, this isn’t exactly the same as actually buying a CD or having an MP3, but I’ve not experienced any practical difference.

We’ll see how it develops, but right now I have to say it comes recommended, at least if you listen to music as much as I do, and particularly since you can cancel at any time..

The Alpha Course "Does God exist?" and online poll ethics

The Alpha Course in the UK are running a marketing campaign at the moment with posters asking the question “Does God exist?” followed by empty tick boxes for Yes/No/Probably. They’ve got the same thing on their homepage at http://uk.alpha.org/ set up as an online poll, complete with “see results so far” link.

Problem is..

Clicking yes always returns 36-34-30. Clicking no always returns 35-34-30. Clicking probably always returns 35-34-31.

The votes are being posted to and the “results” returned by http://uk.alpha.org/?q=/alpha09/poll&vote=[yes|no|probably] - view source to see the html returned with the “results”.

Amusingly you can get the stats as high as 40% “no” by adding lots of votes at once e.g. http://uk.alpha.org/?q=/alpha09/poll&vote=no&vote=no&vote=no&vote=no&vote=no etc. see here: http://bit.ly/somethingfishygoingon

Of course as soon as you vote anything else the numbers return to normal.

Regardless of your religious beliefs, surely this is pushing the line a bit far in terms of presenting it as a real poll when the results are fixed?

I don’t mean this to be focused on religion. The interesting question to me is “is it legit to host an online poll backed by a significant ad campaign and present fake results?” If it helps, try imagining that this is a poll by some other brand asking “Do you think our product is the best?” and presenting you with what they make out to be what other people think.

Seems to me like this is deceiving the public. How do poll results (or not) relate to advertising standards laws? Is this legal?

Say e.g. Apple hosted an online “poll” asking “Are our laptops the best? Yes/No/Probably” with a “see results so far” link. Would that be “just publicity material”? Where would they stand legally?

Update

Unlike me, James wasn’t too lazy to actually e-mail the webmaster - apparently “the result were being cached - [they] didn’t notice as caching is disabled for registered users.”.

The site currently shows a large percentage of noes - hopefully as well as fixing the caching they also ensured that multiple votes in the style of http://bit.ly/somethingfishygoingon are not added to the total!

PHP htmlspecialchars()/htmlentities() invalid multibyte/UTF-8 gotcha with display_errors=true

Here’s a seemingly nonsensical gotcha I just discovered with error handling for htmlspecialchars() and htmlentities() in PHP. Since PHP 5.2.5, if either of those functions are passed invalid multibyte strings (invalid UTF-8, perhaps containing a truncated multi-byte character after improper use of substr() intead of mb_substr()) then PHP triggers the following error and returns an empty string:

PHP Warning: htmlspecialchars(): Invalid multibyte sequence in argument

Now, in general the php ini setting display_errors can be used to control whether errors are output to the browser, the ini setting log_errors can be independently used to control whether errors are written to logfile, and if a custom error handler has been set with set_error_handler() then this is always called for all errors and can then read the values of display_errors and log_errors along with the value of error_reporting() and take the appropriate course of action, right?

Wrong! In this case, htmlspecialchars() and htmlentities() only trigger the error if the value of display_errors is false. If the value of display_errors is true then no error is triggered at all! This seemingly nonsensical behaviour makes it impossible to detect these errors during debugging with display_errors on.

Here’s the bug report, marked as closed - bogus, original modification to the PHP source that added this behaviour, initial fix for this behaviour and subsequent revert of that fix that I believe was incorrect.

I’ve contacted the core PHP developers involved but in the meantime this may help anyone searching Google..

Design by Contract - for efficient coding

Yesterday I picked up this tweet from Joel:

Getting into PHP Exceptions. Very useful :) Example - userExists() function should raise exception when no user passed, not return false!

Hang on, should it? While exceptions in programming languages are without doubt a useful tool, his example reminded me of a problematic style of coding I’ve experienced - something I consider an Antipattern. The style I have in mind is when each function starts with code validating the input parameters before the main work of the function is executed. Here’s an example:

function addUserToGroup($user, $group, $done_by) {
  if (!($user instanceof User)) { throw new Exception(..); }
  if (!($group instanceof Group)) { throw new Exception(..); }
  if (!($done_by instanceof User)) { throw new Exception(..); }
  // ok, we're (finally) good to go:
  $ref = new UserGroupRef();
  $ref->setUser($user);
  $ref->setGroup($group);
  $ref->setDoneBy($done_by);
  $ref->save();
}

If you’d never dream of writing code like this, move along - nothing of interest to see below! If you’re wondering why I think this is a bad approach, though - what’s the problem here?

The code is verbose - 50% is performing “meta” tasks. This takes time to write and makes it harder to maintain and interpret in the future. (I’m a big believer in self-documenting code - key to this is keeping code short and reducing the amount of unimportant information).

The issue gets stronger if we go on to include similar checks in the implementations of UserGroupRef->setUser(), setGroup() and setDoneBy(). The complete system is now “nice and robust” but has doubled in terms of lines of code, and you can imagine how when the code is executed the same checks are called repeatedly at different levels of the call stack. If those other functions are only ever called by code that has already ensured the parameters passed will be valid, surely all the time and effort to write and maintain the additional checks isn’t strictly necessary..

Design By Contract

So how do we tell when we need to do this validation? Enter the concept of “Design by Contract” - a “technique to reduce the programming effort for large projects” (source). The basic idea is that each function comes with a “contract” that should be agreed to when calling it, in the same way that two people may have a contract when agreeing to work together in the real world.

The key elements of Design by Contract are:

• a Precondition - what must be true before the function is called - i.e. the responsibilities of the calling scope. For example, the $user parameter will contain a User object.
• a Postcondition - what will be true after the function has executed - i.e. the responsibilities of the function itself. For example. $user will be a member of $group.
• a Class Invariant - what the function guarantees to leave unchanged (for completeness - so we don’t get unexpected side-effects).

The part I want to focus on is the precondition - specifically what parameters the function will be given - values, datatypes etc. Design by Contract places the responsibility on the calling code to provide valid parameters, not on the called code to validate them. The core idea is “fail hard” - if any part of the contract is broken the entire thing is void - and there’s no guarantee what will happen(!)

That obviously sounds quite scary - I think this interpretation of DBC is a relevant approach only for internal functions within a module or self-contained system. DBC is not a good model for dealing with user input at run-time(!) - and for open-source code or APIs that will be used by other teams or released publicly the extra code to check for and report violation of the contract is probably worth the saving in confusion and support questions.

Using Design By Contract for efficient coding

So the point is to use DBC as the justification for not filling your classes and methods with code validating input parameters, resulting in lengthy code that’s harder to understand and takes longer to maintain. You just need to ensure that every member of the team writing code that uses a particular module works on the principle that it is the calling code’s responsibility to provide valid parameters, something that is often possible with small teams and bespoke modules. DBC also simplifies debugging and unit testing since the intended behaviour of each routine is clearly specified.

Back to Joel’s example - should userExists() throw an exception if passed invalid input? The answer really depends on how it will be used, and what the “contract” for function is understood to be. The contract could be written that userExists() will take responsibility for validating it’s input. However it’s often the case that the calling code will also need to know if it’s not working with valid data, which indicates that validation is best done at the earliest point in the call stack than repeatedly throughout.

It’s also important to keep in mind what real-world effect invalid input may have - for some functions it could result in critical data corruption. However the chances are that when testing for the existence of an invalid user, simply returning false will cause the system to function fine as expected. This leads us to one of my favourite concepts of coding - YAGNI ;)

Comments welcome.

Reliability Bug in Facebook PHP API Batching Support

I recently discovered a bug in the current Facebook API PHP Client library that affects the support for batching calls which I’ve used for the Selective Twitter Status application.

The bottom line is, sometimes the call to batch.run within execute_server_side_batch() as called by end_batch() returns an empty string instead of the expected array of results from each of the batched calls. The code has the following test for an exception but this isn’t triggered in it’s current state since no specific FB exception is being returned, just the empty result:

    if (is_array($result) && isset($result['error_code'])) {
      throw new FacebookRestClientException($result['error_msg'],
                                            $result['error_code']);
    }

The symptom is a bunch of PHP warnings similar to the following as the code attempts to iterate over the empty string as if it was the expected array of results:

PHP Notice:  Uninitialized string offset:  0 in [...]/facebook/facebookapi_php5_restlib.php on line 210
PHP Notice:  Uninitialized string offset:  1 in [...]/facebook/facebookapi_php5_restlib.php on line 210

etc

In my experience the empty result has been in situations where the requested batch operations haven’t been performed, so the fix is to add a test for the empty result and to treat this as an exception. This was my hack:

 if (!isset($result[0])) {
            throw new FacebookRestClientException('Batch call returned invalid result: ' . var_export($result, true) . '; ' . var_export($xml, true), 9999);
    }

As a side note, the batching support as implemented in the current PHP API client is rather limited since if one of the batched functions returns an exception it is thrown immediately, with the result that the return values of subsequent calls are not populated and it’s impossible to know whether they executed successfully. I’ve changed the line that threw the exception to instead set the FacebookRestClientException object as the return value for the relevant call as follows:

    for($i = 0; $i batch_queue[$i];
      $batch_item_result_xml = $result[$i];
      $batch_item_result = $this->convert_xml_to_result($batch_item_result_xml, $batch_item['m'], $batch_item['p']);
      if (is_array($batch_item_result) &&
          isset($batch_item_result['error_code'])) {
        // PATCH: modify to return the Exception object as the result for this call so we can process all results/exceptions
        $batch_item_result = new FacebookRestClientException($batch_item_result['error_msg'], $batch_item_result['error_code']);
      }
      $batch_item['r'] = $batch_item_result;
    }

Hopefully this helps anyone searching Google..

If you can't find yourself in Twitter Search..

I just came across a recent help page from Twitter (from June) about what to do if your tweets are not showing up in the twitter search. As it already says on the Selective Twitter Status help page, your tweets not showing in Twitter search is one of the most common reasons for the app not working for you, particularly if it used to work but suddenly stopped.

To check if your tweets are coming up in Twitter search go to http://search.twitter.com/ and type in your twitter username. For example, my tweets should show up at http://search.twitter.com/search?q=andyy. If your tweets don’t appear, Twitter basically say to check Twitter isn’t having any problems at the moment, and if everything is otherwise fine (and your account is public, of course) then your tweets “have most likely been filtered out of our search index for quality reasons” which sadly isn’t much help (unless you really have been spamming, in which case you just gotta live with it.. ;)

As I mention on the help page, the last thing to try is to protect your Twitter account and then make it public again. You’ll need to post a new tweet after doing this to check if it worked.

New website..Tumblr it is

So, I finally got around to setting up a personal website that I’m not ashamed to show people..

I don’t really blog, but I wanted a home to stick links to my Twitter, FB, LinkedIn profiles etc for people that discover me online, collect links to show off when my projects receive press coverage etc as well as post interesting links and other notes from time to time.

Features I wanted:

• Easy setup, hosted solution, no coding required - I’m lazy, didn’t want to spend time on the technical side of a full Wordpress install etc.
• A service with lots of themes so I could find one I liked and do minimal customisations
• The ability to post stuff with tags, so in future I can set up separate index pages for stuff related to GroupSpaces, Selective Twitter Status, entrepreneurship etc
• My own domain name
• Suitable cool post-from-email/iphone/etc support
• All the above for free!

I looked at Posterous first but it doesn’t support all of the above right now. The winner is Tumblr which was quick to set up, does all the above plus the following bonus features:

• Designed for minimal blogging (posting short snippets, links and videos rather than extended blog posts) that suits the way I want to post
• Handy powerful iPhone app for posting
• Supports Markdown - a handy way of entering formatting quickly as you type

I’ve also integrated commenting powered by Disqus, which took only a couple of minutes and has a bunch of powerful features (sign in with Facebook etc). I think the default Disqus interface confuses visitors with too many features and wordy instructions so I’ve hidden some parts of the comments interface and tweaked some other stuff out of the way using custom CSS.

Let’s see how it goes..