i.com.pk

This blog post is continuation of Part-I. The sample data is increased to 150K Pakistani tweeps now.

Followers

Follower count is no longer a good influence measure. On average each Pakistani tweep gets followed by 129 users. Majority of Pakistanis (about 3/4th) have less than 50 followers. Half of Pakistani twitter users have less than 10 followers. There are about 10,000 tweeps with no follower and about 12,000 tweeps with single follower. This is a very strange trend. If you look deeply into these accounts, you’ll notice that most of them are with default DP and default background. It seems like these are fake accounts, created by social media cells of different political parties to increase follower count of their leaders on twitter.

On the other side, there are just 24 Pakistani’s with more than 50,000 followers. Most of them are politicians and TV anchors. Just 2331 tweeps have more than 1000 followers.

Klout

Klout is more reliable social media influence measure. Out of 150,000 Pakistani tweeps about 40,000 do not have any klout score. About 70,000 have their klout between 11-20. Average klout score is 16.72. About 12,000 have the minimum possible score 10.

Only 22 users have scored above 70 score.

Here is the list of most influential Pakistanis (klout: 70+)

• @etribune 84.38196925
• @SenRehmanMalik 81.16218685
• @RealAliAzmat 81.04070986
• @Razarumi 80.45987283
• @husainhaqqani 80.34610752
• @robcrilly 79.9045397
• @NasimZehra 79.81927076
• @nadeemmalik 79.77388391
• @MubasherLucman 78.87619785
• @sanabucha 78.40538301
• @fasi_zaka 77.92095993
• @marvisirmed 77.49454349
• @DrAwab 73.68917445
• @ayeshahalam 73.56226489
• @ImranKhanPTI 73.5499122
• @akchishti 73.19814707
• @afiasalam 71.77437875
• @KamalFaridi 71.0673759
• @AliAbbasTaj 70.96716798
• @HamidMirGEO 70.72710924
• @umairquraishi 70.637669
• @OmarWaraich 70.59262722

Note: This score may have changed when you’re reading this article.

For this analysis, description of about 150,000 Pakistani tweeps was used. Out of 150K only about 77K (about 51%) users have set description field in their twitter profiles.

Excluding punctuations and stopwords, following is the list of most commonly used words by Pakistani tweeps in their profiles.

1. love
2. pakistan
3. student
4. follow
5. like
6. life
7. pakistani
8. engineer
9. cricket
10. social

Technology used was FreqDist and stopwords of nltk.

Background

I do not want to start this blog post by bashing Posterous. Posterous is a great blogging tool for quickly making blog posts. A couple of years back, some of its unique features convinced me to move my blog from wordpress to posterous. Posterous offered custom domain name for free whereas wordpress was charging for it. I really liked the email to blog post feature, although I never used it other than testing it a couple of times. Another amazing feature of posterous was detecting and making beautiful widgets for external objects like YouTube, github gist etc.

Posterous provides some nice templates but I wanted to have more control over presentation. A few days back youtube was blocked in Pakistan. Some misconfiguration caused problem in loading other google sites. This affected the sites using resources from google like javascripts for google analytics, google maps etc. Same thing happened with my site. The template, I was using was consuming some resources from google. I don’t know why but it was there and there was no way to remove it. So, the end result was a slowly loading page for Pakistani audience.

Another problem was how posterous modifies the HTML of the blog post. Again, I wanted to have more control on my blog post presentation. Inserting a table in a blog post was a trivial task. The WYSIWYG editor cannot handle table, even if it is copy pasted. I had to manually draft HTML and paste it in HTML part of the WYSIWYG editor. And it gets modified when rendered :-(

Using SSGs

The idea of SSG is amazing. Why do I need a dynamic setup for content which is hardly going to be modified in a month. I tried Jekyll & Pelican and decided to use Pelican. Why Pelican? It was because I am more biades towards Python. Jekyll is an equally good or may be better SSG.

Being a geek, I like writing in plain text editors more than WYSIWYG editors. Writing in Markdown and reStructuredText is fun. One can keep his energies focused on writing rather than formatting the content. My content is saved as content not as HTML markup. It has better revision management using git or any other version control system. This can easily be imported to any other application. The content is saved in files, not in DB. I can write offline and publish when I am online.

I have full control over the page rendered. I can design and optimize it as I want. I do not have to worry about security or scaling as all the content is purely static.

User Experience & Minimalism

I am not a UX expert but I do not want a lot of distractions in my content. Here is what I did to improve UX:

• Removed comments, users can tweet the feedback.
• No facebook like or twitter tweet button.
• No tags, category or author name with each post.
• Using grey instead of pure black for text.
• Worked on typography
• Using typogrify

Migration

Jekyll provides a posterous importer but Pelican does not. Currently pelican provides only following imports:

• WordPress
• Dotclear
• RSS/Atom feed

For posterous I had to write my own importer which consumes Posterous API. Here is the code:

def posterous2fields(api_token, email, password):
    """Imports posterous posts"""
    import base64
    from datetime import datetime, timedelta
    import simplejson as json
    import urllib2

    def get_posterous_posts(api_token, email, password, page = 1):
        base64string = base64.encodestring('%s:%s' % (email, password)).replace('\n', '')
        url = "http://posterous.com/api/v2/users/me/sites/primary/posts?api_token=%s&page=%d" % (api_token, page)
        request = urllib2.Request(url)
        request.add_header("Authorization", "Basic %s" % base64string)
        handle = urllib2.urlopen(request)
        posts = json.loads(handle.read())
        return posts

    page = 1
    posts = get_posterous_posts(api_token, email, password, page)
    while len(posts) > 0:
        posts = get_posterous_posts(api_token, email, password, page)
        page += 1

        for post in posts:
            slug = post.get('slug')
            if not slug:
                slug = slugify(post.get('title'))
            tags = [tag.get('name') for tag in post.get('tags')]
            raw_date = post.get('display_date')
            date_object = datetime.strptime(raw_date[:-6], "%Y/%m/%d %H:%M:%S")
            offset = int(raw_date[-5:])
            delta = timedelta(hours = offset / 100)
            date_object -= delta
            date = date_object.strftime("%Y-%m-%d %H:%M")

            yield (post.get('title'), post.get('body_cleaned'), slug, date, 
                post.get('user').get('display_name'), [], tags, "html")

The above code produced pelican fields which can later be passed to fields2pelican which uses pandoc to tranform html content to markdown or reStructuredText.

Deployment

The site is deployed on heroku Cedar Stack which supports Pyhton applications. It is served from great wsgi app called ‘static‘, gunicorn and gevent.

Update: Using my own fork of static for performance tweaks.

Here is the list of domain managed by MarkMonitor and have their nameservers pointing to dns2.freehostia.com & dns1.freehostia.com

1. biofreeze.com.pk
2. blackstone.pk
3. blogspot.pk
4. itunes.pk
5. gmails.pk
6. zynga.com.pk
7. chrome.com.pk
8. chrome.pk
9. visa.com.pk
10. bx.com.pk
11. abbvie.com.pk
12. abbvie.pk
13. cgma.pk
14. chacos.com.pk
15. cimacpa.pk
16. cisco.pk
17. ciscosystems.pk
18. blogspot.com.pk
19. cpacima.pk
20. cpaintl.pk
21. cpaldglobal.pk
22. cpalwglobal.pk
23. drivealliance.pk
24. eastman.biz.pk
25. eastman.net.pk
26. eastman.org.pk
27. ebay.pk
28. monatin.pk
29. everyblock.pk
30. youtube.pk
31. 3com.web.pk
32. hp.web.pk
33. revlon.pk
34. streetwear.pk
35. windows7.pk
36. windows8.pk
37. windowsrt.pk
38. yahoo.pk
39. yahoomaktoob.pk
40. zynga.pk
41. firstdirect.com.pk
42. flickr.pk
43. fordgofurther.pk
44. gbuzz.pk
45. gmailbuzz.pk
46. gmail.pk
47. googlebrowser.com.pk
48. google.pk
49. googlebuzz.pk
50. googlechrome.com.pk
51. abbviepharmaceuticals.pk
52. abbviepharmaceuticals.com.pk
53. hewlettpackard.pk
54. hexagon.com.pk
55. hsbcamanah.biz.pk
56. hotmail.com.pk
57. hpcloud.com.pk
58. hp.com.pk
59. hpscalene.com.pk
60. hsbc.biz.pk
61. hsbcadvance.com.pk
62. hsbc.pk
63. hsbcpremier.com.pk
64. hsbcprivatebank.biz.pk
65. hsbcamanah.com.pk
66. hsbcdirect.com.pk
67. hsbcnet.com.pk
68. hsbcpremier.biz.pk
69. hsbcpremier.pk
70. hsbcprivatebank.com.pk
71. investdirect.biz.pk
72. investdirect.com.pk
73. ipod.pk
74. jaiku.pk
75. kellyservices.com.pk
76. maktoob.pk
77. markmonitor.pk
78. microsoftsmartglass.com.pk
79. microsoftsmartglass.pk
80. xboxsmartglass.com.pk
81. xboxsmartglass.pk
82. msn.org.pk
83. windowsstore.pk
84. windowsstore.com.pk
85. opteron.com.pk
86. parkplaza.pk
87. paypal.pk
88. postini.pk
89. scalene.com.pk
90. schwab.biz.pk
91. schwab.com.pk
92. sonystyle.com.pk
93. streetwear.com.pk
94. theworldslocalbank.com.pk
95. genapp.pk
96. genapp.com.pk
97. generationapp.pk
98. generationapp.com.pk
99. windows.com.pk
100. windows7.com.pk
101. windows8.com.pk
102. 3com.biz.pk
103. 3com.fam.pk
104. 3com.net.pk
105. 3com.org.pk
106. gchrome.com.pk
107. aicpacima.pk
108. apple.pk
109. google.com.pk
110. microsoft.pk

According to some reports there are about 1.9M twitter users in Pakistan. This was mentioned by someone in #SOCMM12 but there doesn’t seem to be any source of this information.

I had been collecting twitter data for quite some time. Sample data contains more than 100K Pakistani twitter users crawled using twitter API. Only public profiles who have mentioned Pakistan or some pakistani city name in their profile were considered for this analysis. This data contains almost all active tweeple of Pakistan.

Here are the results of data analysis:

Gender:

gender.c, with custom names database of more than 5,000 names, was used on first names in twitter profiles:

According to www.stopbadware.org and ESET following government websites contain malwares and they are NOT safe for your computer.

• phc.gos.pk (Shaheed Benazir Bhutto Housing Cell)
• gulbergtownlahore.gov.pk (Gulberg Town Lahore)
• wasafaisalabad.gop.pk (WASA Faisalabad)
• moip.gov.pk (Ministry of Industries)
• khushabpolice.gov.pk (Khushab Police)
• sped.gos.pk (Special Education Department, Government of Sindh)
• fatada.gov.pk (FATA Development Authority)
• bisp.gov.pk (Benazir Income Support Programme)