The Wayback Machine - https://web-wp.archive.org/web/20151119023912/http://making.meetup.com/
Making Meetup

Sorting Out Japanese

head

For years, Meetup has been available in 6 different languages. These are:

  • English
  • French
  • German
  • Italian
  • Portuguese
  • Spanish

And while these languages all have their own nuances and challenges when it comes to localizing for them, it was poor preparation for some of the issues we would run into when localizing for our next supported language…

Japanese

Yes, that is right, Meetup is now available in it’s 7th supported language, Japanese! It is our first supported language with a non-latin-based alphabet.

We have run into many issues because of this, but the most interesting presented itself when it came time to implement alphabetical sorting.

Alphabetical Sorting

On first glance, alphabetically sorting a list of strings seems like a trivial task.

>>> Ordering
  .natural()
  .sortedCopy(["eagle","cat","dog","bird","falcon","duck"])
[bird, cat, dog, duck, eagle, falcon]

However many things can actually complicate this, such as capitalized letters:

>>> Ordering
  .natural()
  .sortedCopy(["eagle","cat","dog","bird","falcon","duck","Dog"])
[Dog, bird, cat, dog, duck, eagle, falcon]

A user would expect Dog and dog to be sorted next to each other. However, because these strings are sorted by character code, this is not the case.

To do a more language-friendly sort, you can use something called a collator.

>>> Ordering
  .from(Collator.getInstance(Locale.US))
  .sortedCopy(["eagle","cat","dog","bird","falcon","duck","Dog"])
[bird, cat, dog, Dog, duck, eagle, falcon]

This performs the sorting based on rules from the American English locale. So how about Japanese?

Japanese Alphabets

The first thing to note is that there are multiple alphabets in play here:

  • Kanji: Thousands of symbols borrowed from Chinese and that can form words. Ex: 東京 - 東 means East and 京 means Capital city, so that’s Tokyo).
  • Hiragana: Phonetic script (46 characters), for words that can not be described with Kanji, or to make the pronunciation of a word using kanji transparent. Each character corresponds to a sound, like ma, ka, sa…. Ex.: とうきょう (To-ky-o)
  • Katakana: Same as Hiragana (48 characters) but to represent the pronunciation of foreign words. Ex.: マクドナルド (ma-ku-do-na-ru-do, which is how the Japanese would pronounce McDonalds)
  • Romaji: Latin characters, used to make the pronunciation of Japanese words transparent for those who can’t read the Japanese alphabets (ex: Tōkyō), or used to look trendy, foreign, in some cases.

Japanese ordering

Strings in Japanese can be made up of characters in any of these alphabets, and using ‘dumb’ sorting, these alphabets wouldn’t sort together at all. However you can use a collator based on the Japanese locale, and that gets you pretty far already.

  • 健康 (Health)
  • ダンス (Dancing)
  • テクノロジー (Tech)
  • 家族 (Family)
  • ゲーム (Games)

Once sorted with the collator, it becomes:

  • ゲーム (Games)
  • ダンス (Dancing)
  • テクノロジー (Tech)
  • 健康 (Health)
  • 家族 (Family)

See Below:

x = [u”健康”, u”ダンス”, u”テクノロジー”, u”家族”, u”ゲーム”]
>>> Ordering
  .from(Collator.getInstance(Locale.JAPAN))
  .sortedCopy(x)
[ゲーム, ダンス, テクノロジー, 家族, 健康]

The problem

This will sort words starting with Katakana and Hiragana correctly, but will always put the words starting with a Kanji at the bottom of the list. Why? Because a computer does not know how a Kanji is pronounced, and ordering in Japanese is based on pronunciation, following the strict Gojūon (literally, “fifty sounds”) ordering table for kana alphabets. First should come the words starting with the sound a then with i, u, e, ka, ki, ku, etc.). With Kanji, the characters used do not each correspond to the same sound every time. The pronunciation changes depending on context, and currently computers can not guess that.

Ex.: 今日 can be read as きょう (kyō) or こんにち (konnichi).

The solution

Given a large list of arbitrary kanji strings, we would be unable to sort them correctly without human intervention giving us an indication of how it’s pronounced in its context. But luckily for us, we don’t have a large list of arbitrary strings! We have a relatively small list of static, already known strings. So we did in fact have a solution.
The thing we needed to sort was the list of categories that Meetup Groups can belong to. We just needed the phonetic spellings of our categories which had Kanji-based names, as we learned earlier that any Kanji-based name can be converted into a more transparent phonetic version.

We asked our translators to help provide these, and then created a new locale called ‘Japanese Phonetic’ (ja-x-phonetic) and a new translation file containing these phonetic translations. And then, when ordering the list on the backend, we temporarily translated the categories into ja-x-phonetic, sorted them using a Japanese collator, and then re-translated them back into normal Japanese while maintaining the same order. Et Voilà!

cat

Conclusion

So we solved our problem, but this is an issue which continues to exist for things like user generated content. If we had a large list of member names in Kanji that we wanted to sort, this would be impossible to do without either transliterating each of these ourselves (not scalable), or prompting users to provide phonetic spellings of their names in addition to the Kanji one (more scalable).

This issue came up unexpectedly, but taught us a good lesson about not making any assumptions when dealing with language. If you want to develop a global product, you have to deal with your product in a global way, and that means building things that make sense in ALL languages.

Meetup and Slack

Meetup recently made the switch to Slack for our team communication platform and our clearinghouse of puppy gifs. One of my favorite features on Slack is how incredibly easy it is to create integrations. In order to get familiar with how the webhook system works, I decided to write a little ruby app that uses the Meetup API to post the next available Meetup event for a given group. If you are an organizer who is using a Slack team to facilitate conversation outside of your Meetup events, host Meetups in your office, or just want to let your coworkers know when something interesting is happening, this might come in handy. Enjoy!

https://github.com/jeffcampbell/meetup-slack


Jeff is a Senior QA Engineer at Meetup

Apple In-App Purchase Mock Service (Fake it ‘till you make it)

A configurable sandbox to test auto-renewing Apple IAP subscriptions

TLDR; We made a thing that pretends to be an Apple In-App Purchase service to make it easier for people to test their code. It’s open sourced and can be found here: https://github.com/meetup/apple-of-my-iap

The long story:

As part of Meetup’s iOS offering, we recently launched the ability for our members to use Apple In-App purchases (IAP) when buying Meetup subscriptions.

Implementing this feature was a lot of fun. The same kind of fun as running through a gauntlet full of sharp axes, hot flames, and killer piranhas.

But before we get into that….

What exactly is an Apple In-App Purchase, and more importantly why do I care? (Hint: It’s not Apple Pay)

An Apple In-App Purchase (IAP) is a purchase of digital goods or services through Apple’s native payment processing platform available on all iOS devices. If you’re an iOS user and you’ve ever been prompted by this screen:

then chances are you’ve made an In-App Purchase before. Typical use cases of include:

  • unlocking premium features within an application
  • purchasing extra levels or items from within a game
  • purchasing a subscription to digital content

Since a customer’s payment information is already stored in their iTunes account, they have the ability to make quick and painless payments without having to pull out their wallet and re-enter credit card number information. One tap and they’re done. This also means that you don’t have to spend time designing, building, and maintaining finicky payment forms. 

And if that’s not enough incentive for you, Apple will kick you out of the iTunes store for attempting to sell digital goods/services in any other manner other than through IAP.

Sounds like a deal to me.

Meetup meets IAP

Here at Meetup, we elected to use the auto-renewing feature of Apple IAP, which means that Apple will automagically take care of all the nonsense that comes with scheduling and charging subscription payments. Hooray!

All we have to do is ask Apple who paid for what, and then ensure that we are delivering the right service and products to the right people. Since the Meetup platform exists on the web as well as on both iOS and Android, we delegate subscription management to the server in order to ensure a uniform experience across all 3 platforms.

So for any given member with a Meetup subscription bought through IAP, we query Apple’s iTunes endpoint and receive a response similar to this:

{
   "status": 0,
   "environment": “Sandbox”,
   "receipt": {
       "receiptType": “ProductionSandbox”,
       "adamId": 0,
       "appItemId": 0,
       "bundleId": “com.meetup.iphone”,
       "applicationVersion": “409000”,
       "downloadId": 0,
       "versionExternalIdentifier": 0,
       "requestDate": “2015-04-23 23:33:34 Etc/GMT”,
       "requestDateMs": “1429832014510”,
       "requestDatePst": “2015-04-23 16:33:34 America/Los_Angeles”,
       "originalPurchaseDate": “2013-08-01 07:00:00 Etc/GMT”,
       "originalPurchaseDateMs": “1375340400000”,
       "originalPurchaseDatePst": “2013-08-01 00:00:00 America/Los_Angeles”,
       "originalApplicationVersion": “1.0”,
       "inApp": [
           {
               "quantity": “1”,
               "productId": “com.meetup.iphone.test.7days”,
               "transactionId": “1000000151887289”,
               "originalTransactionId": “1000000151887289”,
               "purchaseDate": “2015-04-15 21:46:04 Etc/GMT”,
               "purchaseDateMs": “1429134364000”,
               "purchaseDatePst": “2015-04-15 14:46:04 America/Los_Angeles”,
               "originalPurchaseDate": “2015-04-15 21:46:06 Etc/GMT”,
               "originalPurchaseDateMs": “1429134366000”,
               "originalPurchaseDatePst": “2015-04-15 14:46:06 America/Los_Angeles”,
               "expiresDate": “2015-04-15 21:49:04 Etc/GMT”,
               "expiresDateMs": “1429134544000”,
               "expiresDatePst": “2015-04-15 14:49:04 America/Los_Angeles”,
               "webOrderLineItemId": “1000000029487694”,
               "isTrialPeriod": “false”
           },
           {
               "quantity": “1”,
               "productId": “com.meetup.iphone.test.7days”,
               "transactionId": “1000000151887546”,
               "originalTransactionId": “1000000151887289”,
               "purchaseDate": “2015-04-15 21:49:04 Etc/GMT”,
               "purchaseDateMs": “1429134544000”,
               "purchaseDatePst": “2015-04-15 14:49:04 America/Los_Angeles”,
               "originalPurchaseDate": “2015-04-15 21:48:06 Etc/GMT”,
               "originalPurchaseDateMs": “1429134486000”,
               "originalPurchaseDatePst": “2015-04-15 14:48:06 America/Los_Angeles”,
               "expiresDate": “2015-04-15 21:52:04 Etc/GMT”,
               "expiresDateMs": “1429134724000”,
               "expiresDatePst": “2015-04-15 14:52:04 America/Los_Angeles”,
               "webOrderLineItemId": “1000000029487695”,
               "isTrialPeriod": “false”
           }
       ]
   },
   "latestReceiptInfo": [
       {
           "quantity": “1”,
           "productId": “com.meetup.iphone.test.7days”,
           "transactionId": “1000000151887289”,
           "originalTransactionId": “1000000151887289”,
           "purchaseDate": “2015-04-15 21:46:04 Etc/GMT”,
           "purchaseDateMs": “1429134364000”,
           "purchaseDatePst": “2015-04-15 14:46:04 America/Los_Angeles”,
           "originalPurchaseDate": “2015-04-15 21:46:06 Etc/GMT”,
           "originalPurchaseDateMs": “1429134366000”,
           "originalPurchaseDatePst": “2015-04-15 14:46:06 America/Los_Angeles”,
           "expiresDate": “2015-04-15 21:49:04 Etc/GMT”,
           "expiresDateMs": “1429134544000”,
           "expiresDatePst": “2015-04-15 14:49:04 America/Los_Angeles”,
           "webOrderLineItemId": “1000000029487694”,
           "isTrialPeriod": “false”
       },
       {
           "quantity": “1”,
           "productId": “com.meetup.iphone.test.7days”,
           "transactionId": “1000000151887546”,
           "originalTransactionId": “1000000151887289”,
           "purchaseDate": “2015-04-15 21:49:04 Etc/GMT”,
           "purchaseDateMs": “1429134544000”,
           "purchaseDatePst": “2015-04-15 14:49:04 America/Los_Angeles”,
           "originalPurchaseDate": “2015-04-15 21:48:06 Etc/GMT”,
           "originalPurchaseDateMs": “1429134486000”,
           "originalPurchaseDatePst": “2015-04-15 14:48:06 America/Los_Angeles”,
           "expiresDate": “2015-04-15 21:52:04 Etc/GMT”,
           "expiresDateMs": “1429134724000”,
           "expiresDatePst": “2015-04-15 14:52:04 America/Los_Angeles”,
           "webOrderLineItemId": “1000000029487695”,
           "isTrialPeriod": “false”
       }
   ],
   "latestReceipt": “MIJA5AYJKoZIhvcNAQcCoIJA1TCCQNECAQEx …”
}

The Problem: Testing with IAP

Apple provides an iTunes sandbox environment which can be used to test In-App Purchases without spending real money. For auto-renewing subscriptions, the sandbox environment allows you to create a subscription which then automatically renew itself six times before expiring. Since many subscriptions periods last one week at the very least, Apple provides shorter test durations for the sandbox environment which are mapped to production durations in the following manner:

Production -> Sandbox

1 week        ->  3 minutes

1 month      ->  5 minutes

2 months    ->  10 minutes

3 months    ->  15 minutes

6 months    ->  30 minutes

1 year         ->  1 hour

However, there were some serious limitations with the iTunes sandbox which made it difficult to develop and test for, especially for non-iOS engineers like myself.

  1. You need to have an iOS device with IAP functionality already built into your app in order to simulate an IAP purchase. This would require our iOS engineers to try and build IAP functionality into the app before any backend support for it existed.
  2. It’s impossible to manipulate test subscriptions once they have been created in the sandbox. Testing edge cases such as refunds, cancellations, early renewals, late renewals, etc. are only doable by writing mock subscription creation cases within the iOS app. This would require our iOS engineers to spend extra time creating tooling around a feature that they couldn’t yet test. 
  3. The shortest amount of time it takes for a subscription to expire is 18 minutes, which is particularly irritating when you’re trying to debug expiration logic in your application.

Points #1 and #2 would have created heavy inter-team dependencies between our iOS engineers and our backend engineers, something which would have slowed down development immensely.

Point #3 is just annoying.

Unable to get what we wanted from Apple’s iTunes sandbox environment, we decided to roll our own mock service which simulates IAP subscriptions. Our requirements were pretty straightforward. We wanted a service that:

  • would mimic responses we’d receive from the iTunes store
  • was isolated from our codebase and behave as a 3rd party service
  • could simulate IAP transactions without an iOS device
  • could simulate different IAP subscriptions states (refunded, cancelled, etc.)

So we built it!

The end result is a service that any of our engineers could spin up at a moment’s notice, create mock subscriptions, test weird edge cases, and then shut it all down with a few keystrokes.

If you’re interested in trying out our service, simply grab the code from our github repo: https://github.com/meetup/apple-of-my-iap and take it for a spin!