How I customized the Wikipedia app to match the needs of droidwiki.de

“Do you have an app?” This questions is and will become more important when nearly half of your website-traffic is generated by mobile phones. While it’s very easy to get a mobile optimized version of your MediaWiki using the MobileFrontend extension, it’s a lot harder to get an app for your MediaWiki based website (there’s no “MediaWiki App”). You may already know, that there are (for Android) two repositories holding mobile app code for an Android app in the Wikimedia own git repository, WikipediaMobile and apps-android-wikipedia. While the first one is based on the PhoneGap framework and isn’t maintained anymore, the 2nd one is widely known, it’s the source of the official Wikipedia app. Now, one could think, that it’s easy to customize this code, like other components of the MediaWiki eco-system. But unfortunately, the Wikipedia app codebase is completely developed for the Wikipedia environment. It’s a bit more difficult to customize it, not only a config option you need to change.

IMPORTANT: This blog post isn’t meant as a tutorial to know step by step how to customize and build your own app for your MediaWiki with the Wikipedia app source code. It’s meant to give some insights in how I managed this task and maybe give some tips and tricks if you have the same plan. However, I hope that you can learn a bit from this post 🙂


The environment

I really love my PC, and I’m a Windows user (buh!), and I know why, with all pros and cons 😛 But the Wikipedia app is developed by people who don’t use Windows (as far as I know), and I do any other developing work on an Ubuntu VM, too, so I decided (to go around any possible difficulty when using a Windows OS) to setup a new Ubuntu VM for the App customization, only. I choosed Ubuntu 14.04 (I already use the same OS version for other development task related to droidwiki.de and MediaWiki) with 4 GB RAM and 200 GB of storage (later you’ll see, that after I completed the first useable and “read-for-Google-Play-upload” build the used storage is 18 GB (nearly 10%)). I installed the basic packages, like git, the jdk (you should use java 1.7 not java 1.8) and, of course, the Android Studio.


Setting up the git repository

Even if the source code of the Wikipedia app is versioned in git already, it’s very important for me to have my own repository where I can do whatever I want, commit changes I made (for droidwiki.de only) and roll back to an earlier (working and already customized) version, if needed. So I decided to create a new repository in droidwiki.de’s gerrit instance, which is named apps/android/droidwiki (like the Wikipedia app repository apps/android/wikipedia). To have my own area in this repository which isn’t changed when I upgrade the codebase to the latest Wikipedia app version, I decided to create a directory called src/ in the repo. Into this one I cloned the Wikipedia app code.


The first build

import_projectTo make sure, that the setup basically works, I started Android Studio, run through the first start process (downloading the Android SDK, the latest API code and an optimized AVD, which I can’t use, later more ;)) and selected that I want to import an existing (gradle) project. At this point it’s possibly a good idea to give you the hint, that the Wikimedia apps team has an information page where you can read how to “hack” the Wikipedia app. Especially the setup-steps and possibly all other information given there are very helpful to understand how the codebase works in general. This is also important for a customized app, which is my goal (you remember ;)). After the processing of the project files by Android Studio (Loading modules, indexing and so on) I give it a try and clicked the “Run” button and let gradle build the project. This takes a bit of time… Now, let’s see, if the result works! Because the fact, that I’m using VirtualBox to emulate the Ubuntu guest, I can’t use hardware virtualization in the Android virtual device (AVD), which would be very very slow, so the better alternative is (for me) to connect my phone and test the app directly on a physical device. I’m pretty sure there’re ways to handle this problematic in a nicer way (and use an AVD for testing), but personally I’m fine with my solution. After selecting the target device, Android Studio installs the app using adb and starts it: There we go! After a short transfering and installation time, the Wikipedia app starts on the device!


 Basic costomization of the code base

Before I started to customize the look and feel of the app (e.g. remove menu items, that doesn’t apply to droidwiki.de) I changed some general stuff. E.g., I don’t want to install the org.wikipedia package, I want a de.droidwiki package. The easiest way to achieve this would be to navigate to the Java folder of the “app” module in the project navigator (left side) and select the “org.wikipdia” package. Pressing Shift + F6 (or right click on it, Refactor -> Rename) let’s me rename the package. But, unfortunately, it only renames the last part (wikipedia) of the package name (org.wikipedia), which would result in org.droidwiki, which isn’t what I want. Because I haven’t found an easy workaround to resolve this, I used the “Replace in path” feature of Android Studio to semi-manually replace any usage of org.wikipedia inside the java/ directory of the project navigator. Another place to rename any org.wikipedia is the AndroidManifest.xml and any gradle build file, which is easily possible to be made in Android Studio. I also needed to change the usages in the assets/ and res/ folder. Because of the amount of changes it’s probably a good idea to use sed for it or the global “Replace in path” of the “app” module. After that I renamed the source code directory app/src/main/java/org/wikipedia/ to app/src/main/java/de/droidwiki/, I did the same for app/src/androidTest/java/org/wikipedia/ and app/src/testlib/java/org/wikipedia/ (only used for testing, but to have a clear workspace I did this, even if I don’t plan to run any of these tests, yet). After the re-indexing of Android Studio I gave it a try and started a new build of the app to see if there are any usages left, which aren’t resolved and prevents a build. Like I expected there wasn’t any left usage, so the build finished and the app was installed on my device again, but this time with the de.droidwiki package name (so, now I had two instances of the Wikipedia alpha app on my phone).

Screenshot_20160207-000844
The first article loaded in the first working DroidWiki.de app build, *yay* 🙂

The next big thing to change are request based. Currently, the app retrieves the contents from wikipedia.org/w/api.php?…, our domain is slightly different:

    • First of all: DroidWiki uses droidwiki.de, not wikipedia.org (logical)

li style=”text-align: left;”>The MediaWiki base directory (script path) isn’t w/, droidwiki.de has it’s article path at /$1 and the script path at / which changes the api entry point to /api.php

Both things need to be changed at several places to make the app working (that’s why I don’t call it easily configurable):

  • in the assets/ directory (both has a base tag, which currently points to https://wikipedia.org, I changed it to https://www.droidwiki.de)
    • the index.html
    • preview.html
  • in java/de/droidwiki/server/mwapi/MwPageService.java
    • the api.php queries are defined in the MwPageEndpoints interface)
  • in the AndroidManifest.xml
    • if the app should support to open links to the wiki, e.g. when the user clicks on a search result link of the wiki, the intent filter needs to be changed, currently it will trigger for any wikipedia url. I removed the intent filters completely, because my app shouldn’t support deep linking
  • in java/de/droidwiki/util/UriUtil.java
    • the isValidPageLink method checks the domain and the url target against the wikipedia.org scheme
  • in java/de/droidwiki/Site.java
    • the getScriptPath method defines the script path according to the Wikipedia sheme
    • the titleForInternalLink uses the article path according to the Wikipedia scheme
    • the forLanguage method creates a Site object for the given language which uses a different base url depending on the language code, I changed it to always return the droidwiki.de Site object (droidwiki.de doesn’t provide different languages for different sub domains)
    • isSupportedSite has a regex against wikipedia.org
    • the urlToLanguage method returns the language according to the language in the url (de, en, …), for droidwiki.de it always returns de now

After a new build (again), the app should now load the content from droidwiki.de and not from Wikipedia anymore. Now, the app in general could be used for droidwiki.de, but there’re several things, which aren’t working, yet, doesn’t apply for droidwiki.de or are branded for Wikipedia. In the next steps this will be changed!


Fine tuning

The first working build makes me very happy (after finding (hopefully) any places where I’ve to change the wikipedia.org -> droidwiki.de), but there are still things that need to be changed: The branding and wording of Wikipedia, the not loaded lead image, the fact, that the app tries to load Wikipedia:Haupseite (german main page of Wikipedia) at startup and so on. Let’s go!

I think, the easiest thing would be the main page, so let’s find out, where it’s defined, which project has which main page. Because I know the main page name of at least two Wikipedia projects (en and de), it’s very easy to use the find function in Android Studio to find it: java/de/droidwiki/staticdata/MainPageNameData.java. The class MainPageNameData simply holds a data map of languages to main page names. Because droidwiki.de has only one main page (and doesn’t has different wikis for different languages), it’s easy to remove the initialization of the DataMap (to save space) and change the valueFor method to always return “Hauptseite” (the german word for “Main page” and the main page of droidwiki.de). That’s it! After a new build, the main page “Hauptseite” will be loaded, instead of “Wikipedia:Hauptseite”.

Ok, the next thing is: The app name. Currently the app is named “Wikipedia alpha”, even if I’m ok with “alpha”, I don’t want, that my app is called “Wikipedia”, so I changed it to “DroidWiki”. But there is the first “problem”: The app name is defined in the strings.xml (res/values/strings.xml), which is a translateable set of strings used in the app. There’re also translated versions of these strings for many different languages (thanks translatewiki.net at this point for translating all the MediaWIki stuff!). So, I _could_ change it file by file, but why I should do that? 😛 Android Studio unfortunately hasn’t a function to replace strings in the values-* folders, but on any good Linux distribution you’ll find sed, which is very helpful when replacing strings in files (in combination with find it’s possible to do that for multiple files). The command I used was:

find . -type f -exec sed -i "s/Wikipedia/DroidWiki/g" {} \;

in the src/main/res/ directory. With it, I should also replaced any other string containing Wikipedia to now use DroidWiki. Wording: Done!

Now, the App icon looks good for Wikipedia, but not for DroidWiki. I already have a new icon, I only need to apply it. The app icon for an android app is defined in the AndroidManifest.xml as the android:icon value for the application tag, which, for the Wikipedia app, points to “@mipmap/launcher“. So, the only thing to do is: Replace the launcher.png file in any mipmap-* folder of the res/ directory with the droidwiki launcher icon. App icon: Done!

lead_default_darklead_defaultI also don’t like the lead image placeholder image (which uses the Wikipedia globe). I decided to use a clear gray (for the light theme) and black (for the dark theme) image (with a trend to a lighter color). To apply both images, I simply needed to replace the files at res/drawable/.

There are also some occurrences of the Wikipedia “W”, which can be easily replaced, too. I simply replaced these files with the droidwiki own logo:

  • drawable-hdpi (about_logo.png, w_nav_mark.png)
  • drawable-mdpi (about_logo.png, w_nav_mark.png)
  • drawable-xhdpi (about_logo.png, w_nav_mark.png)
  • drawable-xxhdpi (about_logo.png, w_nav_mark.png)
  • drawable-xxxhdpi (w_nav_mark.png)

I also placed a file called droidwiki_about_grayscale.png into each listed directory (drawable-xxxhdpi excluded), which is a grayscale version of the about_logo.png and used for the splashscreen background. Why? The Wikipedia “W” is black/white already, which means it’s ok to use it as a splashscreen background (the splashscreen itself is a dark gray), so the effect of the splash bg looks good. The DroidWiki logo is colored, which would look strange for an otherwise dark splashscreen, a grayscale version, however, doesn’t look too bad. To apply this different splashscreen logo to the the splashscreen itself, I needed to change res/drawable/splash_bg.xml file, which is a layout file. Changing the android:src=”@drawable/about_logo” to android:src=”@drawable/droidwiki_about_grayscale” is enough.

The not loading lead image, yes, the biggest problem so far I think. With a bit of debugging I found out where the image url is loaded, so first I simply logged the url from where the app wants to load the image (the fact, that the other images are loaded without any problem let me think, that the problem is related to the url, from where the lead image should be loaded). So I added the following line to the java/de/droidwiki/page/leadimages/ImageViewWithFace.java ImageViewWithFace class into the loadImage method:

L.d("Load image from: " + url);

 

the full method should look like:

public void loadImage(String url) {
        L.d("Load image from: " + url);
        ImageRequest request = ImageRequestBuilder.newBuilderWithSource(Uri.parse(url))
                .setPostprocessor(facePostprocessor)
                .build();
        PipelineDraweeController controller = (PipelineDraweeController)
                Fresco.newDraweeControllerBuilder()
                        .setImageRequest(request)
                        .setAutoPlayAnimations(true)
                        .build();
        setController(controller);
    }

Screenshot_20160207-014634After building the app and starting it, it seems I found the problem. While all images are loaded correctly, the app tries to load the lead image for the page “Android” from: https:/images/thumb/f/f9/Android_Homescreen_Stock_4.jpg/384px-Android_Homescreen_Stock_4.jpg, which, of course, can’t work. I’m not sure, why this happens, the query to get the leadimage is the same for droidwiki.de and wikipedia.org, but the response is different. While the wikipedia.org response returns upload.wikimedia.org link (e.g.  //upload.wikimedia.org/wikipedia/commons/thumb/1/18/T-Mobile_G1_launch_event_2.jpg/640px-T-Mobile_G1_launch_event_2.jpg), the local link is returned for droidwiki.de (e.g. /images/thumb/f/f9/Android_Homescreen_Stock_4.jpg/384px-Android_Homescreen_Stock_4.jpg). I assume, that this simply isn’t handled in the Wikipedia app code-base, as this should never happen for a WMF wiki (that’s one more time, where I think, that the Wikipedia app code base isn’t made for other wikis as WMF ones). To solve this problem I changed the url building method in the LeadImagesHandler class (java/de/droidwiki/page/leadimages/LeadImagesHandler.java), which is loadLeadImage from:

private void loadLeadImage(@Nullable String url) {
        if (!isMainPage() && !TextUtils.isEmpty(url) && isLeadImageEnabled()) {
            String fullUrl = WikipediaApp.getInstance().getNetworkProtocol() + ":" + url;
            articleHeaderView.setImageYScalar(0);
            articleHeaderView.loadImage(fullUrl);
        } else {
            articleHeaderView.loadImage(null);
        }
    }

to:

private void loadLeadImage(@Nullable String url) {
        if (!isMainPage() && !TextUtils.isEmpty(url) && isLeadImageEnabled()) {
            String fullUrl = WikipediaApp.getInstance().getNetworkProtocol() + "://" +
                    WikipediaApp.getInstance().getSite().getDomain() + url;
            articleHeaderView.setImageYScalar(0);
            articleHeaderView.loadImage(fullUrl);
        } else {
            articleHeaderView.loadImage(null);
        }
    }

Now, the code not only adds the network protocol, it also adds the domain of the site, which now should allow the app to load the lead image. Let’s test it! And after another new build: The lead images are there, *yay*.

Because the fact, that droidwiki.de supports german only, a language selection button in the search bar doesn’t make any sense. That’s why my next focus is to remove the selection. The visible part is mostly done in res/layout/activity_page.xml, where I simply needed to remove:

<TextView
 android:id="@+id/search_lang_button"
 android:layout_width="30dp"
 android:layout_height="30dp"
 android:gravity="center"
 android:clickable="false"
 android:focusable="false"
 android:background="@drawable/lang_button_shape"
 style="@style/AppTheme.ActionModeStyle"
 android:textColor="@android:color/white" />

after that, I had to remove any reference from main/java/de/droidwiki/search/SearchArticlesFragment.java, where a click on the language button would be handled. And, in general, that’s it, after the next build, the language button shouldn’t appear anymore.

After a new test build I found the problem, that no internal link was clickable, I always got the error message “This link can not be showed”. Ok, what’s going on? After a short research I think I found a missing URL change in the LinkHandler class (java/de/droidwiki/page/LinkHandler.java), which checks, if an URL is internal and supported or not by checking, if it starts with “/wiki/” (which is never true for droidwiki.de internal links, as it doesn’t use the /wiki/ prefix). Removing it and leave “/” alone works fine (but maybe has un-intentional behaviours, as the app was never meant to handle any actions, that’s why I turned the apps’s deep linking feature off, I can’t easily know, if the link is really supported, e.g. several action= links).

The next part are the links at the bottom, called bottomlinks 😉 They currently do some confusing things when adjusted to droidwiki.de, which needs to be changed! To achieve this, only two changes were needed (again some leftovers from the customization of the URL; it’s possibly a good idea to use the find function to find further places which needs to be replaced), the getUriForDomain and getUriForAction methods of the PageTitle class (java/de/droidwiki/page/PageTitle.java).

Ok, now back to some more visible changes: The navbar. The navigation bar at the left side as several items, which aren’t useful for droidwiki.de, something like Nearby, or “Support DroidWiki”. DroidWiki doesn’t have the GeoData extension, which makes Nearby unuseable, and the wiki doesn’t take donations, which makes the button more useless, too. So, I simply remove them from the layout (in res/menu/menu_nav_drawer.xml). It would also be possible to remove any usages of these menus, the classes handling the requests and clicks, but… this would take too long (a personal decision), and the positive point (decreased file size) isn’t so important I think (buh!). However, there were some code-usages for this nav bar items, which need to be removed, in the NavDrawerHelper class (java/de/droidwiki/page/NavDrawerHelper.java).

I did the same thing for the settings, where I simply removed several setting items, such as the language selection menu or the Wikipedia Zero items. Additionally, I changed the appearance of the About activity a bit, so it doesn’t show “A Wikimedia product” anymore and the link “Give feedback” now doesn’t point to the Wikimedia e-mail adress anymore, but to the info@droidwiki.de one. Ok, now, create a testing build one more! I now checked the changes, and if I missed something so far, but, as far as I can tell, it looks good! There were a huge process in the last hour 😛


Adding Google Analytics

The Wikimedia Foundation doesn’t use Google Analytics, and therefor the Wikipedia app doesn’t support tracking users with Google Analytics. However, a very simple per-page event to Google Analytics is pretty easy to implement. I did this already for an older version of the app (see this commit) and just applied the relevant code changes to the new code base again. It’s also possible to pair this tracking with the setting to enable the tracking of the user, which is already there. This setting is accessible with the isEventLoggingEnabled method:

Prefs.isEventLoggingEnabled()


Final build

The build of the signed APK finished!
The build of the signed APK finished!

Ok, now the time has come: build the app (hopefully) the last time for this update cycle. Gradle takes some time to build the app in the prodRelease flavor and builds and builds and builds. After some minutes I got the great screen you see in the image at the right: APK(s) generated successfully. *yay*

After transfering the apk to my phone and installing it the most interesting moment came: Does the app start and work like expected? That you’ll see after the ad… 😛 No, just a joke. The app runs without any problem, with all the new features, fixes and changes of the latest Wikipedia app code base. That’s amazing. But the conclusion: The Wikipedia App in it’s current version isn’t very easy to customize and useable for other wikis. It would be great, if the developers could work at this, that would make the life much easier for us third party users 🙂

Some information:

  • After finishing the Google Play build, df -h gives: /dev/sda1 193G 18G 166G 10% /
  • This blog post is based on the App source code of the 7th February 2016 (2.1.140), commit 90b935b656ac54df7224db6cdacdcf31da4b76c7

Some links

The app was not found in the store. 🙁

4 thoughts on “How I customized the Wikipedia app to match the needs of droidwiki.de”

  1. Hi Florian,

    Great writeup of how you did it.

    BTW, there’s nothing preventing you from running AndroidStudio directly on Windows. I don’t think you need to run this in a VM. I believe one of the Android app devs uses Windows, too. It’s just that the instructions on the “hacking” page were written by Mac/Linux users. Most of the stuff should be easy to transfer to Windows. Nowadays it’s easy to setup AndroidStudio and Git.

    You could also just clone the repo and use a different branch for your customizations. Instead of renaming a ton of files and replacing package and import strings I would just change the applicationId in your app/build.gradle file (https://github.com/wikimedia/apps-android-wikipedia/blob/master/app/build.gradle#L54).

    1. Hi Bernd!

      Thanks for your input! I know, that AndroidStudio should run on Windows, too. However, my whole development environment is based on a Linux VM, so that’s (at least for me) the easier way for me.

      I think I’ll try the idea of just changing the applicationId in the build.gradle. I’m not sure, what you mean with “different branch for your customizations”. Do you mean a new branch in git (which would be self-explaining) or is there a feature in the gradle build process, which you refer to?

      Best,
      Florian

    1. During the hackathon and some other app development (for another app) I know decided to have the Android app development outside of my VM, too 😛

Leave a Reply

Your email address will not be published. Required fields are marked *