Introducing the New Zealand Trade Intelligence Dashboard

I’d like to introduce to you the New Zealand Trade Intelligence Dashboard. This is one of the New Zealand government shiny web apps that should have been mentioned in Peter Ellis’ post. If you are only interested in TRADE, you may go to ‘Commodity Intelligence’ section directly.

The dashboard is developed using R and the shinydashboard package, and it is my very first shiny app from designing to coding. As I have learnt a great deal during its development, I plan to write some follow-up posts on topics such as how to start your first shiny app, how to choose between shiny and shinydashboard, how to choose the best interactive graph/table packages, how to design your app and some tricks in using reactiveValues and some useful functions such as shinyjs::hide, shinyjs::show, shinyjs::disable and shinyjs::enable.

OK. Let’s cut to the chest and see what the dashboard can do for you (with a deliberate emphasis on the Commodity Intelligence tab because of its powerful analytical ability).

First impression

The dashboard has five panels on the left – main dashboard, market intelligence, commodity intelligence, FAQs and StatsNZ releases.

The main dashboard contains “spoon fed” information on trade, which does not require any user input.  On initial load, it only displays high level information in order to save loading time. More detailed trade information will be displayed after clicking the “Show more details” button.

first

Market intelligence

This tab requires a very simple user input — either one market, or multiple markets or one market group. Additional information on bilateral investment stocks and visitor movements are added. This is the MOST loved and used tab by colleagues from the Ministry of Foreign Affair and Trade.

second

!!!!!!Commodity intelligence!!!!!!

I love this tab! Why? Because it offers superb analytical ability and flexibility (in my opinion), and it is super easy to use.

After clicking “Commodity Intelligence”, you will see three sub-tabs – Exports, Imports, and Intelligence by HS codes. I will focus on the first and the third one as Imports tab is pretty much identical to its counterpart except that global analysis is NOT available.

Exports –>  Pre-defined

After clicking “Exports”, you have two choices either Pre-defined or Self-defined.

Pre-defined means a list of commodities and services defined by Statistics New Zealand (this option does not give user any flexibility to form their own commodities). Those pre-defined ones are principle or key exports earners. You can search or just select as many as you want and then build a report.

The report has two-stages.

First, it will produce high level information for each commodity/service’s trend and growth performance, and then give user an opportunity to select ONE commodity/service for further analysis. Note that users can use the filter boxes above each column in the growth table to narrow down number of commodities. For example, users can filter commodities above certain export values, above certain CAGR5 (5 year compound annual growth rate) or above certain ABS5 (5 year absolute value change).

At the second stage, after you select ONE commodity, it will take quite a while (30 seconds, so just be patient) to produce the commodity’s market analysis both domestically (for NZ) and internationally (for the whole world). The global analysis part accounts for most of the waiting times because it downloads lots of data from UN Comtrade. Also be aware of the query limits of 100 per hour from UN Comtrade , which means the global analysis could be unavailable after reaching the limit. You may need to wait for another hour (There is a note at the very bottom of the page telling the number of query left and when the limit will be reset).

My favorite graph is the sankey plot that shows the trade flows from exporters to importers, which also gives user the option to show and hide EU-28 internal trade.

third

Exports –>  Self-defined

It is the coolest option! That is because users will have the flexibility to build their own commodity list based on all 2, 4, and 6 level HS codes.

Users have to follow some STRICT rules to build the HS codes to commodity name grouping/mapping.

  1. It has to be a CSV file
  2. The first column has to be HS codes
    • you can mix level 2, 4, 6 HS codes
    • you can miss the “0” in front of Chapter 1 to 9 as the missing “0” will be auto-filled
  3. The second column has to be the description/name of the commodity

After successfully uploading your CSV file, you can build the report and the content of the report is identical to that of Export –> Pre-defined.

Here are two example CSV files. So give a try.

  1. Link to Statistics New Zealand Pre-defined commodity CSV file
  2. Link to the CVS file containing all level 6 HS code under Chapter 84, 85, and 86

fourth

Exports –>  Intelligence by HS Codes

This is the most convenient tab!

After selecting this tab, you will see a table with three columns – HS level, HS code, and Classification (HS description).

Regular expression rules are applied to each column via the filter boxes above. Users can search a commodity by simply typing the name in the search box or the filter box above Classification. For example, type firework in either search box or box above Classification column.

User can also use the table to generate a CSV file that can be used in Export –> Self-defined. Try this

  1. Type 6 in the box above the HS level column
  2. Type ^84|^85|^86 in the box above the HS code column. ^ means starting with. | means or. 84 to 86 are the HS chapters
  3. Click show “All” to display the full table
  4. Click CSV button to download the table as a CSV file
  5. DELET the first column
  6. Now you can use this CSV file as an input for Self-defined analysis

To generate an instant report, users simply click on the interested rows in the table. SIMPLE AS!!

fifth

Final thoughts

The shinydashboard package is awesome. It is very easy to learn for experienced R users. It offers a very simple structure to follow. The dashboard is also MOBILE compatible out of box, which is a big PLUS! The dashboard can be easily adapted to display any countries’ trade intelligence.

Advertisements
Posted in R related | Leave a comment

Have I chosen the right power company?

Do you always wonder if I have chosen the right power company and have not been over charged? Your questions may be answered here (if you reside in Wellington, New Zealand).

Power costs per day and per month_Oct_2012 shows which company provides the cheapest electricity and natural gas for different amount of kWh used per day or per month. Two look-up tables (one based on daily usage Monthly Power Bill and Annual Saving Lookup Table based on Daily readings and the other based on monthly usage Monthly Power Bill and Annual Saving Lookup Table based on Monthly readings) are provided for you to check:

1. potential monthly bills for different companies and different daily/monthly usage
2. potential saving on bills per YEAR for different companies and different daily/monthly usage

One myth busted for Wellintonians is that Powershop, thought to be the best deal, is actually the 3rd most expensive power company among the other 8 companies, if you use more than 3 kWh per day or 120 kWh per month (average NZer use 7000 kWh per year or 580 kWh per month or 20 kWh per day).

Posted in R related | Leave a comment

R becomes a critical tool in government departments

Situation and Outlook for Primary Industries (2012) just published by New Zealand’s Ministry for Primary Industries (click to download page) demonstrates well that R is a limitless tool for analysis and graphing, and the capability of using R is growing in government departments.

Special thanks to Andy South, whose rworldmap package enables an easy and systematic way to create map-pie charts. Also, Joris Meys’ solution helps to adjust an Atlantic-centered map to Pacific centered map (It would be nice if rworldmap package has Pacific-centered map as an alternative). R AnalyticFlow is one of the handiest GUI for R.

Derek McCrae Norton’s presentation in the 2011 useR conference gives an extremely effective way to marketing R in big organizations with long histories. If you would like to build up the presence of R in your organization, please read and follow Derek’s advice.

The extra surprise is that our publication team absolutely loves graphs (.eps) produced by R. The reason is simple — they do not need to do anything apart from adjusting graph positions. What they used to do with M$ excel graphs is very onerous — change all color themes, fonts and axis. Of course, this process will come back and forward several times. With R, once these cosmetic things are fixed, what’s left is just mass productions.

Posted in R related | 3 Comments

R tells you where weapons go

As an ameturer programmer (one without proper trainings in any mainstream programming language — C and Java) , the more I use R the more I understand the saying — “You are only bounded by your imagination”.

The other day I suddenly recalled that someone did a very impressive Facebook map. I then thought it would be nice if I can put these “flows” on the map (or of the same sort) created in my first post. So, I googled around and found this brilliant blog that teaches you how to make flows (Great circles) step by step.

Again, thanks to R, its great community and its openness, I created the following map of international weapon export in 2010 (from top 7 exporters). I hope someone who knows about weapon exports could share some stories with us. But, let’s all pray for PEACE.

Posted in R related | 10 Comments

When Wellington meets the “animation” package

The “animation” package is great for creating .gif files (of course, it also produces video and flash files thanks to Yihui Xie). By using this package, I would like to show you a nice spot in Wellington, NZ. At this spot, you can see west coast, east coast and South Island at the same time. If you are in NZ for the Rugby World Cup and can spare some time (20 mins bush walk), I highly recommend you to pay a visit.

Its address is:  -41.234164, 174.779841 on Google Map.

Posted in R related | 4 Comments

Tutorial on using the rworldmap package

This blog following up my previous oneattempts to explain how the geo-pie map was created.

I do not know how to attach a .rflow file in this blog. What you can do is to copy the following code into Notepad and save it as XXX.rflow, and open it by RAnalyticFlow. By using it, you can see the task is divided into 7 steps, and run up to any node you want.

Before you run the code below, you have make sure that required packages are installed. If you would like to run the 7th part of the code, you need to download an image of R-bloggers. I hope to make a generalized function out the code ONE DAY, but without any promises though. After running the code, you will have

Here is the plain R code.

################# 1. Clear working space ###############
rm(list=ls())

################ 2. load packages ####################
## make sure you have them installed
require(doBy)
require(plyr)
require(rworldmap)
require(TeachingDemos)
require(ReadImages)

################ 3. make trade data #################
### make sure your country names is by UN convention
### Otherwise you need to modify your countries names
trade.data <- data.frame(Country=c("China",
                                   "United States",
                                   "United Kingdom",
                                   "Brazil",
                                   "Indonesia",
                                   "Germany",
                                   "Russia",
                                   "Australia",
                                   "Malaysia",
                                   "South Africa")
                         )

### fabricate some data for each country
### I just generate numbers ramdomly and make sure
### they are positive
trade.data$product1 <- abs(rnorm(10,10,30))
trade.data$product2 <- abs(rnorm(10,50,30))
trade.data$product3 <- abs(rnorm(10,100,30))
trade.data$product4 <- abs(rnorm(10,150,30))
trade.data$product5 <- abs(rnorm(10,200,30))

################## 4. make our Geo-pie plots #################
### you can find more info make by using
### "?mapPies" commands without the quotes

##### 4.1 merging with existing data
sPDF <- joinCountryData2Map(trade.data,
                            joinCode = "NAME",
                            nameJoinColumn = "Country")

## This the data that we will plot
dF <- sPDF@data

### make our pie plot
par(mai= c(0,0,0.6,0),
    xaxs = "i",
    yaxs = "i")

mapPies(dF =dF,
        nameX="LON",
        nameY="LAT",
        nameZs =c("product1",
                  "product2",
                  "product3",
                  "product4",
                  "product5") ,
        zColours=c("red",
                    "green",
                    "magenta",
                    "yellow",
                    "blue"),
        oceanCol = "lightblue",
        landCol = "wheat",
        addSizeLegend=F,
        addCatLegend=F,
        mapRegion="world",
        xlim=c(-181,181),
        ylim=c(-81,80))

title(main=paste("Our lovely Geo-Pie plot using rworldmap package"),
      cex=3)

legend(-180.1516,90,
       legend=c("Product 1",
                 "Product 2",
                 "Product 3",
                 "Product 4",
                 "Product 5"),
       col=c("red",
             "yellow",
             "green",
             "magenta",
             "blue"),
       pch=16,
       cex=0.8,
       pt.cex=1.5,
       bty="o",
       box.lty=0,
       horiz = F,
       bg="#FFFFFF70")

################ 5. Lets make a barplot on our map ##############
## derive the data for barplot -- top 10 markets for
## product1
tmp_product1 <- orderBy(~-product1,dF)[1:10,]

### make the barplot as a function
subp1.1 <- function(...){
	barplot(rev(tmp_product1$product1),
             width=1,
             names.arg=rev(tmp_product1$Country),
             las=2,
             border="transparent",
             horiz=T,
             col="darkgrey",
             main=paste("Top 10 markets for product 1"),
             cex.main=0.7
             )
      mtext(1,
            text="NZ$ million",
            line=1.7,
            cex=0.7
            )
      grid(col="black")
}

################# 6. plot the barplot and make a bit more lovely ##
#### make our background for the barplot
#### you have to fiddle a bit on the position
subplot(rect(-180,-90,-100,-32,
             col="#FFFFFF70",
             border=NA),
        x=0,
        y=0,
        vadj=1,
        hadj=1)

#### let's plot
w_x <- 0.66
w_y <- 1
pos <- c(-160,-81)

par(mar=c(4,4,1,1),
    cex.axis=0.7,
    cex.lab=0.7,
    bg="white",
    cex.main=0.7)

subplot(subp1.1(),
        x=pos[1],
        y=pos[2],
        size=c(w_x,w_y)
        )
############ 7. (Optional) put a logo on your map ###############
rblog <- read.jpeg("Rbloggers_logo.JPG")
#Get the plot information so the image will fill the plot box, and draw it
pos_rblog <- c(150.7,58.4,181,73.8)
rasterImage(rblog,
            pos_rblog[1],
            pos_rblog[2],
            pos_rblog[3],
            pos_rblog[4])
rect(pos_rblog[1],
     pos_rblog[2],
     pos_rblog[3],
     pos_rblog[4],border="black",
     )
Posted in R related | 2 Comments

Let R fly: Visualizing Export Data using R

Having been using R for 5+ years, visiting www.r-bloggers.com daily, learning cool tricks from it and knowing cool UseRs from it, I finally decide to quit being an observer and start to be a contributor to this wonderful community. In this very first of my blog, I would like to demonstrate how to use a very cool R GUI — RAnalyticFlow and three useful R packages — rworldmap, gregmisc and TeachingDemos to visualize New Zealand primary export data (the data used in this post are freely and publicly available from www.stats.govt.nz).

RAnalyticFlow is a free GUI for R. It lets you to create flow charts of R codes. I find it really useful to break down big chunks of R codes into small manageable pieces. It also allows you to put non-essential codes (plots or checking data) out of essential ones. Below is an example

I am not going to details about this GUI but I do encourage you to download and have a play (runs on Mac, Windows and Linux). It is addictive that I must warn you though. A little downside is that it only supports 32-bit R.  But the developers are approachable if you want a version that supports 64-bit.

My goal here is to plot two world maps that show

  1. what primary-sector-products have been exported to which countries for the year ended June 2011
  2. what is the change on last year

“rworldmap” package offers functionality to visualize data on a world map. “mapPies”  is fairly flexible function that is suitable for the first job. Forming the data is pretty straightforward, which I will skip here. One thing that I like to point out is that “trim” function from “gregmisc” package is a useful function to eliminate spaces ahead and at the rear of strings (worth to keep in your pocket).

After having the map plotted, I like to have some barplots on the map to display top 10 markets for each product. “subplot” function from “TeachingDemos” package plots basic R graphs on an existing plot for a given position. For the background of these barplots, I used “rect” function. The final product rolls out like below

The second job is to show the change in export revenue on last year. “mapBars” from “rworldmap” function offers the functionality to do that, but it plots bars as stacked vertical or horizontal rectangular, which is different from conventional bar plots as we expect. Instead, I used “subplot” and “barplot” functions to consummate the job. The final product of the second job goes like

To conclude, I’d like to encourage you to give a try on RAnalyticFlow and consider using “rworldmap” package when you have worldwide data to visualize.

Posted in R related | 8 Comments