Design against crime & Microsoft Azure with Shiny
Guest blog by Mariam Elgabry Microsoft Student Partner at UCL Crime Science Department uses azure to host a shiny webapp
Microsoft student partners IN ACTION!
Mariam Elgabry is a Microsoft Student Partner at the University College London completing her Masters of Research on Security and Crime Science whilst entering her PhD on future crimes. She has graduated from Imperial College London with a Masters in Bioinformatics and was a Microsoft Student Partner Lead for the years 2016 – 2017, establishing the 3Hack, an interdisciplinary pre-hack for the Microsoft ImagineCup, TourdeTech an international initiative for computer science outreach and was selected to attend the MVP Summit in 2016 in Seattle, USA. Mariam is an acting Special Sergeant at the Metropolitan Police of London, volunteering in her spare time. She has also completed an internship at Toren Consulting during which she developed a Shiny App that produces an interactive map of local crime per London neighbourhood and generates a report on the prevalent crimes and hot spot areas. It is this app we will later learn how to deploy a shiny app onto Microsoft Azure services.
toren consulting
Toren Consulting Ltd (torenconsulting.co.uk) is a specialist security design consultancy for the built environment. The company was founded in 2017 by Mark Tucknutt MPhys CPhys MSc MSyI (LinkedIn) and provides security advice and design services to the property sector. Toren supports office, hotel and residential developments and fit-outs from concept design through detailed engineering design to construction and handover.
One of Toren’s frequent tasks is to deliver crime risk assessments, often in support of a project’s sustainability objectives via a BREEAM Security Needs Assessment. A key aspect of this is consideration of reported crime in the area local to a project, including a review of data available via police.uk. We engaged with UCL to find a student from the Security and Crime Science course who could help us to improve our crime risk assessment workflow. Mariam has developed for us the app described in this post, which we expect will both improve the quality of our crime risk assessments and reduce the time taken to produce them.
DEPLOYING A SHINY APP ONTO MICROSOFT AZURE Step-by-step
After developing your awesome shiny new app, you’ll have to host it through a web service if you want other to share its awesomeness. That’s where Microsoft Azure comes in and provides for a secure platform to host your application on. Essentially what you’ll learn today is a step-by-step process (with a few tangents here and there on the obstacles I faced, which will hopefully help you if you face them as well!) of how to build a shiny-server to run on Linux as a web interface to host your R shiny app. So let’s get started!
First and foremost, you’ll need to set up your MSDN account with Microsoft Azure, as a Microsoft Student Partner (and as a new user!) you get various credits given to you, make sure you make good use out of them! Once you are all set up, you will have to create a new Ubuntu virtual machine (VM), ideally you want the latest but most stable version. Select the name and size, remember the username you’ve created and select password for your authorisation option. Include the ssh and http ports in your configuration (we will later select the 80 port for our endpoint). Complete your setup and wait patiently until azure has worked its magic and has – viola – created your VM!
As Azure is completing your configuration and creating your VM, make sure you have PuTTY downloaded and installed (www.putty.org/). Grap the IP address from the “Overview” section within your newly deployed VM and Copy & Paste into PuTTY.
This will prompt you to login with the username and password that you created on your Azure account. Once you’ve gotten the green light there (I know, it’s so confusing typing in the password without the cursor moving!), we need to update and upgrade to the latest versions. Type in the following commands in your command line:
$ sudo apt-get update
$ sudo apt-get upgrade
If it says some have not been upgraded, manually upgrade the ones it points out to you. We then need to getting cracking on installing R and, of course, install shiny:
$ sudo apt-get install r-base
$ sudo su - -c "R -e \"install.packages('shiny', repos='https://cran.rstudio.com/')\""
The shiny package does the reporting and the server allows for the hosting, so now we need our server! Configure it all with the following commands:
$ sudo apt-get install gdebi-core
$ wget https://download3.rstudio.org/ubuntu-14.04/x86\_64/shiny-server-1.5.7.907-amd64.deb
$ sudo gdebi shiny-server-1.5.7.907-amd64.deb
This will produce your new best friend, the .conf file, where you will be defining everything on how you want your app to run – so get acquainted to this fast! You now need to open your .conf file and change the port in which you are allowing traffic into your shiny server from. You will be changing the default port of 3838 to port 80 (or whichever you prefer). Use the following commands to do so:
$ sudo nano /etc/shiny-server/shiny-server.conf
(you can sudo vi or vim or whatever editor suits you best – I quite like nano! You sudo everything as an administrator or else it will tell you you are not authorised to make such changes, keep this in mind when you’re trying to look at your log files later down the line during debugging phase…)
Good advice also states, with every change, restart your server:
$ sudo restart shiny-server
Now if everything has gone according to plan, on your browser type in your IP address, semicolon, the portal you just authorised (80), and you should see a lovely shiny welcome page!
You will most likely have an error on the right of your page in the two panels. These panels hold a demonstration of two shiny apps that use rmarkdown – so let’s go ahead and install that into our VM so we can see them on our welcome page!
$ sudo su - -c "R -e \"install.packages('rmarkdown', repos='https://cran.rstudio.com/')\""
If at any point (not that this happened to me..no…not at all..), after trying to run and unrun things, the homepage disappears (again…not that this EVER happened to me..) check the configurations on your azure portal Overview page. The settings of your DNS and assigned name. I changed it from dynamic to static and ran through the installation process again – if it is already installed, it won’t re-install it, but if there is an update available it will update it. So really doesn’t hurt…
Once all that works out, refresh your page and (hopefully) voila! You should see the applications!
My app – i want my app!
Up until now we’re doing great, everything is set up and working, but we want to deploy our own app – the mission continues! Firstly, you’ll need to copy over the necessary files from your local host onto your brand spanking new (and shiny) VM – just in case you’re such a noob that you don’t know your local host name (once again…not that any of this relates to me and what I had to deal with through deployment..) open up your PC command line (windows, type in cmd in your start page) and type the following command:
$ ipconfig /all
On windows, you can change this to something you like and can remember (and type faster) by going to your Settings > System > About (– who doesn’t like naming their computer?!)
If you’re a FURTHER noob (and I am not saying I came across these issues whatsoever…she says as she whistles away) and forgot your password (because you use the high tech facial rec or easier-to-remember PIN instead) then open your command line as an Administrator (right click on command line and select “Run as Administrator”) and type in the following command:
$ net user Administrator *
This will allow you to set up your new password, re-type it and it will overwrite your previous one – keep it safe and for the love of God, remember it!
Right back to shiny app deployment, once you’ve collected yourself, scp your files over (I did it from my computer command line to the VM – rather than from PuTTY, but either way works):
$ scp -r file_to_be_copied user@DNSname:
(remember, you’re copying your files reclusively using scp -r, stating the file and then telling it where to send it off in the format of username setup on your azure “at” the DNS name or IP address again taken from y our overview page on your Azure, semicolon path – here I would suggest you send it to /srv/shiny-server and then mkdir a folder to keep your app files in – for this app I created the folder “TorenApp”)
Once you have all the necessary files (namely, your ui.R and server.R scripts needed to run your app – I also have an .Rmd file as users of my app can download written content, using the knitr package), open the .conf file to edit the directory of your app, in order to display it on the webpage. Under location, site_dir needs to be altered to app_dir. This instructs the Shiny Server to attempt to serve a single application hosted at the given directory:
# Define the location '/TorenApp'
location /TorenApp {
app_dir /srv/shiny-server/TorenApp;
}
If you reload your server (sudo reload shiny-server) and refresh your page, you should see your app – if you’re super-duper lucky! If not, move onto the debug section below!
the torturous debugging phase
This phase is never easy, because your app will function exactly how you want it on your local machine – which may be of a different operating system to lets say Linux – which is your current VM OS! The best way to debug is by checking your log files and looking at where the system breaks by addressing the errors. Use this command to get to your log files:
$ cd /var/log/shiny-server
$ nano appName-shinyuser-yyyymmdd-hhmmss-41509.log
Add the following line into your code (make sure to remove this later):
options(shiny.sanitize.errors = FALSE)
Edit your shiny server .conf file and add the following line after "run_as" :
$ sudo nano /etc/shiny-server/shiny-server.conf
preserve_logs true;
If you’re not that comfortable to work on command line, download X2go (https://wiki.x2go.org/doku.php), which is a lovely interface.
The go-to things I’d make sure I do in debugging if things aren’t working are:
· For one, make sure you have two files for your app, I had it in a single file and ran it through there, but the server here expects the format in two separate files in order to be run and hosted properly.
· It goes without saying that all your dependencies should be explicitly downloaded through your script – don’t be clever, pacman does NOT work. What worked for me was manually installing every package onto the VM and only loading library calls in the script. You only have to do this once, so might as well do it and create an environment your app runs on.
· Make sure the R version you used on your local host is the same as the one you have downloaded onto your VM, for the dependencies to work the same. I for example had to update to the latest version of R – you’d think this would by default be downloaded when installing R, for some reason it was running in an older version.
· Check that your script doesn’t include any relative pathways and explicitly write the new ones.
Well that’s a wrap! Hope you’ve found this blog interesting and please do get into contact if you face any other issues that aren’t mentioned here or that aren’t covered in the following (awesome) tutorials and blogs!
Awesome tutorials / documentation to help!
https://www.rstudio.com/products/shiny/download-server/
https://www.digitalocean.com/community/tutorials/how-to-set-up-shiny-server-on-ubuntu-14-04
https://sqlbits.com/Sessions/Event14/Shiny_dashboards_in_R
https://docs.rstudio.com/shiny-server/
https://stephlocke.info/Rtraining/shiny-server_build.html
https://www.top-password.com/knowledge/forgot-local-password-but-remember-pin.html