Spoiling you as usual, I have another exciting D3 example for today: merging historical maps! I’ve been meaning to cover this topic ever since I developed a similar project for my Master’s thesis 3 years ago. Merging maps is challenge-worthy for every D3 enthusiast as it requires a number of things to be aligned: the data format should be compatible with D3.js, the maps should be drawn in the same projection, and cover the same time period as country or regional boundaries are far from static. I will demonstrate the idea by mashing up two maps: a digitalised map of II Polish Republic from 1934 with European boundaries from 1939.
Off topic: If you sent me an email via the contact form in the past and I have ignored you – believe me I haven’t. It was the Google Mail filter that treated all incoming messages from the blog as spam (until a week ago). If that was a fun or supportive message – please resend! I am also on twitter as @EveTheAnalyst
I’m happy to announce that more SVG fun is coming! I’ve been blown away by the stats on my previous D3-related posts and it really motivated me to keep going with this series. I’ve fell in love with D3.js for the way it transforms storytelling. I want to get better with advanced D3 graphics so I figured I will start by getting the basics right. So today you will see me doodling around with some basic SVG elements. The goal is to create a canvas and add onto it a rectangle, a line, and a radial shape.
This post demystifies one of the most feared vector functions available in D3.js: the radial line, or d3.radialLine(). Radial lines are constructed with only two attributes: an angle and a radius. The product of the function is a line, but unlike the basic line function, there are no x and y co-ordinates. I fundamentally misunderstood the radial line logic the first time I used it – in fact I had to bring in my boyfriend one late Thursday evening to help me get it right. This guide should help you avoid my mistakes.
A couple of posts back I shared a sneak-peak of my webinar on data security for data monetization initiatives. That was one of many sessions we’ve ran for APAC as Protegrity entered that market in 2018. All sessions had been recorded and now I’m happy to say we’ve published excerpts of those videos on youtube. There are bits on Big Data protection, best security practices for analytical workflows, sessions on hybrid- and multi-cloud environments, as well as spotlights on specific technologies (AWS S3, OneDrive, Salesforce, Elastic Map Reduce, Tableau). The videos are 2- to 7-minute-long and go straight to the demo. I do sympathise with those of you that don’t have the time or patience to sit through a one hour webinar – I hope you like this compact format. Check out the playlist below and get to know the superpowers of Protegrity tech!
I have a million thoughts running through my head as I sit in my sun-clad apartment and look at the empty suitcase in front of me. There’s only a week left of 2018 and my flight to Poland is leaving in a couple of hours. Outside, the Andalusian bees go about their flower business. Unbeknownst to them the rest of Europe is scrubbing snow off their cars and porches. The set up is perfect to reflect on this year’s happenings.
A pretty specific title, huh? The versioning is key in this map-making how-to. D3.js version 5 has gotten serious with the Promise class which resulted in some subtle syntax changes that proven big enough to cause confusion among the D3.js old dogs and the newcomers. This post guides you through creating a simple map in this specific version of the library. If you’d rather dive deeper into the art of making maps in D3 try the classic guides produced by Mike Bostock.
Last Thursday evening I had the opportunity to talk about Hadoop at a Glolent Global Talent community virtual meetup. Glolent connects remote IT workers across the globe and facilitates skill-sharing sessions that any member can join or present at. I rarely demonstrate Hadoop anymore so I needed to take a couple of evenings to brush up on the fundamentals. The talk is a distilled product of that study. I approached Hadoop’s architecture from a historical perspective: I started the talk by introducing the root problem – I/O bottleneck in processing Big Data – and positioned Hadoop Distributed File System as its panacea. There was an obligatory intro to the original processing paradigm on Hadoop: MapReduce, and a classic word count example of Shakespeare’s collected works. That was followed by a review of programming abstractions built on MapReduce and some alternative processing engines to MR – with an emphasis on Spark. A 30-minutes talking slot is very little time so I had to cut out any mention of resource or cluster management. You can judge for yourself how it turned out – I shared the recording below.
Point-in-polygon is a textbook problem in geographical analysis: given a list of geocoordinates return those that fall within a boundary of an area. You could feed the algorithm a list of all European cities and it will recognise which of them belong to Sri Lanka and which to a completely random shape you drew on planet Earth. It applies to many scenarios: analyses that aren’t based on administrative boundaries, situations in which polygons change over time, or problems that aren’t geographical at all, like computer graphics. Not so long ago, I turned to point-in-polygon to generate a set of towns and villages to plot on a map of Poland from 1933. Such list has not been made available on the web and I wasn’t super keen on typing out thousands of locations. Instead, I used that mathematical cookie-cutter to extract only those locations from today’s Poland, Ukraine, Belarus, and Russia that were present within the interwar Poland boundaries. In this post I will show how to perform a point-in-polygon analysis in R and possibly automate a significant chunk of data preparation for map visualisations.
Previously we used OGR2OGR to extract a couple of features from a large geographical dataset. OGR2OGR can do so much more – today we’ll look at its reprojecting capabilities. Reprojection is a mathematical translation of a dataset’s coordinate reference system to another one, like Albers to Mercator. Sometimes the geographical data we receive has to be reprojected to conform to our other datasets before further use. My first encounter with mismatching projections came about during the works on my master thesis project. I struggled with a point-in-polygon function that was supposed to filter a set of points based on a geographical boundary and stubbornly just wouldn’t return anything. I soon found out that my map was digitalised in EPSG:3857 (AKA web mercator used by Google maps) projection and my village coordinates used WGS84 coordinate system. That’s how OGR2OGR and me met.
Data and analytics are fundamentally changing the way business is done. Make sure you’re ahead of the game by enabling your strategic data monetization initiatives while keeping your customer and commercial data secure.