OpenSource Technical Infrastructure: PostGIS, Geoserver and Django …

The technologies underpinning Stepping Into Time will take advantage of free or low cost computing resources and open-source software now prevalent in web and app development. The final technical infrastructure is being finalised but this post summarises the fundamental components.

Technical Infrastructure Diagram
Stepping Into Time Project

Server Side 

The server-side of Stepping Into Time will be built using the Python GeoDjango web-app framework, supplemented by Geoserver, with the generated data hosted in a PostGIS database. The server will run in a industry standard Linux-Apache server environment, deployable on a wide variety of server providers (eg Amazon EC2,  University hosted – to be determined).

The PostGIS database provides an appropriate method for data storage, management, spatial retrieval and processing, and is the most appropriate Spatial  Relational Database Management System (RDBMS) for integration with the rest of the proposed technology stack, ie it is fully supported and the recommended spatial RDBMS for GeoDjango. It will hold the geographic data  in the format of either a point, line or an area (ie vector data) for example each bomb that fell is marked at the street level as point with an X, Y coordinate that denotes its location.

The original maps are scanned they are processed in the desktop GIS to add a geographic reference system  – a method known as geo-referencing (we will use the low cost GIS called Manifold), they will be saved in the GeoTiff file format and saved on the server. The middleware Geoserver will then enable the geographic data to be accessed and served to the tools we are developing, in a variety of standard formats (eg KML, SHP, Map Tiles and WMS georeferenced images) by either the desktop GIS, mobile application or web-mapping application.

GeoDjango is a web framework designed to facilitate the building of GIS-based web applications, and to enable the use of spatial data on the web.  It is based on the Django framework, a Python framework originally designed to handle fast-moving news websites.  Django offers an in-built administration interface, along with the ability to define database data models and queries, and a template language to separate design, content and code.

Client Side

The project’s client-side will make use of modern Ajax design standards to achieve an easy-to-use,  interactive user experience. This will be achieved through a combination of HTML, CSS & JavaScript, facilitated through JQuery, with the essential web-mapping and geo-data contribution mechanism enabled via the Leaflet framework.

The Leaflet open source JavaScript library will be used to provide to end clients (web and mobile) both access and display of various base maps as well as custom data, along with spatial interaction functionality.  JQuery provides a framework of JavaScript functions simplifying the development of rich and interactive applications.

Posted in project details, technical | Tagged , , , , | 1 Comment

About the data we are using (part 2): The Bomb Maps

This post describe in more details the two different types of bomb maps that we will be using in our tools:

  1. Aggregate maps of nightly Bomb Drops during the period October 7th, 1940 to 6th June, 1941 (archive record: HO193/13) [35 Map Sheets]
  2. Weekly plots showing day of week and bomb type: First week of the reporting period  7th – 14th October 1940 for Central & East London only (archive record: HO193/01)  [8 map sheets]. Weekly maps before this time are not available.

Index map for H0 193 13 map series (photo taken from TNA archive record HO192/ 1547-1595)

Aggregate maps of Night Bomb Drops: 

Zoom in of a Night Bombing Aggregate Map, photo taken from HO193/13 map series (Crown Copyright, The National Archives).

  • The detailed maps are at a scale of 1: 12:500 and there are number of maps at 1:25000  for central London – as per the images shown (HO193/13).
  • The London Region (Civil Defence Region 5) is divided into a series of tiles (map sheets) based on the Military Grid Projection System (MGPS).
  • There are 35 different map sheets in the archive that cover each of the tiles.
  • For each bomb dropped a point on the map has been recorded to show the location of where it fell
  • The maps were labelled SECRET but declassified in 1971. 
  • The original maps are very fragile due to the quality of paper available at the time.

Weekly plots

Data collection map tiles for the weekly plots of the first week.

  • 2.5 inch map sheets for each week
  • Plots showing all the bombs that fell that week
  • For the period of the Blitz there are 559 map sheets
  • The project will use weekly bomb plots for the First week of the reporting period- for Central and East London (7th to 14th October 1940) (HO193 / 01)
  • This will provide an overview of daily intensity and proof of concept to hopefully enable the remaining weekly maps to be integrated
  • Maps record point of impact, a symbol is used to classify the type of bomb
  • Each point of impact is in a different colour to denote the day of the week in which the bomb fell.
  • The maps do not record size of bomb or bomb numbers
Posted in data, project details | Tagged , , , , , | 1 Comment

About the data we are using (part 1): How we selected the archive map data to use

To develop a manageable set of data  we had to select from a vast array of archive information. This blog post describes the process of narrowing the focus by selecting a time period, a region and a sub-set of maps.

The Stepping Into Time  project is now using the Bomb Census Maps as the primary data in the tools we are developing (see previous post on ideas to help prevent manifestation of  IPR issues). The maps are part of  an extensive array of material collected during the Bomb Census Survey 1940 to 1945, organised by the Ministry of Home Security. The records are held in The National Archive (TNA),  and we will be using the maps with a non-commercial education licence. This blog post describes how we selected the archive data that will underpin our tools.

Examples of archive documents from the Bomb Census Survey 1940 – 45: The National Archives

Due to the sheer volume of records, photos, documents and information collected during the Bomb Census Survey – the project had to carefully select data to prevent  information overload which might mean we miss our completion deadlines. For this reason we spent some time before Christmas at the Archives with one of TNA’s map archivists Andrew Janes. The image shows a snapshot of the types of documents collected during the bomb census. The array of information had the purpose of providing the government with a complete picture of air raid patterns, types of weapon used and the damage caused  (TNA, 2011).

Selecting a Time Period

To select a manageable data set we first decided to restrict the time period of the project to the period of the most intensive bombing  known as The BlitzThe word ‘Blitz‘ is derived from the from the German term ‘Blitzkrieg’ (lightning war) and is associated with a period of continued aerial bombing by the Germans on Britain  (Imperial War Museum, 2011).  The Blitz started on the 7th September 1940 and continued until the spring of the following year. The  maps in the archive do not completely align with the time period of the blitz so there is a slight mismatch but the reporting period we are investigating is between October 7th 1940 and June 1941 as this is the period in which data were collected.

Selecting a Study Region

Region 5 London Civil Defence Groups
(photo taken from TNA archive record HO192/ 1547-1595)

During the war Britain was divided into Civil Defence Regions – of which there were 12.

London was recorded as region 5 and was sub-divided into 9 Civil Defence Groups. The project will be exploring Bomb Census Maps from across Region 5. As shown in the picture to the left (please excuse my rather poor photography !).

The other regions were as follows:

  1. Northern Region
  2. North Eastern
  3. North Midland
  4. Eastern
  5. London
  6. Southern
  7. South western
  8. Wales
  9. Midland
  10. North Western
  11. Scotland
  12. South Eastern

Selecting the Bomb Maps

The primary focus of the project is not about digitising maps but about clustering different data. Therefore having selected a time period and a study area the final step was to select which of the archive data to use. We have selected to look at two types of map from the HO193 map series in order to (1) obtain an aggregate view of all the bombs dropped during the period of interest and (2) see in detail the first week of the reporting period.

  • Aggregate maps of nightly Bomb Drops during the period October 7th, 1940 to 6th June, 1941 (archive record: HO193/13) [35 Map Sheets]
  • Weekly plots showing day of week and bomb type: First week of the available reporting period : 7th – 14th October 1940 for Central & East London only (archive record: HO193/01)  [8 map sheets]

Data collection map sheets for the weekly plots of the first week.

There are aggregate maps for daytime bombings but the records or not complete as the maps for a 3 month time series appear to be missing within the archive. As we wanted to use as complete data as possible (withing the constraints of what data is available to use) and most of the bombings were at night we selected to use maps of aggregate night bombings which covers a large proportion of the time period known as the Blitz.

We had to limit the number of maps scanned for the weekly plots because for region 5 and the period of interest there are over 500 map sheets in just the HO193 map series.  It would not be possible to geo-process this number of map sheets with the resources and time available, therefore we selected to use the 8 map sheets that approximately represent Central & East London as highlighted in the image.

Watch the blog for forthcoming blog posts describing the detailed information about the two different bomb maps  (HO193/13 and HO193/01) and the other data we are clustering them with.

Posted in data, project details | Tagged , , , | 4 Comments

Presentation to JISC GeoCultures Event, Localising Wartime Past: London’s Blitz

For those of you that missed the event a few weeks ago – here  is an introduction to the project that I presented at the JISC Geco Geocultures event in the beginning of March 2012.
Posted in dissemination, project details | Tagged , , , , | Leave a comment

Developing Wireframes for Blitz Bomb Census Website

This post summarises the development of an initial wireframe for the Stepping Into Time project. The very first user described the wireframe as , “familiar and intuitive” and provided a few points for further consideration. The wire frame that has been developed is based on the principal of parsimony – ie less is more, a notion of interaction design that was explored in my paper titled “A less is more approach to geovisualisation”  in the  International Journal of GIS. At the heart of the  design of the wire frame for the project is a simple, clear and intuitive interface that is easy to use and requires little effort to learn.

What is a Wireframe? 

Wireframes are a useful tool that enable users to explore and experience the proposed design solutions for a website and or mobile app. They are part of the varied set of tools used by User Design experts to assess how users experience a particular interaction design. A wireframe  is a diagram that highlights where  different interaction elements will be located on a web page.

There are various methods of creating wireframes:

  • Draw by hand – ie create design sketches
  • use Visio of photoshop or draw tool bars
  • Use specially designed software – see this link : 10 tools for creating wireframes

The Stepping Into Time website Wireframe – version 20! 

To develop the wireframe that we will explore with our users, is part of an iterative design process based on the user stories that are being developed as part of the User-centred design process. We started with one initial sketch and tweaked it and changed the placement and organisation of the different interaction  elements until we had an initial draft wireframe that we are happy to start testing exploring with our user group and update according to their feedback.

Here is our initial wireframe, which I have annotated to describe the motivations behind the design. (Click on the image to see the enlarged view)

The most important element of the interface is the map. Therefore, the wireframe has been designed to ensure that map takes up the majority of the window.

The search tool bar is located at the top left of the interface. In  experiments conducted by myself and Patrick Weber on the editing interface of Open Street Map,the top left of a mapping interface is the interface location where users expect to find a search bar.

I have grouped together functionality that enables the user to search and explore the data  to an area just above the map  (if there were on the map it would become to cluttered) and have key map interaction elements (Bread crumb navigation trail; map controls; map layers) discretely on the map at the top – a location that eye-tracking has shown to be easily locatable by users.

I have attempted to ensure that the interaction elements follow the F-statistic scanning pattern that has been well documented for web pages. Whilst trying to ensure that the area around the optical centre of the map remains free of interaction elements.

To develop the wireframes I used PENCIL, an open source graphical interface prototyping tool – that is nice and easy to learn. I mocked up a series of pages and then linked the various interface elements to these pages. Pencil is useful as it enables you then to save your wireframes as a simple HTML document where users can click on the elements and follow the links – therefore we end up with a very simple mock HTML proptotype.

One user’s first impressions of the Stepping into Time wireframe

Today, I thought I would explore the initial user response to the wireframe. I found a willing participant, a researcher interested in the project and walked through the HTML prototype wireframes.  The image above shows a mock up  of a few of the pages –  to give you an idea of the interaction experience.

The experience of my first user is summarised as follows :

  • Interface seems familiar
  • Keeping the header content small will be useful for users viewing the website on a 1024*768 screen (Browser display statistics)
  • Interface is intuitive
  • Explore – not sure what this is telling me. The wording does not have enough meaning and does not indicate that you can view data for different times.
  • More – if this option only reveals a couple of links then they should be made available on the interface not hidden away in a more button (requires extra clicking).
  • Linked the natural order of the interface that enabled searching by location to filtering by time
  • Legends – legends for the temporal data could be included in the layers drop down menu
  • data classification – where data information allows should be organised by bomb type not time period such as day of week.
  • What type of searches can I use? perhaps add in a user prompt to indicate what is expected ie “search for street”

More user engagement will be following shortly, to identify more interface improvements and ensure the interface is relevant . Thank you also must be extended to my very first user.

Posted in User-Centred Design | Tagged , , , , , , | Leave a comment

Review of Geocultures event – Geospatial in heritage domain

Some great projects presented their work yesterday at the JISC GECO geocultures event which explored the exploitation of through geographical approaches and the types of tools and techniques that can be used with geo-referenced/geotagged content.

Here is a brief summary of some of the discussion and presentations which raised points relevant to the Stepping Into Time project.

Dr Humphrey Southall, University of Portsmouth  from the OLDMAPSONLINE project provoked discussion around the “unjustifiable certainty about historical locations” that is seen in a number of historical geospatial projects. The unjustifiable uncertainty arises from the pin pointing of places to precise locations marked by a point or polygon. Thus confusing the notions of location versus place. The issue is that the concept of place is more than a definition defined by set of coordinates. Places are fuzzy. Does a label on a mark really demarcate the location of places. Humphrey suggested these types of projects are more suited to a geo-semantic approach which focuses on the explicit relationships and linkages between places.

A project I really liked where I see a few methodological synergies with our Stepping into Time project is the JISC funded Locating London’s Past: search a wide body of digital resources relating to early modern and eighteenth-century London. The project is built around the GoogleMaps API. What was interesting we the presentation focused upon the  lessons learnt around the selecting of this particular API and the forthcoming limitations of its new pricing policy. Dr Jamie McLauglin, University of Sheffield talked about the retrospective examination of using OpenLayers instead. 

A previous JISC  project, (Bridging the gaps between GIS and the Geoweb: IIGLU) I was part of had a discussion of the merits of OpenLayers versus GoogleMaps API . The  project originally started of exploring the adaptation of an existing Google maps API code base but decided to switch to a structured web development framework,  (Geo)-Django, and enabled us to use OpenLayers instead to take advantage of greater capabilities and openess of this library. For the stepping into time project we are prototyping using GeoServer and (Geo)-Django.

 It was good to see PhD students Ashley and David from the Adaptable Suburbs project present the work completed so far. This project is a follow on to a project at UCL called Successful Suburban Town Centres – which I worked on when I was a post-doc.  The new adaptable suburbs project  is exploring  networks of human activity and the form of suburban centres. The team used an interesting piece of software called RX spotlight with an educational licence to batch process the vectorisation of  old maps – the results looked very impressive. I am certainly interested in exploring the results of this software compared to QGIS, ARCGIS or Manifold to see if it is effective for our project.

This is just a small sample for the great presentations and project, if you missed the event live blogging was available, so please take a look. A will upload my slides to the blog later this week.

Posted in dissemination, historical map snippets | Tagged , , , , , , | 1 Comment

User-Centred Design, User Experience Vision and the Stepping into Time Project

…Focus on the the user and all else will follow” this is a quote from Google’s philosophical approach. It encapsulates the idea of putting the user at the heart of the software design process.  The reason, good user-centered design (UCD) enables system learnability, memorability, productivity and trust to name but a few.   In essence engaging the user throughout the design and development process will increase the likelihood that the tools developed are useful and usable and result in a positive user experience. For this reason user-centred design is an important component to the project.

The Stepping into Time project will adopt a user-centred design approach to help ensure that we develop a set of engaging useful and usable tools. There are many techniques that can be taken to embrace UCD and for the project we will define a user experience vision, a set of user stories based upon the vision and the develop wire frames and HTML prototype so the user group can explore an early interface mock-up  and we can review the user experience.

A user experience vision helps to focus development by enabling the project team to construct a mental model of the experience a user will have when using the tools we have designed. So here is my first go at the User Experience Vision for the Stepping into Time Project……

Stepping into Time user experience vision

Stepping into Time links together different types of data related to London during the Blitz, aimed at students and researchers (professional, academic or citizen researchers). Using the web and mobile tools we develop, you will be able to explore and discover where the bombs fell and the damage they caused, identifying the lasting impact upon the urban landscape, as well as uncover images and statistics for your local neighbourhood or area of interest. This project makes previously difficult to access archive content available online providing new interactive representations of the data for the first time.


Posted in project plan, User-Centred Design | Tagged , , , , | Leave a comment

Ideas to help prevent manefestation of IPR issues

Historical maps are not exempt from copyright and their associated restrictions and working out IPR issues with an archive during the grant writing process will remove a number of unforeseen complications. My inexperience in this matter revealed an issue which may be useful for others with similar ideas to be aware of.

Smaller archives use historical maps as part of their business strategy. This is because certain maps contain valuable information pertaining to the location of different types of features in the built environment. Consequently, this information can understandably be used as a mechanism for generating much needed revenue. Maintaining control of the maps and their digital counter parts helps to ensure the copyright is not infringed and the maps and not widely circulated.  Therefore, it is important to be mindful of this when writing a project proposal.

The situation is further complicated, by the motivations of funding bodies who are charged with ensuring that public money is spent wisely by ensuring projects achieve maximum impact for educational use.

For example projects using historical maps can maximise impact and the resulting educational use by applying a process known as geo-referencing,  which  embeds the spatial coordinate system of the British National Grid within the digital images of the historical map, an example can be seen in a previous  JISC project. Once this process has taken place the maps can be deposited in an educational map repository such as DIGIMAP (a university subscriber service that provides access to digital map data to FE/HE). This ensures that the maps are accessible and reusable for educational users. Using the original maps in their digital form are still subject to copyright and requires permission to reuse the material. Using such a repository means the  maps are more likely to become more widely used for educational purposes.

This opening up of data can be problematic for smaller archives because it means their revenue streams can be impacted by the wider availability, albeit in the education domain, of their historical maps –  even if  copyright limitations are still attached. The result is they are are then understandably reluctant to provide access to the maps in digital form.

In my original proposal to JISC I did not include exploring the depositing of the maps within Digimap but it was suggested that it would be a useful addition to the project – of which I whole heartedly agree and the feasibility of which is currently being explored. Although this suggestion made my original project partners uneasy (for reasons I understand and have described above). The result was the manifestation of an unforeseen, unknown risk –  the withdrawal for the partners from the project. Obviously this action was a huge disappointment but one that I do understand.

This is an issue which I have experienced first -hand and sadly resulted in the requirement to source historical maps from an alternative archive. For anyone starting out as an early career academic in a similar position here would be my recommendations, which may to prevent such similar issues happening to you:

  • Speak to the archive in detail during the writing of the proposal – this can take time (even months) as everyone is busy, so allow plenty of time.
  • Speak with the funders about the project idea whilst writing the grant as they can make you aware of exactly what they are looking for.
  • Describe in detail the scope of the project  to the partners following discussions with the funders – providing a project brief does not contain enough information to resolve IPR issues.
  • Consider the following discussions in the bid writing:
    • Use and reuse for educational purposes
    • Understand how they use their archive material and its value for income generation
    • Meet with the archives in person, this is much more efficient than email exchanges and telephone calls
    • Have a contingency plan on the back burner – don’t wait for the risk to manifest.

Good Luck!

Posted in project details, project plan | Tagged , , , , | 1 Comment

British Library Georeferencer

Georeferencing historical maps is a time consuming process as historical maps are matched to a contemporary map with spatial coordinates by clicking on a point in the image that matches a point on the map. The more points you locate the more accurate the image will align with the base map. The process is a key element to all historical GIS projects.

In the Stepping Into Time Project we will be georeferening historical maps of bomb locations. There are a number of crowd sourcing georeferencing projects from a number of the larger archives including this great one from the British Library. In the British Library interface you can select from a wide variety of base layers to use as your mapping base including: OpenStreetMap, Ordnance Survey Open Data UK, Google Street View/Hybrid and Terrain.

It is really fun and I have just tried to georeference a map of Abergavenny from 1813. It is not that easy because the place names have changed (and are in Welsh). To complete the process I used a variety of the base layers available. Including Google Terrain to help locate which valley I was in, Ordnance Survey Open Data and OpenStreetMap to locate the actual villages or farms to act as control points.  The historical map marks all the old mines (which is fascinating) but these are not marked on the modern base maps so it is a little tricky to align despite knowing the area. What is nice is you can review the map you have just georeferenced in Google Earth which is embedded into the British Library webpage. I need to go back and improve the accuracy of my work.

It is quite a fun thing to do and is slightly addictive – like doing a jigsaw puzzle!

Posted in historical map snippets | Tagged , , , , , , , | Leave a comment

Importance of Project Risk Identification… Unknown Unknowns (becoming Known Unknowns)

So whilst I have not blogged a lot this since the beginning of the project a lot of work has been going. This blog will attempt to be very frank in an effort to share the experiences as the lead of my first academic grant (which is rather exciting). The project has had an interesting start to say the least.

The JISC project bidding process requires the completion of a grant funding document. This is a proposal of work that enables the funders to determine if the project is a good idea and is viable. There are many components of this proposal one of which is a section on project risk. In August 2011 when I was putting the bid together I identified 7 risks which if they emerged could jeopardise the success of the project. Risks included items such as lack of user engagement; technical problems with data clustering, changes in user requirements and unavailability of the technical developer. These items are fairly standard on a risk register.

Once the proposal was accepted, the next stage of the project management process is to develop a project plan. This is a detailed document, of which a gantt chart is just one component.   Once completed it is made available to the public via the JISC website (the plan for this project is almost completed and will be released in the coming weeks). Again as part of this plan a risk analysis forms a fundamental element. My risk register now contains 20 items including risks associated with: IPR of historical maps, delays in obtaining historical maps due to workload at archives and time constraints of university processes. So now some of the unknown risks are known risks and I can take action reduce their impact when and if they manifest. Maintaining this register, anticipating risk and seeking alternative solutions that rework the work packages accordingly, will ensure the project completes successfully. A forthcoming blog post will share my experience of IPR issues which led to a change in the historic bomb census maps that will be used in the project.

Posted in project plan | Tagged , | Leave a comment