Location, Location, Location…

By Joe Howell, Oil & Gas Market Manager

Some of the most common challenges I have seen in Oil and Gas companies are related to using the data that they already have.  There are a multitude of operational, efficiency, regulatory, safety, and profitability issues that can be addressed by using a spatial system.  Today I will discuss 4 opportunities for improvement in this dynamic industry.  I will spend the next 4 weeks going into more detail about each.  The 4 categories I will address over the next month are:

  • Aerial inspection of Pipeline and ROW
  • Using GIS to integrate disparate business systems
  • Securing your GIS and improving performance
  • Enabling business innovation through GIS

Pilot Patrol

Let’s face it, aerial inspection is a difficult job.  You have to concentrate on little things like wind speed and direction, altitude, fuel, and any incidental air traffic (crop dusters beware!).  Taking notes on a knee pad about encroachments and other ROW issues is an art.  The use of a simple location-centric application can significantly improve the ease of capture and the quality of ROW information.  Let the system record the location, capture and send a picture of the encroachment, and record other needed information at the touch of a button.   Let the pilot concentrate on flying.  As an example, the Pilot Patrol application used by Energy Transfer Equity (ETE), L.P. improves safety and efficiency.  It eliminates the need for pilots to take detailed notes about the event and its location while they are flying and it significantly streamlines backend workflows.

Integration

Anyone who has worked in Oil and Gas IT knows how difficult it is to collate information.  From Lease Management to SCADA, the number of applications and processes used to track product from well to the end of the gathering system is staggering.  So how do you ever get the “big picture”?  Location to the rescue again; nearly every business system in the industry allows the capture of asset locations.  Capturing the latitude and longitude for simple assets will set you up for a fairly straight forward integration process with other systems.

Location information is the natural integration point of the many disparate systems that it takes to manage the complicated network of assets and activities that surround the entire industry.  Chances are that you already put it on a map; the next step is to let the map do some more work.  GIS brings flexibility to this space by providing a platform to present a clear concise summary of your holdings.

Performance

The performance of existing GIS systems is largely dependent on planning and execution.   Data management and workflows must go hand in hand.  If performance or usability drive personnel back to paper maps, reviewing the organization of the data and the workflows around capture and delivery of this data may provide surprising benefit.   The underlying health of the data impacts everything down stream.

One of the major challenges many companies face is the all too prevalent gap in knowledge between GIS and IT staff.  There is an even bigger gap between management and both groups.  When designing GIS applications, the needs of the individual users are paramount.   GIS database design should be done with personnel who understand both database architecture, spatial data, and the needs of the business.  Doing it right enables innovation, performance, and flexibility in the delivery of information to all levels of the business.

Innovation

Nearly every business person in America is carrying a smart phone.  They use Google/Bing/Yahoo mapping applications and mobile email, web, and productivity software.    This mobile acumen opens up new opportunities for innovation.  Asset management, supply chain, emergency response, weather, and just about anything else you can think of can be done through mobile applications.  By leveraging technologies like Esri’s ArcGIS Online, you no longer need to rely on printouts that are obsolete before the ink is dry.  If you can take the time to understand the needs of the individual, you can build an application that will streamline their workflow.  In short, if you want to do things different and better, think about GIS.

More to Come

Next week I will be discussing in detail the Aerial Patrol application and how it improves quality and communication for pipeline inspection and right of way management.

Advertisements

An ArcGIS GeoProcessing Model for the Obstruction Analysis of Military Airfields

By Ryan Warne

One of the major encroachment issues facing military operations are airfield obstructions.  Aside from natural objects such as hills, mountains, and trees, human construction such as towers, buildings, and wind turbines are continually planned for sites that have a potential to impact safety associated with military operation on airfields.  While a wind turbine several miles away from an airport may seem perfectly legitimate, there are specific height restrictions that must be maintained to protect incoming or departing aircraft.  The Federal Aviation Administration (FAA) provides these requirements that all new or altered constructions near or on an airfield be subject to an Obstruction Evaluation.

As part of the Research and Development program at GISi, I took the opportunity to develop an ArcGIS tool that could be used to provide acceptable locations for new or altered structures around the larger types of military airfields, Class B.

Image 1. A 3D model of a Class B Imaginary Surface

My R&D project was a big success. I was able to create a tool that automated the creation of obstruction zones around Military airfields based on specific user inputs.  It took a while to develop and there are certainly areas of improvement, but this was a proof of concept that a model could be created using out-of-the box tools from ArcGIS 9.3.1. Since the model uses Navy data, and the intention was to empower installation planners to perform site analysis, 9.3.1 was used over 10.0 due to the Navy’s current software availability.  Version 10.0 provides several improvements that would increase the efficiency of the model, but it can be redesigned to that version fairly quickly for non-DoD users.

Image 2. A Class B Imaginary Surface based on NAS Jacksonville runway 10/28.

The tool operates by analyzing the topography of the landscape and comparing this with the FAA Part 77 imaginary surfaces surrounding the military airfield while compensating for user inputs of airfield elevation and the obstruction height – all used to produce an Obstruction Zone (lower right corner of Image 3).  The trick was to create a model using data in the format that it is typically available in for the Navy (e.g., contour lines for elevation) so that this could be usable at other installations.

Image 3: Raster topography and TIN imaginary surfaces combine to create an Obstruction Zone

The complexity of the model is portrayed in Image 4. Recognizing that the reader can’t see the details of the model, here is an overview of the tool’s processes:

  • Create 4 TINs based on the outer transitional slope, inner transitional slope, approach departure surfaces, and the horizontal surfaces of the imaginary surfaces
  • Convert the topography to TINs
  • Modify the topography to subtract the runway elevations and add the Obstruction Heights
  • Compare the adjusted topography and Imaginary Surface elevations
  • Create raster grids of “above” or “below” Imaginary Surfaces
  • Create a subset of the values “above” the Imaginary Surfaces
  • Convert the “above” raster values to vector

Image 4: Airfield Obstruction Model zoomed into the Raster Minus comparisons and Raster Mosaic

The end-result is a polygon that shows areas where the user input value of “Obstruction Height” would penetrate the imaginary surface at that given location (based on topography).  These would be the areas that should be ruled out for new construction or to monitor existing features at or above the “Obstruction Height” entered by the user.

For the test location of Naval Air Station Jacksonville, the model took about 16 minutes to process, mostly due to the size and resolution of the elevation model. The processing time could be improved greatly through a few modifications.  For example, it would be much faster and effective to have specific topologies already loaded and processed into their raster format.  This would save the local computer from having to perform the transformation from vector to raster.  It would also be an improvement if the Model Parameters allowed the user to specify the raster resolution as well as the contour details (subset every 1 ft, 2 ft, 5 ft, etc.).

Another benefit would be to add options to allow compatibility with different airfield types, such as Class A, Outlying Fields, or civilian runways.  While Class B runways are the largest and typical for the bigger Navy installations, adding each of the other airfield classes to the options would complete the toolset for creating obstruction zones.

Right now the model only works if the imaginary surfaces area already created and attributed correctly.  Additional functions so that the user only needs to specify the runway classification as well as two points, one for each end of the runway, would greatly improve the usability of the tool.  Despite all of these potential improvements, the model proves that it possible to create a tool in ArcGIS 9.3.1 to determine obstruction zones for airfields.

Esri Location Analytics Tools in an Organization: Embedding an Application Served from ArcGIS Online into a SharePoint Page

In my last post I took a map that I had authored in both Esri Maps for Microsoft Office and ArcGIS Online and created a web application that I could use on my desktop, tablet, and phone.  Now to complete the requirement I had originally I will show you how to take that same application and embed it into a SharePoint site.  Admittedly, if the end game was ONLY to have a map in SharePoint I would have started in SharePoint using the Esri Maps for SharePoint application. In this case, I had a multipurpose need and went a slightly different route.

To refresh you, the image below is the simple application that I created using the ArcGIS Online application template. Not the prettiest, but functional.

So first, I want to go back to the Summary page for my application. Go to MY CONTENT and click on the title of the application. You will see something like this.

All you need to do here is COPY the url link to the application (paste it into Notepad or somewhere else to use later).  Once done, we are done with ArcGIS Online directly.

Next it is off to SharePoint.

At GISi we have a corporate intranet site that is built on SharePoint 2010. Within this we have a Site that has Pages for all of our offices (addresses, contact info, etc.). What I want to do is embed the hotel application within this page so that anyone within the company can use it.  To start I simply click on the Edit this Page link.

This brings up the editing Ribbon.  I know that I want the Hotel Application to show up in the main part of the page just below the main block of text indicated by Zone 1.

To add the application I have to create a container – web part – to hold the application. So within the Zone 1 block I click on Add a Web Part.

NOTE: Your SharePoint Page may not have a zone but you should still be able to add a web part and configure it as I will do here.

Clicking Add a Web Part changes the Ribbon to a Explorer interface from which to choose the web part you need.  I know I want a Page Viewer web part which is under the Media and Content Category.

I select that and click the Add Web Part to: link – making sure I have the correct Zone indicated.

The result is an empty Page Viewer Web Part.

Next I click the Open the tool pane to configure the web part. And the following dialog will open on the right of your page:

In the URL space, clear out any text that SharePoint has in there and Paste the URL from your application in here.  Once done, you can click the TEST LINK link to make sure that the application loads.

You will also want to open up the Appearance tab and do some additional configuring.  For instance, Add an informative Title.  You will also need to set the Height and Width of the application within the area designated for the Page Viewer web part.  You can do this through trial and error. Set a value, hit Apply and keep going until you have what you want.

Here is the result for our application:

One note, I discovered that for some reason embedding the application in a SharePoint Page causes issues with rendering the fancy 3D graphic I had originally selected for the Hotels. To fix this, I simply went back to the map in ArcGIS Online, changed the symbol to a simple geometry and re-saved the map. The application automatically recognized the change.

So this completes my Simple Consumerization of GIS Trilogy.  In this 3 part series I was able to:

  • Add a map within an Excel spreadsheet using Esri Maps for Microsoft Office and ArcGIS Online,
  • Publish the Map from Excel to ArcGIS Online,
  • Make additions to the map in ArcGIS Online,
  • Create and publish an application using my map that worked on my desktop, tablet, and mobile phone, and
  • Embed the same application in our Corporate SharePoint intranet site.

…and all without writing a line of code!

It’s not a stretch of the imagination that our folks across the company could follow these simple steps to create layers with hotels and restaurants and even preferred running routes near our other offices add these to the Map in ArcGIS Online, and by proxy, to the application.  Almost sounds like crowd-sourcing. Think about amazing your friends when you pull up a on your iPhone a rich mapping application that you created !

Esri Location Analytics Tools in an Organization: Using ArcGIS Online to Publish a Web Application for Distribution

In my last post I showed you how you can use Excel and Esri Maps for MicroSoft Office to create a useful map within Excel and then publish it to ArcGIS Online so you could share the map.  Now I want to walk through the process of using ArcGIS Online to enhance that map, create an application that can be used in a desktop, tablet, and mobile phone browser.

So the first step is to log in to ArcGIS Online. (http://www. arcgis.com).

Enter in your credentials. (Quick note, your username is now case sensitive – a change from other Esri sites).

Once you log in you will be taken to the Main Site. It will look a bit different for you than the screen shot below; as we have an organizationally branded site. The Banner across the top will be identical though.

So first off, I want to see where my map of Hotel locations is published, to check on that I simply click MY CONTENT.  I can see everything that I have published to my account here. You can see that both the Hotels Feature layer is there as a Feature Service and the Map is there as a Web Map, and both are shared to everyone.  I am able to select any one of the items and change the status, delete, or start using them in the map authoring tool here in ArcGIS Online.

I want to Open up the Web Map and work with it a bit to get it ready to for use in an application. So I just double click on the name and I get the Summary page for the Web Map. This shows me all the information I added during the publishing phase and the URL links for both the Hotel feature service and the Base Map feature map service (Topographic).  It also lets me start a Map session by selecting the Open Button and select the Open in ArcGIS.com map viewer.

This starts up the Map Viewer which gives me some options to enhance my map. I can change the base map if I decide I don’t like my original choice, change the way the pop up is configured, change the layer styling, etc. All the things I did in Excel I could have done here. But since I have shared the Feature service someone else can take that and set it up the way they like and re-share it as a new map.  That reminds me, I just realized that this would be a much more useful map if the folks looking for hotels also knew where our office actually was.

Now I don’t remember if I have published a feature service with our office locations myself or if someone else in the organization has, but it doesn’t matter. I can search and see what is there. Selecting the Add drop down from the main panel and choosing Search for Layers, I can see if there is a layer available that makes sense to use.

This brings up a search panel from which I can search through my own content, all of the shared content in My Organization, or all of the content available in ArcGIS Online, and the web for that matter.

Oh look, I did publish a GISi Offices feature Service already,

I click the add button and that feature is added to my map.  Again, this data could have come from anywhere that I searched.

Now that I can show where the office is relative to where the hotels are, I have a much more informative map.  Maybe later someone in the company will get excited and create a service with the local favorite restaurants or places to visit too.

Next I will simply save my map by selecting the Save drop down. I could do a Save As, but don’t need to in this instance.

Now let’s get this Map ready for Prime Time and put it to work. I want to embed this Map in an application that I can in turn use directly from the web, on a mobile device, and insert into a SharePoint site.  I can do this all without pestering a programmer (I prefer to let them wrestle with tougher problems anyway, I think they prefer that as well).

To start, I simply click on the Share button above my map (right next to the Save button I just used).

Wow, look at these options. I have a direct link to the map, I can Share the link through Facebook and Twitter, I can embed the map into a website, and I can Make a Web Application. I can also restrict usage of anything I do here too. One thing to note, at whatever level you share a Map or Application, your features need to be shared at that level as well or they won’t work.

So let’s look real quick at what happens when I click on the Embed in a Web Site button.

Look at that. Man, I can just cut and paste this code into a web application, like our corporate web site, set the size I want the map to be at and what navigation tools to add and BAM we are in business.  This is a little too programmy for what I am trying to do right now though so I am going to go back and take a look at the Create an Application path and see where that leads.

Okay, this looks interesting.  What we have here are a series of template applications waiting for a map. The best thing I can tell you here is to explore each one of these to see what they do and what they look like. Esri makes this easy. Simply hover over any one of the thumbnails and you get a brief description. But better yet, if you pull down that little down arrow next to the word Publish you will get three choices:

  • Publish – which creates and publishes the application for you,
  • Download, which is a good way to provide a programmer a jumpstart on an application that they can customize, and
  • Preview, which is the most useful for seeing how your Map will behave in each of these applications.

I have found one that looks like it will create a web page and work on mobile devices too. An added bonus for me and what my app is supposed to be for. So I select the preview button and this pops up in a new browser window. It is a fully functional application. You can see that the Pop Ups I created in the Excel phase works well.

Being satisfied that this will work for me and that I will also create a mobile version (form adjusts to the size of the browser on the mobile device – it is still a web app) I am going to go back and Publish this Application.

One more time, I need to give the application a name and some metadata about it to help the discovery process for anyone else.

I hit Save & Publish and ArcGIS Online goes to work for a few seconds and then get the following message about the next steps.

Some of the applications do have some configuration that you can perform to futher customize it; like what color scheme to use and what tools to add. In my case I just want to share the application so I click on the link in the form and get sent to my Application Summary Screen where I can manage the application. At this point I just want to share it.

And I have one more chance to determine how widely I distribute the app.

Hit OK and I am done.  Notice the URL that is provided in the summary screen. This is all I need now to share my application; I can send this out to my team. Below are a couple of screen shots from my phone and tablet of the application I just published!

From my Android Phone

From my Android Tablet

In my final blog of this series I will show you how I can embed and use this same application within SharePoint.

Esri Location Analytics Tools in an Organization: Using Esri Maps for Microsoft Office to Publish to ArcGIS Online

During the 2012 Esri Developers Summit I posted a series of blogs. The general theme of them was that the tools that Esri was releasing in the 10.1 roll-out had really begun to consumerize the use of their product suite. I had talked about how much of the mundane (to us GIS Professionals) had been so well automated with the new products that: 1- just about anyone could do basic and even some advanced GIS, and 2- this would free up our GIS Professionals to do even greater things.

So now that the 10.1 suite has been released, I wanted to take a little time to really show how easy things have become. To do this I will use a two of the newest products within the Esri Location Analytics Toolbox, specifically Esri Maps for Microsoft Office and Esri Maps for SharePoint while leveraging the centerpiece of the current Esri technology stack, ArcGIS Online. My intent is to show how workflows that either previously did not exist or were destined to end up on a GIS professionals desk are now viably performed by anyone in an organization.

For an example, I will use a simple problem that we have dealt with in the past.  We want to be able to generate a list of hotels that we recommend for people who come to our various offices, and wouldn’t it be great to actually put them on a map and then integrate it into our corporate SharePoint site and even post that to our external website.  A year ago this would have required that we put a GIS Analyst to work generating and publishing the map, and then a developer to integrate it into our SharePoint site and our external web site.  But now with Esri Maps for Microsoft Office and ArcGIS Online, this can be done by anyone in the organization.

The first step is to generate a list of hotels; five minutes cutting and pasting from an internet search and we have a viable list. I added some information including a ranking, contact number, and hotel url which will all be available to the users once we publish the map.

Assuming you have already loaded the Esri Maps for Office, there should be a menu tab at the end of the standard Excel Tabs.

If you click on the Esri Maps Tab you will be presented with the following Ribbon.

Most of the tools are greyed out because you haven’t added a map or data to the map but that will change shortly. Notice the # 1 arrow, this means that I am signed into my ArcGIS Online account.  If I wasn’t this would say “Sign In” and I would be able to log into my account. You need to do this to be able to publish the map or data to ArcGIS Online. If you need to get an account, follow this link https://www.arcgis.com/home/signin.html

So to insert a map into your excel spreadsheet, just like you would add a chart, just click on the Insert Map button on the ribbon (# 2 on the image above). This will add a unconfigured map to your worksheet. In the image below you can see that I now have more available options on the Ribbon. A Control Panel has been added on the right as well. This behaves much like the Table of Contents does in ArcMap, as you add layers they will show up on the map and in the Control Panel.

So let’s configure the map a bit.

I like to zoom into the area I want to map and then cycle through the BaseMap options (on the Ribbon) to find the best looking and most useful map for what I am trying to show.  To do this, click in the map and then you can use your mouse wheel to zoom in and out, or click hold and drag to pan around.  Once you are centered about where you want to be, click on the BaseMap button in the Ribbon and select a different base map. Continue until you are satisfied. Here is what I have come up with so far:

Now I don’t really care for the Map 1 title, not really very informative. So to change that I click on the configure map icon in the upper right side of the map banner and now I can enter in a useful name for my map.  Once done, I click on the configure button again to close that window and return to the map.

Now I need to actually add the Hotels to the Map.  As you might have guessed, I simply click on the Add Excel Data button on the ribbon and it starts up a Wizard.

It asks me the format of the source of data I have, in this case it is a Cell Range so I select that and hit next. Then I am asked to select what cell range to use (just like it does when making a graph). Notice that I included the header, you don’t need to but I know from experience that it will speed up a step later on.

After I hit OK, I tell it what location type I have, this will tell ArcGIS Online how to do the Geocoding. I know my data is obviously addresses so I select that and click Next.

And here is where I have saved myself some time. Because I had headers in the selected cell range and they were words that the tool recognized, it automatically mapped which columns to use for which part of the address.  If I had not done this I would need to pull down the drop down for each element and select the appropriate column.

Now, once I hit the Add Data to Map, the data I have selected and the information about how to geocode the data gets sent to ArcGIS Online, is geocoded, and sent back to my map. It takes a few seconds. The more records you are mapping the longer it takes.

I now have the hotels plotted on the Map and I want to do some final configuration to the layer to make it a bit more useable. I can do this by right clicking on the name of the layer in the Map Contents Tab in the Control Panel on the right.

I definitely want to Rename the layer from Excel Data to Hotels. Click on Rename and enter “Hotels”.

And I want to use a symbol that represents more what the layer is. To do this Right Click on the Layer Name (Hotels) and select Style from the list; or click on the Style button on the Ribbon.  You have a bunch of choices here. In my case I want to use the Bed icon under the Transportation set.  I also made my icons a bit bigger than standard using the slider bar at the bottom of the form.  This all takes some trial and error but no more than if you were working out the details of a chart in Excel.

One last thing.  I want to configure the Pop-Up box that appears when someone clicks on one of the hotels.  Again, right click on the name of the layer (Hotels) and select Pop-Ups from the list. The following form appears:

Any field that was in the original range of cells you selected will show up here. I want to use the Hotel name for what shows up in the pop-up banner and because of that I don’t need it in the main body of the pop-up, so I uncheck the Visible check box for that record. I want everything else to show up.

The result looks like this:

You can see by adding information to the source spreadsheet you can provide a lot of information to the end users.  The web urls also work from the pop-up boxes. Click on the url and it will launch a new tab in your web browser and take you to that specific link.

Okay, one last step here. I have this potentially useful map and want to share it, but it is stuck in Excel. I can’t pass around the Excel file because not everyone will have the Esri Maps for Office capability. I could use the Create Slide button and that would create a PowerPoint slide of my map, but it wouldn’t be dynamic – the pop-ups wouldn’t work.  But I can publish this to ArcGIS Online and make it available online to anyone in my organization or the general public. And once I have my map online I have a bunch of other options for sharing.

How do I do that? Two ways. I can either just share the Hotels Layer and all the configurations I just did and build a map in ArcGIS Online, or I can Share the entire Map (in this case this would include the selected base map and map extent).

There is both a Share Layer and a Share Map button on the Ribbon. In both cases you will need to enter some information to make your map or layer discoverable and identifiable and set which user groups that you want to allow to use. Below shows how I have configured the Map to share.

Once you hit the Next button it will present one last choice about sharing the layers in the map. Then you hit the Share Map button and your map gets published out to your ArcGIS Online Account. This takes a few seconds to get the data packages moved and perform all the validations to ensure everything is correct. Once you get the green check mark, your Map is up and ready to use in ArcGIS Online.

If you have ArcGIS desktop you can check yourself to make sure everything transferred and that the map is behaving the way you intended. Just start ArcMap, go to add content from ArcGIS Online and you should see your map listed under My Content.  Click on the Open table and the map will load into ArcMap.

So I have created a Map in Excel using data from within Excel and ArcGIS Online, published it as a Shared map to ArcGIS Online, and verified that it is shared by pulling it into ArcMap.

In the next blog post of this series, I will show you how to use the Map I just shared in ArcGIS Online, add content from other users, build an application from a template, publish the application to use as is, and embed it into SharePoint or another web site.