Developer’s Corner: GeoProcessing Tasks 10.1 (Advanced Python)

By Joseph Yoran

We all remember the days of ArcGIS Server 10 when we could simply put a python script in a toolbox and then publish the tools.  If something was wrong with the tool or we wanted to edit the script, we could just edit the python and replace the file and “voila” it would update on the server “automagically”.  Ahh… Those were the times.

With ArcGIS Server 10.1 they have made a more “secure” process wherein your python scripts are tested before being published as a geoprocessing task.  This process, according to Esri, was to force the script to be checked for errors through ArcMap before being published to ensure the success of the geoprocessing task.  Though this process is viable, it has its drawbacks.  For example:

  1. Your python is script ready.
  2. You open ArcMap and execute your script.
    1. Everything checks out.  Good Job!
    2. You now publish this to ArcGIS Server as a geoprocessing task.
      1. Oh No! You forgot to add something!

With the published 10.1 process, you have to do the time-consuming process all over again, executing the script through ArcMap and republishing.  What a pain right?  What if I told you there was a way to complete your python scripts the old fashioned way, the 10.0 way?  Well you are in luck, there is!

Steps taken to modify the python script WITHOUT republishing:

  1. The first time through you will still have to open ArcMap and verify it is a valid script.
  2. You will also have to publish it just like you would through ArcMap to put it on ArcGIS Server.
  3. Once it is on the server, all the python files contained within the directory of the toolbox are now located here.  This is the real bread and butter.
  4. The folder location is going to look something like:

C:\arcgisserver\directories\arcgissytem\arcgisinput\”NameOfTheService.GPServer”\extracted\v101\gptasks\

You should see something like this:

Here are the python scripts that the server is actually using!

  1. From here you can modify them. (NOTE:  You might have to make a copy of them and then make edits and overwrite the original files, or edit it using an application running in Administrator Mode)
  2. That’s it! From then you should be able to make your edits without having to publish over and over.

I have tested this multiple times without problems.  This is definitely a solution for those of us who liked the old ways of 10.0.  It’s a bit tricky to find this folder at first but once you find it you will be glad you did.

Advertisements

120 Hours in the Python’s Den: PyCon 2011

Starting in Version 10, Python is the de facto geoprocessing scripting language for the ESRI suite of software.  I primarily use python to automate the data processing work I perform.  I also use Python to create geoprocessing scripts for the ArcServer environment.  This year I attended PyCon, the Python languages community convention, to further develop my Python skills and see how others are using this powerful language.

I started my excursion to Atlanta for PyCon with three questions in mind:

1)      How can I utilize the multiprocessing capabilities of the python language?  What obstacles have others faced in implementing this functionality with ESRI’s ArcPy package?

2)      I would like to create intuitive input dialogs for my geoprocessing tasks.  How can I leverage other GUI frameworks (Qt, TK, wxPython) to create input dialogs with more built in functionality?

3)      How can we utilize the cloud to reduce the time for complex and time intensive geoprocessing tasks?

I took four classes to start the conference: Python 101, Google App Engine, Creating GUI Applications in Python using Qt I, and Deploying web applications to the cloud.  I learned a few cool tidbits from these four classes.

  • The print statement will be replaced with a print function in Python 3.  Syntactically this change makes a lot of sense.  In case you are wondering, you can actually backport this functionality into the Python 2.6 that comes with Version 10.  Quick code sample:

<code>

# Old

>>> Print “%s is a very fun language for %s” % (‘python’, ‘geoprocessing’)

# New

>>> Print (“{0} is a very fun language for {1}”.format(‘python’,’geoprocessing’))

>>> Print (“{langauge} is a very fun language for {task}”.format(language = ‘python’, task = ’geoprocessing’))

</code>

  • Google App Engine is cool, but I do not have a reason to use it yet.  I feel in the future, using this service, I will be able to quickly stand up a site without having to worry about infrastructure.
  • Qt can run on almost everything and every operating system.  This is a huge plus, however I am going to hold judgment until the dust settles with Nokia.
  • In a little under three hours I was able to setup a distributed, load balanced django application on the Amazon Cloud pulling data from their Relational Database Server and distributed using their CloudFront CDN.  I definitely see this as a resource I will invest more time in learning.

It is amazing to see what people are doing with python, primarily in the distributed computing and web application end of the python development spectrum.  My new goal for the year is to finally get around to learning to develop with Django.  It seems every other presentation was a huge django proponent.  There were also a few interesting talks on the forecast of Python language development and IDE’s for writing your code. Fortunately, for you the Python Foundation videotaped most of the presentations this year, http://pycon.blip.tv/posts?view=archive.

Alright and onto a few takeaway lessons I learned from my experience at PyCon.

  • ESRI is investing tons of resources into building Python as a quick, efficient geoprocessing language for their tools.  PyCon ran concurrently with ESRI’s developer summit; however, they were still able to send out 7+ representatives to PyCon.  I spent a couple nights with the ESRI folks, and to be honest I am incredibly pumped about version 10.1.  Tons of cool new features are coming down the pike.
  • Microsoft is gradually pushing into the Python/Open Source community.  I am in the process of downloading Microsoft’s Python extension for Visual Studio.  The latest and greatest release promises to debug both cPython (what ESRI uses) and IronPython (a .Net variety).  Currently I am using WinPython, the default with ArcGIS, but I see the Visual Studio version quickly superseding its use.
  • The python community could learn a lot from the GIS side of the house.  Almost every presentation involved crunching data, processing it in some way and displaying it graphically.  I definitely see an opportunity to step in and show a more intuitive method for displaying spatial data.

Charming the Snake : Python Tips

While fearing a real python might be justified, you shouldn’t be intimidated by the Python language. Esri has embraced Python as the language that fulfills the needs of its user community. Programming with Python using the new ArcPy site-package in ArcGIS 10 makes GIS programming accessible for the average GIS professional. For longtime GIS users who yearn for the simpler days of Avenue (the scripting language used with ArcView 3.x), Python has some of the same attractive features: a manageable number of well documented classes and functions, scripts that can be easily loaded into a document and executed, code shared as simple text files, and (perhaps best of all) no need to know what QueryInterface means! Python is a widely used, nonproprietary language, so learning Python can be beneficial for non-GIS applications.

To help you get started with Python, this article will highlight a few basic techniques that are commonly required when writing

Python scripts for ArcGIS, such as how to:

  • Reference additional libraries (i.e., import functionality) for use in a script.
  • Accept user-provided arguments.
  • Get a reference to the current map document.
  • Create, open, and write to files.
  • Show messages that report the script’s progress.
  • Execute a geoprocessing tool.
  • Access objects in a map document (data frames, layers, tables).

Importing Additional Modules

Python modules are files used to organize sets of additional functionality so your script can access it. The import statement is one way to bring a module into a script and make the functionality it defines available. To import a module, simply type an import statement at the top of the script window, like those shown below:

import os

import sys

import arcpy

import arcpy.mapping as mapping

This example shows some modules you may commonly work with. The os and sys modules contain functions for working with files on disk. The ArcPy site-package allows access to geoprocessing tools. The ArcPy mapping module provides functionality for working with the map document and the objects it contains.

Reading User Inputs

Commonly, a script will need some input from the user in order to run, and when creating a script, parameters can be defined to specify what information will be entered by the user. The order of parameters defined for the script is important, because parameter values are read from an ordered list of inputs. The following code illustrates how to read user inputs. Note that the

first parameter value is referenced by index position 0:

outDir = arcpy.GetParameterAsText(0)

packageMap = arcpy.GetParameter(1)

checkBrokenLayers = arcpy.GetParameter(2)

GetParameterAsText reads user input as a text string. GetParameter reads user input and returns an object (e.g., Boolean).

Getting a Reference to the Current Map

When scripting in any environment, the current document is often the key object. Once a reference to the current document is obtained, the script can drill down into the additional objects it contains. When scripting for ArcMap, a reference to the map document will provide access to the data frames, layers, and tables the map contains. The example below uses the MapDocument function to get a reference to the current map document and store it in a variable called mxd.

mxd = mapping.MapDocument(‘Current’)

The required parameter for the MapDocument function is either a path to a map document on disk or the keyword Current to get the map in which the script is executing.

Working with a File on Disk

Use the open function (imported in the os package) to open a file on disk:

reportFile = open(reportPath, ‘w’)

The open function takes two arguments: the file name and the file mode. Valid file modes are w for write, r for read, and a for append. If the file exists, it will be overwritten. If it does not exist, it will be created.

Use the write function to write information to a file (if it was opened in either write or append mode).

reportFile.write(reportText)

reportFile.close()

Make sure to close a file when you are done working with it.

Showing Status Messages as a Script Runs

Use the AddMessage function (from the ArcPy package) to display a message as your script runs. The code below produces a message that will appear in the script’s output window as it executes.

arcpy.AddMessage(‘Writing report to ‘ + reportPath)

Executing a Geoprocessing Tool

Any geoprocessing tool that is available from ArcToolbox may be executed programmatically using ArcPy. In fact, each tool is available as a function in ArcPy. Consult the ArcGIS documentation for more information regarding available geoprocessing tools. The example below will execute the Package Map geoprocessing tool (in the Data Management toolbox).

arcpy.PackageMap_management(mxdFilePath, packagePath)

PackageMap_management requires the path to the map document and the path to the output package file.

Get Objects in the Map (Data Frames and Layers, e.g.)

Use mapping.ListDataFrames to get a list of data frames from the map.

dataFrames = mapping.ListDataFrames(mxd, ‘’)

The ListDataFrames function takes two arguments: a map document (or a path to one) and a wildcard string for filtering data frames based on name (e.g., La*). The function returns a Python list of data frames. Similar to the ListDataFrames function, ListLayers returns a Python list object containing layer objects.

layers = mapping.ListLayers(mxd, ‘’, frame)

Conclusion

With the ArcPy site-package, Esri has integrated Python into ArcGIS 10. Python’s relative simplicity and power make it the scripting language of the future for ArcGIS.

For more information, contact Thad Tilton at ttilton@gisinc.com.

Utilizing Python to Aggregate Enterprise Geodatabases

Background

The Navy’s Regional Shore Installation Management System (RSIMS) program manages approximately 35,000 feature classes spread across 12 regions and needed a way to serve up data easily with ArcGIS server. Each region manages multiple schemas within the regional SDE geodatabase representing different geographic areas within their regions. This allows for the customization of data structure and spatial reference to best fit the data for that area. The challenge was to aggregate each regional schema to a single schema for the region as well as to project the data into Web Mercator Auxiliary Sphere, which allows for integration with external web services such as ArcGIS Online. Read more of this post