Fetching data from HTML source

Parsing html to get the data we need can be very frustrating. Lucky, Mathematica has a powerful hmtl import function, you can import raw html data into several different formats. In my experiences, import html as "XMLObject" is usually the best way to go. 
Here is an example: OSCAR Nominees:

xml = Import["http://oscar.go.com/nominees", "XMLObject"];   

We are interested in the list of nomineed films

body = Cases[xml, XMLElement["div", {"class" -> "nominee-by-film"}, ___], Infinity];

Extract titles:

title = Cases[body, XMLElement["span", {"class" -> "title"}, value_] :> value, Infinity] 

Extract the number of nominees:

nominee =
  Cases[body,
   XMLElement["h1", {"class" -> "numberOfNominations"}, value_] :>
    StringCases[value, x : NumberString :> ToExpression[x]], Infinity] ;

Put these two together:

result = Sort[Transpose[{title, Flatten@nominee}], #1[[2]] > #2[[2]] &]

Let's draw a graph to show the top 10 of the most nomineed films:

oscar = Import["http://www.oscars.org/awards/academyawards/about/awards/images/side_oscar.jpg"];
BarChart[result[[1 ;; 10, 2]],
 ChartLabels -> Placed[Flatten@result[[1 ;; 10, 1]], After],
 BarOrigin -> Left, Background -> LightBlue, ChartElements -> {oscar, {1, 1}},
  Axes -> None, LabelStyle -> {Bold, Darker@Blue, 14}] 

For this particular example, you can also try to get the same information directly from WolframAlpha.

Related post: A discussion on Mathmeatica Stackexchange

Play with Bibliographic Data

I’ve got the publication list of a research institute. There are more than 1400 entries, exported in xml format from Endnote. As a data mining project, there are plenty of things you can do with the data. Let’s assume  we are interested in the relationships among researchers and groups. Can we check this out quickly in Mathematica?

First, extract the data:

in the xml, for each publication, the author list is stored as the following:

<authors>
<author>
  <style face="normal" font="default" size="100%">Paul, N.</style>
</author>
<author>
  <style face="normal" font="default" size="100%">Cao, B.</style>
</author>
… </authors>

xml = Import["Bib.xml"];

authors = Cases[xml, XMLElement["authors", _, authors_] -> authors, Infinity];

names = Flatten@Cases[#, XMLElement["author", _, {___, XMLElement["style", _, name_]}] –> name] & /@ authors;

What we get here is the lists of authors for each publication.

Let’ see the relationship between number of authors and number of publications.

Sort[Tally[Length[#] & /@ names], #1[[1]] < #2[[1]] &]

{{1, 265}, {2, 280}, {3, 320}, {4, 224}, {5, 94}, {6, 85}, {7,
  39}, {8, 22}, {9, 20}, {10, 14}, {11, 11}, {12, 14}, {13, 5}, {14,
  3}, {15, 5}, {18, 1}, {19, 1}, {20, 1}}

We can see that most of publications have no more than 4 authors.

Next step, we like to check out the internal relationships among authors. We need to generate a network for authors. For example, if a publication has 4 authors {A, B, C, D}, the network is defined as a circle:

Flatten@(Partition[Append[authors, authors[[1]]], 2, 1] /. {x_, y_} :> {x -> y})

{A -> B, B -> C, C -> D, D -> A}

For all the publications, we get the following network:

There are two large research groups inside this institute. Then re-draw the graph with top 10 contributors, it confirms the information.

No too bad with 10 minutes coding.

I will not release the notebook this time, since I may not have the right to distribute the data. Sorry about it.

Import water data from National Weather Service

Besides real-time water data from USGS in the previous post, we like to import more data from National Weather Service, it has the forecast data on water stage.

The data is distributed in XML format.

NWSxml=Import[http://www.crh.noaa.gov/ahps2/xml/nori3_hgirg.xml];

(* function to extract data from xml *)

extractdata[xml_,group_]:=Module[{dataxml,groupdata},dataxml=Cases[NWSxml,XMLElement[group,_,_],Infinity];
groupdata=Transpose[Cases[dataxml,XMLElement["datum",{},{___,XMLElement[#,_,{x_}],___}]:>x,Infinity]&/@{"valid","primary","secondary"}];
groupdata /. {x_,y_,z_}:> {StringReplace[StringDrop[x,-9],{"T"->" "}],ToExpression[y],ToExpression[z]*1000}];

(* extract data*)

observeddata = extractdata[NWSxml, "observed"];
forecastdata = extractdata[NWSxml, "forecast"];

(* produce the graph *)

Flood stage is at 11.0 feet, it seems the white river may be in the flood stage in next several days.

How to grab photos from Flickr

Happy new year 2009!

I have a post about make a Flickr image wall with Mathematica

First, you need get some basic knowledge about Flickr API, Photo Source Url

The coding is very simple, just play with it, then you can modify it to try something new.

catxml = Import[
   "http://www.flickr.com/services/rest/?method=flickr.groups.pools.getphotos&api_key=your_API_key&group_id=75601600@N00&per_page=500&page=1&safe_search=1", "XML"];

This line will do the query from Flickr group Somebody else's cat, return 500 images’ meta data

cats = Cases[catxml, XMLElement["photo", id_, {}] :> id, Infinity];
ids = #[[All, 2]] & /@ cats;
ids = RandomSample[ids, 100];

This three lines will extract 100 random photo ids

cs="http://farm"<>#[[5]]<>".static.flickr.com/"<>#[[4]]<>"/"<>#[[1]]<>"_"<>#[[3]]<>"_s.jpg" &/@ids;
imgs=Import[#] & /@ cs;

Then get the real url to each photo and import them. Here we use small size (75 by 75) “_s.jpg”

ImageAssemble[Partition[imgs, 10]]

Then put them together into one image

So we finish it just in several lines.

Play with Images: Letter Image

Spell with Flickr is interesting. I modified the Flickr image wall code.
Here is the result:

Fun with Map: Simple Geocoding with Yahoo Map API

Yahoo Map API has a simple REST interface which works well with Mathematica.
Here is an example to show a simple Geocoding function.

Problem: Give an address, return (longitude, latitude)

The basic query format:
http://local.yahooapis.com/MapsService/V1/geocode?appid=YahooDemo
&street=?&city=?&state=?


Let's try it with the address of Wolfram Research Inc.

geocoding[{"100 Trade Center Drive", "Champaign", "IL"}]
{-88.244868, 40.097855}

Is the result right? Let's check it out with Map Image API.


image[geocoding[{"100 Trade Center Drive", "Champaign", "IL"}]]



Notice:
1. I use "YahooDemo" as AppID, if you want to use this function frequently, please apply your own ID, it is free.
2. The Geocoding API is limited to 5, 000 queries per IP per day, Google Map has the similar limitation, so if you need geocoding a large amount of addresses, please use offline geocoding application.

reference: Yahoo Map API, Map Image API, Geocoding API

update:
Google version: see 1th comment , Thanks very much.

Details on Importing GIS Data

If you live US, there are plenty of GIS data published by different government agencies, it is quite possible that the county where you live has its own GIS department now. Most of the vector data come in ESRI shape file format. There are several different ways to get shape file into Mathematica. One simple way is explained here, it uses tools based on shapelib (open source tools to read/write shape file).

Step 1. Find the data you want
Just go to the agency may have the data you are interested. In this example, I will use the Indiana county boundary data I have.
There are some free/open source GIS software you can use to check the data: Quantum GIS, ArcGIS Explorer. (No offense here, I just assume you don't much about GIS.)

Step 2. Output shaple file to data
We use shp2text, and you can find more information on its website.
In command line mode:
shp2text --gpx indiana.shp
the output is the structure of the associated data in DBF file. We like to have county name, it is in Field 1.
Field 0: Type=Integer, Title=`FIPS', Width=5, Decimals=0
Field 1: Type=String, Title=`COUNTY', Width=16, Decimals=0
Field 2: Type=String, Title=`DOQ_DISK', Width=16, Decimals=0
Field 3: Type=String, Title=`FIPS_STRG', Width=16, Decimals=0
Field 4: Type=String, Title=`ATLAS_CODE', Width=16, Decimals=0
Field 5: Type=Double, Title=`AREA_FT2', Width=19, Decimals=3
Field 6: Type=Double, Title=`AREA_ACRES', Width=19, Decimals=3
Field 7: Type=Double, Title=`PERIMETER_', Width=19, Decimals=3

shp2text --gpx indiana.shp 1 0 > output.xml

There will dump the shape file into xml file, of course, if you can dump it into plain text without "--gpx", and reformat the plain text to cvs format. I personally prefer to XML format.


see the whole document here.

Step 3. Process XML file in Mathematica
I listed the code step by step here in case you are not familiar with XML.

dataxml = Import["output.xml"];
(* extract county names *)
countyname = Drop[Cases[dataxml, XMLElement["name", __, {w_}] :> w, Infinity], 1];
(* extract {lat, lon} for each county boundary *)
wholes=Cases[dataxml, XMLElement["rte", __], Infinity] ;
counties = (Cases[#, XMLElement["rtept", __], Infinity] & ) /@ wholes;
dataLatLon = counties /. (XMLElement["rtept", latLonRules_, {}] :> { {"lat", "lon"} /. latLonRules });
(* tidy up a little, of course, you can combine them into one line code*)
test = Flatten[#, 1] & /@ dataLatLon;
test = Map[ToExpression, test];
test = Map[Reverse, test, 2];
(* then draw it *)
Graphics[{EdgeForm[Gray], FaceForm[LightGreen], Polygon[test]}]

Here is the example that uses the data imported as the background map.
Point data is from CityData, and it shows all cities with population > 1000.


Importing point, line type of data is in the similar way.

Play with images: Flickr image wall

I come across one old python script to download the images from flickr. It will be cool if it can be done in Mathematica. The goal at this lunchtime is to grab some thumbnails from group Field Guide: Birds of the World to make a small image wall.

Here is the output

Formated XML output

Image wall

Images are placed in random orders.

Achieved in 9 lines of code, not too bad.

reference: Flickr API, Photo Source Url

Created a larger version (100 photos).

Check weather with Mathematica

wxml = Import["http://www.weather.gov/data/current_obs/KBMG.xml", "XML"];

First[Cases[wxml, XMLElement[#, _, {w_}] :> w, Infinity]] & /@ {"weather", "observation_time"}

{"Light Snow", "Last Updated on Jan 1, 4:53 pm EST"}