PostgreSQL & PostGIS Cheatsheet

This is a collection of information on PostgreSQL and PostGIS for what I tend to use most often.

Installing Postgres & PostGIS
Using Postgres on the command line: PSQL
Importing Data into Postgres
Exporting Data from Postgres
Joining Tables
Upgrading Postgres
PostGIS common commands
Common PostGIS spatial queries
Spatial Indexing
Importing spatial data into PostGIS
Exporting spatial data from PostGIS
Other Methods of Interacting With Postgres/PostGIS

Installation

Postgres

to install on Ubuntu do: apt-get install postgresql
to install on Mac OS X first install homebrew and then do brew install postgresql
to install on Windows…

Note that for OS X and Ubuntu you may need to run the above commands as a super user / using sudo.

Set Up

On Ubuntu you typically need to log in as the Postgres user and do some admin things:

log in as postgres: sudo -i -u postgres
create a new user: createuser --interactive
type the name of the new user (no spaces!), typically the same name as your linux user that isn’t root. You can add a new linux user by doing adduser username.
typically you want the user to have super-user privileges, so type y when asked.
create a new database that has the same name as the new user: createdb username

For Mac OS X you can skip the above if you install with homebrew.

For Windows….

Starting the Postgres Database

On Mac OS X:

to start the Postgres server do: postgres -D /usr/local/var/postgres
or do pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start to start and pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log stop to stop
to have Postgres start everytime you boot your Mac do: ln -sfv /usr/local/opt/postgresql/*.plist ~/Library/LaunchAgents then to check that it’s working after booting do: ps ax | grep sql

PostGIS

On Ubuntu do apt-get install postgis
On Mac OS X the easiest method is via homebrew: brew install postgis
(note that if you don’t have Postgres or GDAL installed already it will automatically install these first).
to install on Windows…

psql

psql is the interactive unix command line tool for interacting with Postgres/PostGIS.

Common Commands

log-in / connect to a database name by doing psql -d database_name
for doing admin type things such as managing db users, log in as the postgres user: psql postgres;
to create a database: CREATE DATABASE database_name;
to connect to a database: \c database_name;
to delete a database DROP DATABASE database_name;
to connect when starting psql use the -d flag like: psql -d nyc_noise
to list all databases: \l
to list all the table in a database: \dt
to list extensions installed in a database: \dx
to quit psql: \q
to grant privileges to a user (requires logging in as postgres ):

GRANT ALL PRIVILEGES ON DATABASE mydb TO myuser;
to enable the hstore extension ( for key : value pairs, useful when working with OpenStreetMap data) do: CREATE EXTENSION hstore
to view columns of a table: \d table_name
to list all columns in a table (helpful when you have a lot of columns!):
select column_name from information_schema.columns where table_name = 'my_table' order by column_name asc;
to rename a column:
alter table noise.hoods rename column noise_sqkm to complaints_sqkm;
to change a column’s data type:
alter table noise.hoods alter column noise_area type float;
to compute values from two columns and assign them to another column: update noise.hoods set noise_area = noise/(area/1000);
to search by wildcard use the like (case sensitive) or ilike (treats everything as lowercase) command:
SELECT count(*) from violations where inspection_date::text ilike '2014%';

to insert data into a table:

INSERT INTO table_name (column1, column2)
VALUES
    (value1, value2);

to insert data from another table:

INSERT INTO table_name (value1, value2)
SELECT column1, column2
FROM other_table_name

to remove rows using a where clause:
DELETE FROM table_name WHERE some_column = some_value

list all column names from a table in alphabetical order:

select column_name
from information_schema.columns
where table_schema = 'public'
and table_name = 'bk_pluto'
order by column_name;

List data from a column as a single row, comma separated:
1. SELECT array_to_string( array( SELECT id FROM table ), ',' )
2. SELECT string_agg(id, ',') FROM table
rename an existing table:
ALTER TABLE table_name RENAME TO table_name_new;
rename an existing column of a table:
ALTER TABLE table_name RENAME COLUMN column_name TO column_new_name;

Find duplicate rows in a table based on values from two fields:

  select * from (
    SELECT id,
    ROW_NUMBER() OVER(PARTITION BY merchant_Id, url ORDER BY id asc) AS Row
    FROM Photos
  ) dups
  where
  dups.Row > 1

credit: MatthewJ on stack-exchange

Bulk Queries are efficient when doing multiple inserts or updates of different values. For example,

--- update some rows with new values
  UPDATE election_results o
  SET votes=n.votes, pro=n.pro
  FROM (VALUES (1,11,9),
               (2,44,28),
               (3,25,4)
        ) n(county_id,votes,pro)
  WHERE o.county_id = n.county_id;

--- insert new values
  INSERT INTO election_results (county_id,voters,pro)
  VALUES  (1, 11,8),
          (12,21,10),
          (78,31,27);

The INSERT and UPDATE queries can be combined to what is often referred to as an UPSERT query:

  WITH
  -- make a temporary (as in for this query only) table of values
  n(ip,visits,clicks) AS (
    VALUES ('192.168.1.1',2,12),
           ('192.168.1.2',6,18),
           ('192.168.1.3',3,4)
  ),
  -- update existing rows
  upsert AS (
    UPDATE page_views o
    SET visits=n.visits, clicks=n.clicks
    FROM n WHERE o.ip = n.ip
    RETURNING o.ip
  )
  -- insert missing rows
  INSERT INTO page_views (ip,visits,clicks)
  SELECT n.ip, n.visits, n.clicks FROM n
  WHERE n.ip NOT IN (
    SELECT ip FROM upsert
  );

credit: FASTER DATA UPDATES WITH CARTODB

Importing Data

import data from a CSV file using the COPY command:

  COPY noise.locations (name, complaint, descript, boro, lat, lon)
  FROM '/Users/chrislhenrick/tutorials/postgresql/data/noise.csv' WITH CSV HEADER;

import a CSV file “AS IS” using csvkit’s csvsql (requires python, pip, csvkit, psycopg2):
```
  csvsql --db postgresql:///nyc_pluto --insert 2012_DHCR_Bldg.csv
```

Exporting Data

export data as a CSV with Headers using COPY:

  COPY dob_jobs_2014 to '/Users/chrislhenrick/development/nyc_dob_jobs/data/2014/dob_jobs_2014.csv' DELIMITER ',' CSV Header;

to the current workspace without saving to a file:

  COPY (SELECT foo FROM bar) TO STDOUT CSV HEADER;

from the command line w/o connecting to postgres:

  psql -d dbname -t -A -F"," -c "select * from table_name" > output.csv

Joining Tables Using a Shared Key

From CartoDB’s tutorial Join data from two tables using SQL

Join two tables that share a key using an INNER JOIN(Postgresql’s default join type):

  SELECT table_1.the_geom,table_1.iso_code,table_2.population
  FROM table_1, table_2
  WHERE table_1.iso_code = table_2.iso

To update a table’s data based on that of a join:

  UPDATE table_1 as t1
  SET population = (
    SELECT population
    FROM table_2
    WHERE iso = t1.iso_code
    LIMIT 1
  )

aggregate data on a join (if table 2 has multiple rows for a unique identifier):

  SELECT
    table_1.the_geom,
    table_1.iso_code,
    SUM(table_2.total) as total
  FROM table_1, table_2
  WHERE table_1.iso_code = table_2.iso
  GROUP BY table_1.iso_code, table_2.iso

update the value of a column based on the aggregate join:

  UPDATE table_1 as t1
  SET total =  (
    SELECT SUM(total)
    FROM table_2
    WHERE iso = t1.iso_code
    GROUP BY iso
  )

Upgrading Postgres

This Tutorial was very helpful for upgrading on Mac OS X via homebrew.

WARNING: Back up your data before doing this incase you screw up like I did!

Basically the steps are:

Shut down Postgresql:
launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
Create a new Postgresql9.x data directory:
initdb /usr/local/var/postgres9.4 -E utf8

Run the pg_upgrade command:

 pg_upgrade \
 -d /usr/local/var/postgres \
 -D /usr/local/var/postgres9.4 \
 -b /usr/local/Cellar/postgresql/9.3.5_1/bin/ \
 -B /usr/local/Cellar/postgresql/9.4.0/bin/ \
 -v

Change kernel settings if necessary:
```
 sudo sysctl -w kern.sysv.shmall=65536
 sudo sysctl -w kern.sysv.shmmax=16777216
```
- I also ran sudo vi /etc/sysctl.conf and entered the same values:
```
kern.sysv.shmall=65536
kern.sysv.shmmax=16777216
```
- re-run the pg_upgrade command in step 3

Move the new data directory into place:

 cd /usr/local/var
 mv postgres postgres9.2.4
 mv postgres9.3 postgres

Start the new version of PostgreSQL: launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
- check to make sure it worked:
```
psql postgres -c "select version()"
psql -l
```
Cleanup:
- vacuumdb --all --analyze-only
- analyze_new_cluster.sh*
- delete_old_cluster.sh*
- brew cleanup postgresql
  (* scripts were generated in same the directory where pg_upgrade was ran)

PostGIS

PostGIS is the extension for Postgres that allows for working with geometry data types and doing GIS operations in Postgres.

Common Commands

to enable PostGIS in a Postgres database do: CREATE EXTENSION postgis;
to enable PostGIS topology do: CREATE EXTENSION postgis_topology;
to support OSM tags do: CREATE EXTENSION hstore;

create a new table for data from a CSV that has lat and lon columns:

  create table noise.locations
  (                                     
    name varchar(100),
    complaint varchar(100), descript varchar(100),
    boro varchar(50),
    lat float8,
    lon float8,
    geom geometry(POINT, 4326)
  );

inputing values for the geometry type after loading data from a CSV:
update noise.locations set the_geom = ST_SetSRID(ST_MakePoint(lon, lat), 4326);
adding a geometry column in a non-spatial table:
select addgeometryColumn('table_name', 'geom', 4326, 'POINT', 2);
calculating area in EPSG 4326:
alter table noise.hoods set area = (select ST_Area(geom::geography));

Common Spatial Queries

You may view more of these in my intro to Visualizing Geospatial Data with CartoDB.

Find all polygons from dataset A that intersect points from dataset B:

SELECT a.*
FROM table_a_polygons a, table_b_points b
WHERE ST_Intersects(a.the_geom, b.the_geom);

Find all rows in a polygon dataset that intersect a given point:

-- note: geometry for point must be in the order lon, lat (x, y)
SELECT * FROM nyc_tenants_rights_service_areas
where
ST_Intersects(
  ST_GeomFromText(
   'Point(-73.982557 40.724435)', 4326
  ),
  nyc_tenants_rights_service_areas.the_geom    
);

Or using ST_Contains:

SELECT * FROM nyc_tenants_rights_service_areas
where
st_contains(
  nyc_tenants_rights_service_areas.the_geom,
  ST_GeomFromText(
   'Point(-73.917104 40.694827)', 4326
  )      
);

Counting points inside a polygon:

With ST_Containts():

SELECT us_counties.the_geom_webmercator,us_counties.cartodb_id,
count(quakes.the_geom)
AS total
FROM us_counties JOIN quakes
ON st_contains(us_counties.the_geom,quakes.the_geom)
GROUP BY us_counties.cartodb_id;

To update a column from table A with the number of points from table B that intersect table A’s polygons:

update noise.hoods set num_complaints = (
    select count(*)
    from noise.locations
    where
    ST_Intersects(
        noise.locations.geom,
        noise.hoods.geom
    )
);

Select data within a bounding box
Using ST_MakeEnvelope

HINT: You can use bboxfinder.com to easily grab coordinates of a bounding box for a given area.

SELECT * FROM some_table
where geom && ST_MakeEnvelope(-73.913891, 40.873781, -73.907229, 40.878251, 4326)

Select points from table a that do not fall within any polygons in table b
This method makes use of spatial indexes and the indexes on gid for better performance

SELECT
  a.gid,
  a.st_address,
  a.city,
  a.st_num,
  a.the_geom
FROM
  points AS a LEFT JOIN
  polygons AS b ON
  ST_Intersects(a.the_geom, b.the_geom)
WHERE b.gid IS NULL;

credit: Nicklas Avén

Make a line from a series of points

SELECT ST_MakeLine (the_geom ORDER BY id ASC)
AS the_geom, route
FROM points_table
GROUP BY route;

Order points in a table by distance to a given lat lon
This one uses CartoDB’s built-in function CDB_LatLng which is short hand for doing:
SELECT ST_Transform( ST_GeomFromText( 'Point(-73.982557 40.724435)',),4326)

SELECT * FROM table
ORDER BY the_geom <->
CDB_LatLng(42.5,-73) LIMIT 10;

Access the previous row of data and get value (time, value, number, etc) difference

WITH calc_duration AS (
 SELECT
 cartodb_id,
 extract(epoch FROM (date_time - lag(date_time, 1) OVER(ORDER BY date_time))) AS duration_in_seconds
 FROM tracking_eric
 ORDER BY date_time
)
UPDATE tracking_eric
SET duration_in_seconds = calc_duration.duration_in_seconds
FROM calc_duration
WHERE calc_duration.cartodb_id = tracking_eric.cartodb_id

Select population density

In this query we cast the geometry data type to the geography data type to get units of measure in meters.

SELECT pop_sqkm,
 round( pop / (ST_Area(the_geom::geography)/1000000))
 as psqkm
 FROM us_counties

Repair Invalid Geometries
Sometimes when data is imported into PostGIS geometries get screwed up. If you get an error message like:

ERROR:  GEOSIntersects: TopologyException: side location conflict at -116.03227135270012 33.309736898054787

You can try doing:

UPDATE tablename SET geom=ST_MAKEVALID(geom) WHERE NOT ST_ISVALID(geom);

Spatial Indexing

Makes queries hella fast. OSGeo has a good tutorial.

Basically the steps are:
CREATE INDEX table_name_gix ON table_name USING GIST (geom);
VACUUM ANALYZE table_name
CLUSTER table_name USING table_name_gix;
Do this every time after making changes to your dataset or importing new data.

Importing Spatial Data to PostGIS

Using shp2pgsql

Create an SQL file with a CREATE TABLE statement:

 shp2pgsql -I -s 4326 nyc-pediacities-hoods-v3-edit.shp noise.hoods > noise.sql

Or for using the geography data type do:

 shp2pgsql -G -I nyc-pediacities-hoods-v3-edit.shp noise.nyc-pediacities-hoods-v3-edit_geographic > nyc_pediacities-hoods-v3-edit.sql

If you’d like to transform your data from one SRID to another, you may pass two EPSG codes separated by a colon to the -s flag:

 shp2pgsql -I -s 2236:4326 shapefile_in_2236.shp schema.table-name > table-name.sql

Then run the SQL using psql:

 psql -d nyc_noise -f noise.sql

Or for the geography type above:

 psql -d nyc_noise -f nyc_pediacities-hoods-v3-edit.sql

Alternatively do both of the above in a single command:

 shp2pgsql -I -s 4326 nyc-pediacities-hoods-v3-edit.shp noise.hoods | psql -d nyc_noise

Using osm2pgsql

To import an OpenStreetMap extract in PBF format do:
osm2pgsql -H localhost --hstore-all -d nyc_from_osm ~/Downloads/newyorkcity.osm.pbf

Using ogr2ogr

Example importing a GeoJSON file into a database called nyc_pluto:

ogr2ogr -f PostgreSQL \
PG:"host='localhost' user='chrislhenrick' port='5432' \
dbname='nyc_pluto' password=''" \
bk_map_pluto_4326.json -nln bk_pluto

Exporting Spatial Data from PostGIS

The two main tools used to export spatial data with more complex geometries from Postgres/PostGIS than points are pgsql2shp and ogr2ogr.

Using pgsql2shp

pgsql2shp is a tool that comes installed with PostGIS that allows for exporting data from a PostGIS database to a shapefile format. To use it you need to specify a file path to the output shapefile (just stating the basename with no extension will output in the current working directory), a host name (usually this is localhost), a user name, a password for the user, a database name, and an SQL query.

pgsql2shp -f <path to output shapefile> -h <hostname> -u <username> -P <password> databasename "<query>"

A sample export of a shapefile called my_data from a database called my_db looks like this:

pgsql2shp -f my_data -h localhost -u clhenrick -P 'mypassword' my_db "SELECT * FROM my_data "

Using ogr2ogr

Note: You may need to set the GDAL_DATA path if you git this error:

ERROR 4: Unable to open EPSG support file gcs.csv.
Try setting the GDAL_DATA environment variable to point to the
directory containing EPSG csv files.

If on Linux / Mac OS do this: export GDAL_DATA=/usr/local/share/gdal
If on Windows do this: C:\> set GDAL_DATA=C:\GDAL\data

To Export Data
Use ogr2ogr as follows to export a table (in this case a table called dob_jobs_2014) to a GeoJSON file (in this case a file called dob_jobs_2014_geocoded.geojson):

ogr2ogr -f GeoJSON -t_srs EPSG:4326 dob_jobs_2014_geocoded.geojson \
PG:"host='localhost' dbname='dob_jobs' user='chrislhenrick' password='' port='5432'" \
-sql "SELECT bbl, house, streetname, borough, jobtype, jobstatus, existheight, proposedheight, \
existoccupancy, proposedoccupany, horizontalenlrgmt, verticalenlrgmt, ownerbusinessname, \
ownerhousestreet, ownercitystatezip, ownerphone, jobdescription, geom \
FROM dob_jobs_2014 WHERE geom IS NOT NULL"

note: you must select the column containing the geometry (usually geom or wkb_geometry) for your exported layer to have geometry data.

PostgreSQL & PostGIS cheatsheet (a work in progress)

PostgreSQL & PostGIS Cheatsheet

TOC

Installation

Postgres

Set Up

Starting the Postgres Database

PostGIS

psql

Common Commands

Importing Data

Exporting Data

Joining Tables Using a Shared Key

Upgrading Postgres

PostGIS

Common Commands

Common Spatial Queries

Spatial Indexing

Importing Spatial Data to PostGIS

Using shp2pgsql

Using osm2pgsql

Using ogr2ogr

Exporting Spatial Data from PostGIS

Using pgsql2shp

Using ogr2ogr

Other Methods of Interacting With Postgres/PostGIS

PGAdmin

Python

Node JS