Text file processing with IDL

A friend recently asked me how to process a text file. It was a collection of measurements of plants roots (size and mass) made at various depths from different pits; looking like (after some formatting):

id     depth_name     mass     size
1     30-50              0.5         10
1     30-50              0.45       3
1     30-50              0.3         4.2
1     50-70              0.7         5
1     50-70              0.72       5.3
...    ...                     ...           ...
2     30-50             0.8         4

This friend needed to sum up roots lengths for a same pit and depth.
First you have to format well your file. By well formatted, I mean: use a separator like a comma ‘,’ or a space (if not ambiguous). Then read the file with the READ_ASCII command:

pro process_file
textfile='path/to_my/textfile.txt'
data=read_ascii(textfile, template=ascii_template(textfile))
end
read_ascii

excepts a file template (you can indicate to jump some heading line, choose a column separator and indicate the data type per column). In this case, I choose a string format for the second column.
Alright. The read_ascii command stores the text file value in the data structure. In this case, data has the following fields:
data.field1 stores pits ids
data.field2 stores depths labels
data.field3 stores the mass column
data.field4 stores the roots length column.
Note the structure is ‘field’ par idl, you can’t change it.

Le’s say we want the list of pits ids:
listpitds=data.field1(uniq(data.field1))
uniq function returns ending position of continuous series of values in an array.
Let’s loop over the list of pit ids, and for each pit id, get the list of unique depths-labels (again!) and for these selected lines, sum up roots lengths:

for ipid=0, n_elements(listpits)-1 do begin
  ; now get list of depths NAMES
  wherePits = where(data.field1 EQ listpits[ipid])
  allDepths=data.field2(wherePits)
  listDepths=allDepths[uniq(allDepths)]

  ; now for this pit id, sum for each type of depths
  for idepths=0, n_elements(listDepths)-1 do begin
    ; get position of data to sum
    wts = where( (data.field1 eq listpits[ipid]) AND (
           strcmp(data.field2, listDepths[idepths]) ) )
    print, 'pits: ',strcompress(listpits[ipid]), ', at depth: ', 
           listDepths[idepths],' cm total root length is ',
           total(data.field5[wts])
  endfor
endfor

Easy no?
Complete program is below:

pro process_file

textfile='E:\field_data.csv'

;myTemplate=ascii_template(textfile)
;save, myTemplate,filename='E:\myTemplate.sav'
restore, 'E:\myTemplate.sav'
data = read_ascii(textfile, template=myTemplate)

; now data are in data.field1, data.field2 etc.
; to see: help,data,/structure
data.field2=strcompress(data.field2)
; what is the list of id (i.e. of pits)?
; Caution: assume list already sorted, i.e. pits id are not mixed...

; get list of pits id
listpits=data.field1[uniq(data.field1)]

; loop over the list of pits
for ipid=0, n_elements(listpits)-1 do begin
  ; now get list of depths NAMES
  wherePits = where(data.field1 EQ listpits[ipid])
  allDepths=data.field2(wherePits)
  listDepths=allDepths[uniq(allDepths)]

  ; now for this pit id, sum for each type of depths
  for idepths=0, n_elements(listDepths)-1 do begin
    ; get position of data to sum
    wts = where( (data.field1 eq listpits[ipid]) AND (
          strcmp(data.field2, listDepths[idepths]) ) )
    print, 'pits: ',strcompress(listpits[ipid]), ', at depth: ', 
         listDepths[idepths],' cm total root length is ',
         total(data.field5[wts])
  endfor
endfor
end

0 Responses to “Text file processing with IDL”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





%d bloggers like this: