---
author: einar
comments: true
date: 2006-10-23 20:32:58+00:00
layout: page
slug: text-files-with-python
title: Text files with Python
wordpress_id: 123
categories:
- General
- Linux
- Science
header:
    image_fullwidth: banner_other.jpg
---

Finally I cleaned up my code enough to post it here. It's probably still ugly, but not as ugly as when I wrote it down the first time. It's all about manipulating text files, to be precise tab-delimited files. All the snippets are published under the [GNU GPL](http://www.gnu.org/licenses/gpl.html) v2 (not that I think that anyone would use them, but just in case...).
<!-- more --> I started with one file, listing [Entrez Gene](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=gene) identifiers for a number of genes coming out of a statistical analysis, along with their [SAM](http://www-stat.stanford.edu/~tibs/SAM/) scores to see if they were differentially expressed or not. I needed to add information such as gene name and symbol, [Gene Ontology](http://www.geneontology.org), and [Uniprot/SwissProt](http://www.expasy.uniprot.org/) IDs, leaving out the statistical paramters. I tried at first to use the EUtils package of [Bioptyhon](http://biopython.org), but due to both my lack of skill and the total lack of documentation, I dropped the idea and moved to a different plan.

First, I used [DAVID](http://niaid.abcc.ncifcrf.gov/) to obtain all the annotation data I needed. There are some columns that are redundant, so I decided to remove them as well. Once I had the original files (with the SAM data) and the DAVID results, I could start:







  1. 


import sys





  2. 


import csv





  3. 


import re





  4. 


import tempfile








I used csv to easily handle comma-delimited files, tempfile to handle temporary files securely, sys to get command line arguments and re to do some regular expression matching and substituting. Basically, I had a series of functions that first of all obtained the SAM data and encoded them (1, up-regulation; 0, not differentially expressed; -1 down-regulation), creating a dictionary:







  1. 


def getSAMFlag(file):





  2. 


sam_dict={}





  3. 


for row in file:





  4. 


if row[0] == "locuslink":





  5. 


continue





  6. 


if row[22] != "NA":





  7. 


if float(row[22]) > 0:





  8. 


row[22] = "1"





  9. 


elif float(row[22]) < 0:





  10. 


row[22] = "-1"





  11. 


else:





  12. 


row[22] = "0"





  13. 


sam_dict[row[0]]=row[22]





  14. 


return sam_dict








I had also to prepare the file coming out of DAVID, stripping the useless fields. As csv.reader gives an iterator  that returns a row for each cycle, it turned out to be quite easy:







  1. 


def displayColumns(file,dest,cards=0):





  2. 


file_csv = csv.reader(file,dialect="ncbi")





  3. 


dest_csv = csv.writer(dest,dialect="ncbi")





  4. 


for row in file_csv:





  5. 


if cards == 1:





  6. 


if row[4] =="GENE_SYMBOL":





  7. 


row = row[0:2] + row[4:8] + row[3:4]





  8. 


dest_csv.writerow(row)





  9. 


continue





  10. 


geneCardsURL = "< URL removed >"





  11. 


preURL = "< a xhref=\""





  12. 


postURL = "\">"





  13. 


endURL = "< /a>"





  14. 


row[4] = re.sub(", ","",row[4]) # Togliamo la virgola e lo spazio da fine colonna





  15. 


row[4] = preURL + geneCardsURL + row[4] + postURL + row[4] + endURL





  16. 


if row[3] == "":





  17. 


row[3] = "N/A"





  18. 


row = row[0:2] + row[4:8] + row[3:4]





  19. 


dest_csv.writerow(row)








The optional "cards" parameter creates a HTML to link to the [GeneCards](http://www.genecards.org) database in order to query gene symbols. I removed the URL just for formatting purposes (and for some reason "a href" becomes "a xhref"), but it's easy to fetch it by querying by gene symbol. This code creates a new table with Entrez Gene ID, Gene Name, Gene Symbol, chromosome and cytoband and also the GO Cellular Component level 3 (adding "N/A" if there is no annotation). The re.sub is used to remove a comma followed by a space that is present at the end of the Gene Symbol annotation.Once I had all of this, I wrote a function to write the SAM results into this new table:







  1. 


def writeSAM(file,data_file,dest):





  2. 


file_csv = csv.reader(file,dialect="ncbi")





  3. 


dest_csv = csv.writer(dest,dialect="ncbi")





  4. 


data_file_csv = csv.reader(data_file,dialect="ncbi")





  5. 


sam_dict = getSAMFlag(data_file_csv)





  6. 


sam_keys = sam_dict.keys()





  7. 


for row in file_csv:





  8. 


if row[0] == "ENTREZ_GENE_ID":





  9. 


row.append("Flag SAM")





  10. 


if row[0] in sam_keys:





  11. 


row.append(sam_dict[row[0]])





  12. 


dest_csv.writerow(row)








The if for ENTREZ_GENE_ID is used to add a header ("Flag SAM") to the columns. There's nothing much to say about the actual program, if not pointing out the very easy creation of temporary files:







  1. 


temp = tempfile.NamedTemporaryFile()








And last but not least, the class definition of the dialect "ncbi" that I used to parse the text:







  1. 


class ncbi:





  2. 


delimiter = '\t'





  3. 


quotechar = '"'





  4. 


escapechar = None





  5. 


doublequote = True





  6. 


skipinitialspace = False





  7. 


lineterminator = '\n'





  8. 


quoting = csv.QUOTE_NONE








This is invoked using the csv.register_dialect method after instantiating:







  1. 


ncbi = ncbi()





  2. 


dial = csv.register_dialect("ncbi", ncbi)








Even though my programming style is probably bad, I have to notice that the code I presented is not in the order it appears in the script (obviously). In any case, if there are any suggestions to improve, let me know.