Introduction Segment Assembly   Program Features / limitations   Key notes   Installation   Work flow   Running LDMAP-cluster on Iridis   FAQ / Troubleshooting Program history Source code   Download LD maps   Related publications   CHROMSCAN-cluster program breakdown known problems other tools...
	SP = Sub Process; SO = Sub Output

LDMAP program has been modified to be deployed on the Iridis computing cluster at University of Southampton. The construction of the genome-wide Linkage Disequilibrium Unit (LDU) maps is made possible by adopting the segmental approach. This method divides an entire chromosome into segments, and the LDU map in each segment is constructed independently as an individual job. With the availability of the large computing resource, the segments are able to compute in parallel. The diagram above illustrates the concept of parallel processing. Once the segmental maps are constructed, the segments are concatenated to assemble a whole chromosome map.  The segment size (number of SNPs) is optimized to minimize computing load but lose only negligible amounts of information (1000 - 2000 SNPs per segment) for delivering an effective turnover computation time.
 

The Linkage Disequilibrium unit map is constructed in segments, and these segments can be concatenated in 2 ways:
        1) Contig Assembly
        2) Contig-Free Assembly (more efficient, recommended approach)
 

The genome-wide LDU maps for the human genome of four populations have been constructed using the Phase 2 HapMap public release #20 (January 2006). The maps are summary here.

The Iridis cluster is put together from dual processors AMD Opteron machines, and consists of over 900 processors capable of performing both 32 bit and 64 bit computing under Linux environment. This configuration is commonly known as the Beowulf cluster. The cluster job scheduler is managed by open-PBS (Portable Batch System). This is a shared facility which offers high performance computation for researchers across the whole university.

Fine scaled LDU maps are made using the AMD Opteron on Iridis cluster, click here for more details.

The modified LDMAP program is named LDMAP-cluster. The predecessor LDMAP-Condor is deployed on a Linux Condor Pool of ~80 nodes at the Computational Engineering and Design Centre (CEDC), School of Engineering Sciences at the university.

The main features / limitations of LDMAP_cluster are:

Work Flow Overview: Constructing LD maps using LDMAP_cluster:

The work flow diagram can be downloaded in pdf format here.

There are 2 components for the installation:
    -       download the LDMAP_cluster program (zip fle)
    -       download the SNP dataset file

Log on and Download the LDMAP_cluster program:

The program can be executed from any sub-directory.

1) Log on to your Iridis account (via SSH Secure Shell), instructions here

Account holder of  previous Iridis account should has access to the current Iridis cluster, please make sure the above host is used for logging in.


2) Download the program

There are 2 ways to download the source code and onto the home directory (~) on Iridis:
       i)  FTP (for both source code + dataset files)
       ii) HTTP (for source code only)

It is recommended on FTP, "binary" mode is used for downloading the zip file, and "ascii" mode is used for transferring the dataset files.

FTP has to be initiated from Iridis for connecting to other servers, vice versa is not possible.


The LDMAP_cluster is packaged in a zip file LDMAP_cluster_<date>.zip, please replace <date> with the release version date showed on the top of the page, i.e. "ver. 16Oct06" = "LDMAP_cluster_161006.zip".

Initiate FTP from Iridis to connect to Cedar:

3) Extract the program from the zip file :

The installation is completed.

The program automatic updates if a newer version of LDMAP_cluster becomes available

(this function is currently disabled for security reason).



Download the SNP dataset for Phase II HapMap

Alternatively for other SNP datasets,  please use SSH Secure Shell Client (from your local PC to Iridis) to transfer the dataset.

1) The LDMAP_cluster program is executed by:

 

2) The program requires the user to define the following inputs:

    -    the SNP dataset file (e.g. ceu10.dat)

    -    Parallel Processing: Construct the map from multiple segments (select "1")

    -    Non-Parallel Processing: Construct the map from entire dataset without segmentation (select "2")

 

                                                              
    -    the # of SNPs per segment (2000 SNPs)
    -    the size of "Interval Window"(100 SNPs)  
    -    the size of "Buffer Zone" (100 SNPs)

    -    the physical distance (500 kb)

    -    the hours of computing per segment (10 hrs)

e.g. Assigning 2000 SNPs per segment:

    first segment    =   2000 SNPs          +   100 SNPs X 1 contig
  
    middle segments  =   2200 SNPs          +   100 SNPs X 2 contigs
   
    last segment     =   1000 - 1999 SNPs   +   100 SNPs X 1 contig
         or
    last segment     =   2000 - 2999 SNPs   +   100 SNPs X 1 contig (see program history)

 

If option 2 is selected to construct the map from entire dataset without segmentation, the user only requires to define the followings:

    -    the size of "Interval Window"(100 SNPs)  
    -    the physical distance (500 kb)

    -    the hours of computing per segment (10 hrs)

 

Please note that the purpose of option 2 is to construct the LD map from small dataset without losing any information in the absent of segmentation. Ideally the dataset should not exceed 4000 SNPs, as it will prolong the computing time and make the process inefficient (long queuing time for long computing job; possible time out if insufficient computing time is assigned). 

 

 

Each submission script consists of two subJobs. Each subJobs represents a split segment of the SNP dataset (2nd + 3rd lines). This is followed by the submission script filename (4th line) and the submission job number (last line).





Once the jobs have been submitted to Iridis, new files and folders are created by the LDMAP_cluster program. Each segment is submitted from an individual folder (seg_#) accompanied by a PBS submission script (submitfile_#.csf).

3) Monitoring the submitted jobs on Iridis:

"checkSeg" command provides a simple status summary of the submitted jobs:


more details about checkSeg please click here.

Alternatively, there are 2 commands that also display the status of the submitted job(s): "qstat" and "showq" commands.

[wwsl@blue15 ~]$ qstat

"qstat" provides a more details summary about the submitted job. The long number represents the job id for the submitted job. Typically, a segment of 2000 SNPs would take 1-2 hrs to execute; a segment of 2000 SNPs would take 2-5 hrs to execute, some take up to 10 hrs, depending on the number of iterations are needed.

Quite often the job names are too long to be displayed on the list from "qstat", the left part of the job name which distinguishes the dataset is often cut off, therefore it is hard to tell which dataset the submission belongs to. Try "qstat -a" to display the left part of the job name. It is preferable to use the "qstat -a" as it gives a better account (i.e. executed time) than "qstat". Click here for more details.
 

[wwsl@blue15 ~]$ qstat -a

"showq" command displays the job queue of all users on the Iridis cluster. It consists of 3 lists: Active Jobs / Idle Jobs / Blocked Jobs.

If a user has over 40 jobs running on the cluster, the excess jobs are blocked from executing. This makes sure that a fair share of the resource is available for all the users.

Should anything go wrong (e.g. incorrect parameters are input, the wrong dataset is submitted), use "qdel <job id>" to remove the submitted jobs before they started running on the computing cluster. More than one job number can be placed in the arguments of "qdel". Click here for more details.

[wwsl@blue15 ~]$ qdel 541385

Currently the user is required to manually remove the folders and submission scripts related to the finished or wrong dataset. The following command is recommended:

Remove the folders and submission scripts associated to a wrongly submitted dataset

[wwsl@blue15 ~]$  rm -rf chb4_1.dat*

4) Retrieving Results:

Run "checkSeg" to confirm if all segments have completed.




Under rare circumstances, there might be job(s) or segment(s) shown as "Idle" among the completed segments on the list from "checkSeg". The "Idle" segment(s) will not be found on either  "qstat -a" or  "showq". Please click here for information.

e541359) are generated by the PBS for every job submission. These files contain the job and error log for each submitted job.

[wwsl@blue15 ~]$ retrieveResult

The two folders are created by "retrieveResult". The log folder contains all the job / error log for the submitted jobs. The result folder contains all the segments for the dataset.

 

[wwsl@blue15 ~]$ cd result_chb4_2.dat_1143996184/

Inside the result_# folder, there are two types of file:

[wwsl@blue15 ~/result_1143996184]$ CreateLDmapFree

 

6) Backing Up result and useful information:

It is recommended to back up the two folders:
    1) result_<dataset>_<time>     
    2) log_<dataset>_<time>         
 

e.g.

 

They contain valuable information about the LD maps as well as the computing log from the PBS.

The easiest way to transfer the result (the above 2 folders) from Iridis is via the software SSH Secure Shell where the folders could "drag and drop" to anywhere on the Windows machine. Alternatively, FTP could be used to export the two folders where it is necessary to zipped the folders before transferring. "ascii mode" is recommended for transferring flat files.


7) Cleaning Up

To free more space for the new dataset, remove the folders and submission scripts of the complete dataset:

Storage Space Limit
A space of 5Gb is given to every user on Iridis.

To check how much space is used out of the given 5Gb. At your home directory, type "du -h -s <folder>". Replace "wwsl" with your <userID>.

[wwsl@blue15 ~]$ du -h -s ../wwsl

Concatenating LD maps constructed from different datasets.
Coming Soon......




Effective Iridis Submission (execution time v.s number of jobs):

The PBS gives higher priority for the jobs with shorter execution time and request lower number of processors first before the jobs request more processors and longer execution time.

LDMAP_cluster adjusts the requested execution time according to the number of SNPs in the segment (job). It is important to quote a sufficient time for the job to complete. The job is removed from the computing node once the requested time is up whether it is finished or not.




Typically a segment of 2000 SNPs (excluding contig(s): 100-200 SNPs) requires 2 - 5 hrs to complete it execution. The execution time varies between different segments although they contain the same number of SNPs. The computing time is largely depends on the number of  Newton-Raphson iterations required for the segment to converge. Usually 800 iterations are achieved within the first hour; 1600 iterations in ~2hrs; 3200 iterations in ~4hrs.Some segments might require >3200 iterations, therefore 10 hrs is defaulted as the requested execution time for all segments containing 2000 SNPs.

Currently (09Apr06), the LDMAP_cluster processes one sample (SNP dat file) at a time, the next sample is manually submitted after the completion of the previous run. This creates a time gap between each run. A way to






Program Outlook:

The construction of LD maps has benefited greatly from deploying the program on the large computing resource on Iridis at the Southampton University.

The National e-Science Grid
The computational and data gird resources provided by the National Grid Service (NGS). It consists of 7000 processors.
 

The LDMAP_cluster and its source code are freely available to non-commercial users only.

Please click here to download the source code (ver. 210906).

Please click here to download the source code (ver. 161006).

Alternatively, under Linux environment, please use the following command replacing the version number if necessary.

Lau, W., Kuo, T., Tapper, W., Cox, S., Collins, A. Exploiting large scale computing to construct high resolution linkage disequilibrium maps of the human genome. Bioinformatics 2007: 23; 517-519. [ ABSTRACT ]

Collins, A., Lau, W., De La Vega, F. M. Mapping Genes for Common Diseases: The Case for Genetics (LD) Maps. Human Heredity 2004: 58; 2-9 [ ABSTRACT ]

29Mar06   

• Transformed the LDMAP_Condor to LDMAP_cluster program under Iridis (e.g. job submission script).
• Added function to take care of the the last segment. If it contains > 800 SNPs it will merge to the previous segment as one job; otherwise submit as an individual job (this limit increases to >1000 SNPs on 12Apr06). see table

• The minimum number of SNPs in the last segments is 1000 SNPs (up from 800 SNPs, 29Mar06). This increases the accuracy of the construction of LD map in the last segment. see table
• checkSeg command now embedded the function to delete the dataset file and temp.dat (~30Mb) from the segment folder once the job started to process; and to delete the intermediate file (~40Mb) once the segment map (ter file) is constructed.
• Added display for estimate required space (Mb) in the UI of LDMAP_cluster program before submission, based on the assumption of 2000 SNPs per segment and the SNP dataset file size of 15Mb.
• Fixed bugs in mergeFreeContig3.c that implements Contig-Free Assembly.

• Amended wordings for checkSeg.

• Only the ldmapper1+ executable and the dataset file in each segment folder rather than including all the source code

• Added feature for user to have the full control to alter "Interval Window size" and physical distance (kb) for the informative SNPs. 

• Fixed Bug in respect for constructing the LD map from entire dataset in single segment submission.
• Added option for user to choose entire dataset or segment for constructing the LD map.
• Amended web link, and UI wordings.
• Fixed potential bug on submitting the job before creating the folder and moving the dataset file, this has been reported previously in the chromScan_cluster where the dataset is large and requires longer time to duplicate.

• Added feature to allow user to execute LDMAP_cluster from any sub-directory.